Consultant: Final Project Evaluation- AVE (Amplify, Verify, Engage) London
Job description
Position: Final Project Evaluation- AVE (Amplify, Verify, Engage) Contract type: Consultant Location: Remote (London preferred) Start date: ASAP Duration: Approximately 6 weeks -To be completed by Friday 8th December 2023. INTRODUCTION: The Institute for War & Peace Reporting (IWPR) is an independent not-for-profit organisation working to empower local voices on the frontline to drive positive change in areas of conflict, crisis or transition around the world. For over 30 years IWPR has been promoting reliable information and public debate to build more stable, just and inclusive societies. The Amplify, Verify, Engage (AVE): Information for Democratisation and Good Governance in Eurasia project promotes democratic growth, participatory governance, and cross-border cooperation, by strengthening civil society and independent media across Central Asia, Eastern Europe, and the Caucasus. Building on IWPR's previous Norwegian-supported initiatives - including the Giving Voice, Driving Change project - AVE continues to promote democratic growth, participatory governance, and cross-border cooperation, by strengthening civil society and independent media across Central Asia, Eastern Europe, and the Caucasus. Running from 01 May 2021 to 31 December 2023 (30 months), the Amplify, Verify, Engage (AVE) project has three key objectives: Amplify journalists and independent media voices producing verified, evidence-based reporting; Verify reliable information by strengthening the capacity of civil society and independent media to counter misinformation and disinformation; Engage civil society, independent media, public officials and the general public to foster increased dialogue and cooperation. Established in collaboration with and funded by the Norwegian Ministry of Foreign Affairs (MFA), the project directly supports Norwegian priorities and values, builds on long-standing Norwegian supported programming in the region, and expands on IWPR’s extensive local and regional networks of independent media and civic groups. More information and content from the project may be found on IWPR’s website. OBJECTIVE OF THE EVALUATION: The overall objective of this evaluation is to provide a final review of the Amplify, Verify, Engage (AVE) project by assessing its performance according to the OECD-DAC criteria and other specific criteria. This final evaluation has three main objectives: To assess and analyse the achievements after 30 months of project implementation, compared to the expected results and in line with the theory of change, proposal and results framework that underpinned the design and ambitions of the project. This will involve determining the current level of achievement of the project's indicators (for project objectives and results) and comparing the final achievements with the baseline situation. As for the theory of change, the aim will be to examine the hypotheses of the causal links between actions and results and to confirm or disconfirm the explicit or implicit causal steps of the theory of change. Capture observable changes in practices, behaviour and attitudes adopted by individual actors in the media and civil society sector, by targeted stakeholders including communities, state actors, internationals bodies and any other individuals and structures that are reached directly or indirectly in the activities through direct implementation and sub-granting; highlight the relevant achievements and lessons, with a view to informing the continuation of the project and/or the development of other following projects. To assess objectively, quantitatively and qualitatively, the relevance, efficiency, and effectiveness of the project interventions during this time, the sustainability of the results achieved, following the international evaluation criteria of the OECD/DAC as well as the internal and external management and coordination of the project. To identify and document good practices / project successes that are recommended for replication to enable scale up/out into new initiatives EVALUATION METHODOLOGY: The evaluation methodology will be proposed by the consultant in his/her technical proposal. The consultant will have to propose two types of robust methodology to: Test the validity of the theory of change and its underlying assumptions, including the analysis of results and outcomes achieved and the logic chain linking interventions to results; Assess the impact, relevance, efficiency, effectiveness, and sustainability of the project interventions, as well as the management of the project including the coherence and coordination within project partners and complementary pro. The AVE project has a narrative proposal, theory of change and results framework. It also has a content assessment tool, and content tracking control, as the project supports large amounts of media content. Regional teams submit reports on a quarterly basis, as well as a monthly update for management. The consultant will combine quantitative and qualitative approaches as necessary to adequately address the evaluation objectives and answer the different evaluation questions mentioned below. The evaluation should also assess the gender sensitivity of the project and whether the different needs of women and men were taken into consideration, this includes the collection of sex-disaggregated data. Preferred qualitative evaluation methodologies, largely utilised by IWPR are: > outcome harvesting - where the OH can’t be applied fully, the consultants will be asked to develop outcome statements for significant results achieved. Note that IWPR has carried out multiple rounds of harvesting to date, with many examples of outcomes documented. > impact stories of change for a selected number of project beneficiaries > stakeholders engagement index - to measure changes in engagement among organisations or networks the consultant may consider (optional) using the IWPR engagement index The consultant(s) will be asked to update a qualitative evaluation database, store relevant impacts/outcomes, lessons learned, recommendations. The database will be available to conduct qualitative database analysis, consultants are welcomed to use it. The proposed methodology should include the following phases: Preparatory phase: review of all project documentation, updated analysis of the project implementation context, preparation of a work plan jointly with the project team; outlining the evaluation questions kickoff call with staff, donor (as appropriate), propose the evaluation methodology and the development of data collection tools and a sampling methodology. As a result of this phase the consultant will have to submit a plan/inception report, outlining the approach. Upon sign of the report the consultant(s) may initiate the work. Fieldwork: Recruit and train local data collectors (if needed). Field test questionnaires with the different target groups through a pilot or pre-survey; conduct data collection, consolidate, validate, triangulate project existing data. Attend/facilitate/lead projects online/in person meetings/workshops to gather data as appropriate. Note any travel requirement should be outlined in the plan and in the agreed overall budget and will be allowed only after appropriate security checks. Analysis of qualitative and quantitative data: triangulated and, if contradictions appear, explore and present hypotheses as to why such contradictions might exist; preliminary findings will be presented to the project team and the evaluation committee, and the consultant will need to take into consideration the project team's feedback. DELIVERABLES: Some key deliverables must be submitted in draft form to IWPR and the evaluation committee before being finalised as indicated in the table below. All deliverables and data from the evaluation will be owned by IWPR and may be used for internal and external reporting and communications. Deliverable 1 Plan Proposed plan and methodologies for the evaluation and case studies development, to be approved by the project manager. Deliverable 2: Conduct online meetings, workshops with IWPR’s 3 regional directors (Ukraine, Central Asia and Eastern Partnership), MEL specialists and field staff and other program key stakeholders (beneficiaries / donor / ambassadors) to capture, document, triangulate results. Deliverable 3: Draft evaluation: submit to the evaluation committee a draft evaluation and related annexes that outline the main findings with related evidence, including recommendations and lessons learned. Visualisations and case studies are encouraged. Deliverable 4 Final evaluation report and summary: in Word format, to be produced in English, should be a maximum of 30 pages (excluding annexes/attachments) and should include the following elements. Summary of the evaluation findings: 2 pages max with overview of overall achievements/performance and key actionable recommendations. Deliverable 4 Evaluation dataset A database (excel) with list of outcomes, significance, contribution and related categorization used for the analysis. REPORTING: A proposed full evaluation structure is outlined in the Annex 1. The final full evaluation report should provide with following information and analysis: Overall assessment of the level of success of the project as per OECD DAC criteria: coherence, efficiency, impact, effectiveness, relevance, sustainability. Outline key actionable recommendations for future programming in the light of current needs (national/regional context), project goal and project successes, good practices and lessons learned. List of significant outcomes achieved, intended as observable changes in actions, relationships, policy or practice of both beneficiaries and external actors linked to the project activities, categorise them using existing and unplanned performance indicators, by DAC criteria and other relevant categories (e.g. gender, type of beneficiary, date etc). Outline significance and relevance of each outcome in light of implementation context at national, regional and international level. Outline contribution of the project for each outcome. Beneficiaries reach: overview of beneficiaries reached directly and indirectly: number, type, gender, country and any other relevant disaggregation. Outline significant challenges and/or negative changes that have limited or affected achievement of success, analyse effectiveness of mitigation strategies applied (if any) and level of impact of those challenges. Outline lessons learned for future implementation. Track progress as per project performance matrix/M&E framework (baseline, targets and year by year progress) Include visualisations (graphs/tables) and quotes as appropriate to support deeper understanding of evidence and analysis. TIMEFRAME: The official start date is expected to be Monday 30th October 2023. The work should be fully completed by Friday 8th December 2023, a total of 6 weeks. Week 1: Signing of contract with consultant(s) Week 1/2- Activity: Consultant Literature review of the project Preparation of the Inception Report, including: A detailed methodology note, a data quality assurance plan, work plans, data collection and data analysis: Evaluation protocol; Development of data collection tools (questionnaires, interview guide, Outcome Harvesting form or other proposed methodologies). Deliverables: Draft inception report; Draft data collection tools. Week 3/4- Activity: Evaluation committee Revision and resubmission and validation of final inception report Deliverables: Final inception report; Finalised data collection tools. Week 3/4- Activity: Consultant Field data collection (qualitative and quantitative). Deliverables: Detailed workplan and operational plan for data collection. Week 3/4- Activity: Consultant(s), Evaluation committee Preliminary analysis of data and presentation of preliminary findings. Deliverables: PowerPoint presentation of preliminary results. Week 4 (end)- Activity: Consultant Submission of first draft of the evaluation report in English. Deliverables: Draft final evaluation report. Week 5- Activity: Consultant(s), Evaluation committee Review of first draft of the evaluation report. Week 6- Activity: Consultant Integration of inputs in 1st draft and submission of 2nd draft. Week 6 (end)- Activity: Consultant Submission of final report in English with all annexes. Deliverables: Final evaluation report in English. Early 2024 (following project close): Donor Approval of evaluation report by donor ETHICS AND DATA PROTECTION: The assessment will be conducted in accordance with IWPR policies on non-discrimination, conflict sensitivity, data protection, safeguarding and ethics. All relevant policies will be communicated to the consultant(s) with their contract. It is expected that the consultant(s) will demonstrate in their proposed methodology how they will address non-discrimination, data protection, safeguarding and ethics, and conflict and gender sensitivity. QUALITY ASSURANCE: Evaluation Committee: An evaluation committee will be established consisting of IWPR staff (IWPR MEL manager, program Director and Manager). Implementing partners and donor will be involved by the committee at the kickoff and completion of the effort to ensure validity and utilisation. The committee will accompany the process and ensure the quality of the analysis and methodology proposed in the inception report, and its implementation after submission of the preliminary results and a draft of the final report. Data quality: Quality data are data that are not systematically biased and do not distort representativeness or coverage. A data quality assurance plan proposed by the consultant will take into account anything that might go wrong with the data collection in advance and develop a strategy to prevent these problems. The plan should be shared in the inception report. This data quality assurance plan will focus on three elements: data collection tool/survey design, field management of data, and high frequency monitoring. KNOWLEDGE, SKILLS, AND ABILITIES: We are looking for a consultant/team of consultants/international or national consultancy firm meeting the following requirements: A minimum of a Master's degree in a relevant field (political science, sociology, international relations, conflict analysis and management and similar disciplines); At least 8 years of professional experience in the above-mentioned fields with proven experience in conducting quantitative and qualitative research, in particular in conducting evaluations of complex, multi-stakeholder projects in the areas of peacebuilding, media, conflict and/or organisational governance and theory-based evaluations; Experience in the evaluation of organisational and individual resilience to disinformation, media literacy, organisation sustainability. Gender expertise should also be a tangible part of the team. Experience with quantitative data analysis software or the ability to subcontract work (if quantitative analysis method is proposed); Excellent writing skills in English. Knowledge of relevant national languages to South Caucasus, Moldova, Ukraine and Central Asia will be considered an advantage; Excellent report writing skills; Knowledge of the socio-cultural, economic and political context of the South Caucasus, Moldova, Ukraine and Central Asia; Knowledge and experience of gender sensitive approaches, including conducting baseline studies or assessments on gender and conflict issues. APPLICATION PROCESS: Applicants interested in this tender may submit to IWPR the following: A letter of interest; Two evaluation reports from a previous project of a similar nature; Up-to-date CVs of the proposed consultants for this evaluation; Two (2) references from organisations for which the consultant or firm has carried out recent evaluations for similar projects (optional) Details of daily fee rate in EUR. Please note: shortlisted candidates will be expected to prepare a short evaluation proposal for this consultancy (maximum 5 pages) if selected for final consideration. This will include a technical proposal including the description of the methodology and a financial proposal (number of days and daily rate). This is not necessary for the initial application process, and will be requested only from shortlisted candidates. Principals only. No calls please. Only those who have been considered for an interview will be contacted. IWPR is an equal opportunities employer and positively encourages applications from suitably qualified and eligible candidates regardless of sex, race, disability, age, sexual orientation, gender reassignment, religion or belief, marital status, veteran status or pregnancy and maternity. IWPR will never ask for payment for recruitment.
