Evaluation of the Medicos del Mundo strategy 2015-2020 in Ukraine

INTRODUCTION

MdM is an independent association that works to make the right to health effective for all people, especially for vulnerable populations or victims of natural disasters, famines, diseases, armed conflicts or political violence. Our projects are carried out both in Spain and in 19 countries in America, Africa, the Middle East and Europe. 

 MdM has been operating in Ukraine since 2015 to improve access and quality of healthcare services for conflict-affected communities in Luhansk -Government Controlled Area (GCA) & Non- Governmental Controlled Area (NGCA)-, and since 2018, Donetsk GCA. Since 2017 ECHO has been funding MdM activities in Ukraine 

 This document establishes the terms of references of the external evaluation of the MdM strategy of intervention in Ukraine 2015-2020.

 STRATEGY CONTEXT, BACKGROUND AND OBJECTIVES

 Millions of people continue to suffer from the ongoing conflict between the Ukrainian government and pro-Russian groups in eastern Ukraine. After more than seven years of armed conflict, the vulnerability of the more than 2 million people in need living in the areas along the Line of Contact (LoC) has grown at an alarming rate. The LoC between the Governmental Controlled Area (GCA) and the Non-Governmental Controlled Area (NGCA) has stabilized, but despite regular attempts to declare ceasefires, armed clashes are still common within 5 km on both sides. The populations in Donbass Region, especially those living close to the LoC, continue to experience significant disruption to their daily lives due to the many challenges created by the ongoing conflict.

 The social and economic effects of the conflict have been exacerbated due to the recent pandemic. Fortunately, from the beginning of the epidemic the number of confirmed COVID-19 cases had not confirmed the initial concerns and the health system has been able to build its capacity, but during winter and spring 2021 the worsening of the situation was in place with the second and third waves of the huge increase in number of confirmed cases followed by high level of hospitalizations and number of deaths (currently Ukraine is in top-10 countries worldwide on the new number of deaths per day). Additionally, the vaccination campaign is showing very slow progress. So as the rest of the world, Ukraine will be affected by the pandemic for quite some more time. Especially in the two conflict affected regions, Donetsk and Luhansk the risk of a potential growing local transmission would have an exacerbated impact due to three contextual factors: (1) ongoing armed conflict, (2) high proportion of elderly and other vulnerable population in the region (including health staff) and (3) deterioration of the health care system as a result of the cumulative impact of the armed conflict and reform processes (I.e., decentralization and health reforms) 

 Since 2015, MdM received funding from ECHO, DAHI, WHO, World Bank, GFFO, UHF, OFDA.

THEORY OF CHANGE OF THE MDM STRATEGY IN UKRAINE:

 The projects implemented by MdM in Ukraine in the period between 2015 and 2020 have had the following general objective: “to reduce suffering, increase and maintain human dignity, and safe lives of the conflict affected population, particularly women and girls, in Luhansk Oblast, Eastern Ukraine.

 To achieve this global objective, in 2018, the MdM Strategy for Ukraine was defined with the following five specific objectives:

1. Improved the equity to access Health care services for conflict affected people
2. Increased quality MHPSS services are available for the population affected by the conflict
3. Reduced incidence and reduced morbidity and mortality caused by HIV, TB and Hep C; and reduced inequity in access to care and inequality in health outcomes
4. Improved access to SRH, FP and Gender responsive health care, human rights and quality of life. 
5. The conflict affected population is free of all forms of gender-based violence and from the threat of such violence

 To achieve these objectives, different actions were implemented: gender-responsive quality mental health and psychosocial services and GBV support services; tackle physical and financial barriers to access primary health care services including SRH and MHPSS; addressing the lack of information on health-related topics of the population in particular in the rural areas; capacity building of established local health authorities and communities; raising awareness as well as advocating towards decision makers in particular regarding MHPSS and GBV identification and response services. 

 MdM provides immediate and direct support to some of the most isolated and affected communities along the LoC while developing the capacity of local health authorities. The Health Outreach teams provides PHC services (incl. MHPSS and SRH), free of charge medication and cash for referrals (transportation and medical examinations at the SHC). Capacity building activities on PHC, GBV and MHPSS are organized for local healthcare personnel and community outreach workers. MHPSS integrated into PHC provision showcase practical examples for further development of comprehensive MHPSS services on community level and integrating mental health into primary healthcare provision. MdM keeps supporting PHC and SHC facilities along the LoC, which benefited from donations, and extending life-saving equipment donations to the major healthcare centers in Luhansk NGCA to address deficiencies in emergency medical service. Distribution of maternity and new-born kits are carried out in LNGCA. In addition, a new project seeks to increase resilience of communities and the local health system in Donetsk and Luhansk region (GCA) by improving availability and access to quality multi-disciplinary PHC services (including SRH and MHPSS) to meet health needs of the conflict affected population with a special focus on vulnerable groups. In COVID-19 context, MdM strives for operational agility in needs assessment, adapted adjustments in delivery of care and support to the healthcare system in COVID-19 response to allow continuity of care for patients and foremost to enable local health systems to cope with additional constraints and the need to overhaul preventive and curative approach. 

 MdM outreach medical teams in locations along the LoC in Luhansk GCA provide comprehensive Primary Health Care (PHC), Sexual and Reproductive Health (SRH) and Mental Health and Psychosocial Support Services (MHPSS). MdM operates through a coordination office in Kyiv, a field base in Sievierodonetsk (Luhansk GCA), a sub-office in Stanitsa Luhanska (Luhansk GCA) which works as a hub for MHPSS activities (also accessible to Luhansk NGCA residents) and a second field base in Bakhmut (Donetsk GCA). MdM operates in Luhansk NGCA and has been legally accredited to operate since October 2017. 

 These interventions have been financed by different donors as ECHO, DAHI, WHO, World Bank, GFFO, UHF, OFDA.

OBJECTIVES AND PURPOSE OF THE EVALUATION

 Almost six years after the start of the Medicos del Mundo intervention in Ukraine, the purpose of this evaluation is to obtain information on the achievement of the planned objectives, as well as recommendations that will be integrated into the new Strategy of the MdM Mission for 2021-2024

Specifically, the evaluation objectives are the following:

1. Review the coherence of the intervention logics that have guided the work of Medicos del Mundo in Ukraine. To do this, both the different strategic documents and the logic followed in practice by the different interventions must be considered.

2. Provide an independent assessment on the fulfillment of the objectives of Medicos del Mundo in Ukraine.

3. Assess the implementation of the different interventions and highlight the main strengths and weaknesses of the approaches, modalities of intervention and human and material resources used by Medicos del Mundo.

4. Assess the mainstreaming of the MdM cross-cutting approaches – gender and age, protection and rights-based approach – in the health interventions in both GCA and NGCA. As well to assess the activities/strategy through the conflict sensitivity loop.

4. Assess the relevance of the COVID-19 response by MdM in Eastern Ukraine including the operational adjustments implemented by the mission. 

Regarding its usefulness, it is expected that this evaluation will provide relevant information and strategic-level and operations-level specific recommendations to Medicos del Mundo for the update of the Strategy in Ukraine and for the adjustment of modalities for humanitarian interventions for the next period 2021-2024.  The results of this evaluation will also be used for accountability to the main funder of Medicos del Mundo in Ukraine, ECHO.

EVALUATION OBJECT AND SCOPE

 The object of this evaluation is MdM’s interventions (implemented between 2015-2020) and the overall MdM strategy in Ukraine these programs are aligned with.

 Regarding the geographical area, the program has been implemented in communities situated in Luhansk oblast, both GCA and NGCA.

 Annex I present a preliminary list of the main actors involved in the evaluation process.

 Annex II lists some of the main reference documents for the Program.

EVALUATION CRITERIA AND QUESTIONS

 The evaluation will have two levels of analysis: a global level, with a strategic view of the work carried out by MdM in Ukraine, and an operational level, with the perspective of the set of interventions implemented during the 2015-2020 period.

 The evaluation should answer the following criteria and questions:

RELEVANCE:

• Are the strategic/interventions objectives based on quality need assessment, including gender and age analysis, risk assessments, socio cultural analysis?
• To what extent the strategy/interventions (including specifically for PHC, SRH, MHPSS thematic) are responding to the identified needs/gaps defined by the national strategies and the Humanitarian needs overview?
• To what extent the strategic objectives are relevant to the needs and priorities of beneficiaries?
• To what extent are the strategy / interventions supported by a clear and coherent theory of change? Have the different levels of objectives/results been clearly identified? Does the hierarchy of objectives respond to a logic of cause-effect relationships?
• To what extent are the strategic/interventions objectives still valid? To what extent is the strategy compatible with the emerging needs/change in context?
• To what extent is MdM responding to needs of beneficiaries related to the COVID-19 outbreak?
• To what extent are the implementation modalities considering the current institutional and administrative framework in Eastern Ukraine and recent reforms (health, decentralization) implemented on the national level?

EFFECTIVENESS:

• What has been the progress made towards the achievement of the expected results and objectives? 
• Has the interventions (including PHC, SRH, MHPSS components) reached those who are the most vulnerable and at the right time?
• What are the good practices and the obstacles or shortcomings encountered? How were they overcome? 
• Have the interventions with different thematic such as PHC, SRH and MH been effective in supporting the implementation of the regional expressed/assessed needs?
• To what extent have the interventions helped the health/social institution to address the increase in risks/disparities caused by the current COVID-19 context?

EFFICIENCY:

• How the program implementation affected efficiency of delivery so far?
• Has there been a cost-efficient operational management (i.e.: organizational structure, funding, logistics…) to maximize results?
• To what extend the quality of proposed activities in NGCA is in line with MdM good practice and standards?  
• Is the program output data efficiently used for national\regional advocacy? 
• To what extent the program is possible to be continued/adapted in terms of its implementation in the current COVID-19 context?

 SUSTAINABILITY

• To what extend responsibilities and/or delivery of services have been transferred to key national partners or is it feasible to be done with the current strategy? 
• How likely are the objectives reached under the MdM interventions to continue benefiting the vulnerable groups in the long term?
• Are the local partners/health authorities have adequate capacities and/or resources to continue the provision of the health services? To what extend MdM built capacity of local partners in a sustainable way?

IMPACT:

• Have the interventions achieved any unforeseen results so far, either positive or negative? For whom?
• Are there risks and what are they to cause negative impact in the long run?

RESULTS BASE APPROACH:

• Are the indicators, intended to gauge progress towards the expected results, of good quality (according to the MdM standards)?
• To what extent have been the interventions’ monitoring mechanisms effective for measuring and informing management of project performance and progress towards targets?

COMUNNITY BASE APPROACH:

• To what extent the interventions have been designed/implemented following the community-based principles?
• Were local communities involved in the design/implementation/monitoring/evaluation of the interventions? To what extent key national partners have been involved in the strategy/interventions’ conceptualization and design process? 
• Are the interventions outputs shared openly with the local stakeholders?

INNOVATION

• To assess if the strategy brought innovative solutions to reach the most vulnerable.
• To assess if the innovative elements are scalable? What were expected /unexpected impact produced by these innovations?

CROSS-CUTTING ISSUES

• How have cross-cutting approaches been integrated in the design and implementation of MdM interventions in Ukraine?

1. EVALUATION APPROACH

The evaluation should be based on a clearly established methodology with a sound design. In order to articulate in a systematic and logical way the judgment on the object of this evaluation it is expected that the evaluation team will develop an evaluation matrix, which will relate to the questions above (and refine them as needed), the criteria for evaluating them, the indicators and the means for verification as a tool for the evaluation. Final evaluation matrix with revised and refined evaluation questions will be validated and approved in the evaluation inception report.

 The evaluators will reconstruct the theory of change that correctly reflects the objectives of Medicos del Mundo Strategy in Ukraine, the lines of intervention and the main actions carried out to achieve these objectives.

 In its proposal, the evaluation team must ensure the complementarity of methods and sources of information, highlight the existing limitations, and specify how and to what extent the cross-cutting approaches contemplated in the Evaluation Framework of MdM (gender approach, anthropologic approach and environmental approach) will be included in the analysis. 

 It is the responsibility of the evaluation team to present in its proposal the theoretical-methodological framework appropriate to the purpose, objectives and expected utility of the evaluation (adjusted to the time and resources available to carry it out) as well as a coherent approach that allows operationalization of the evaluation criteria and questions in the evaluation matrix.

1. EVALUATION MANAGEMENT

 The evaluation will be carried out ensuring the participation of the main stakeholders involved. The structure of the evaluation will be made up of an Evaluation Monitoring Committee made up of the Quality, Evaluation and Internal Audit Unit, the MdM Mission in Ukraine, and the International Operations Department at headquarters, with the following responsibilities:

• Approve the terms of reference
• Evaluate the offers for the selection of the evaluation team.
• Participate in constant feedback on the evaluation process, receive the products delivered by the evaluation team, and send their comments and suggestions to the Quality Unit.
• Validate the preparatory report and approve the different products of the evaluation
• Provide the evaluation team with contacts and access to all relevant information for the evaluation.
• Contribute to the dissemination of the results of the evaluation. 
• Facilitate the use of the evaluation recommendations.

The Quality, Evaluation and Internal Audit Unit will be responsible of.

• Design the Terms of Reference in consultation with the Evaluation Monitoring Committee.
• Advise and methodologically supervise the evaluation process.
• Maintain a permanent dialogue with the evaluation team and with the Monitoring Committee.
• Compile the comments on the different evaluation products and communicate them to the evaluation team.
• Carry out quality control and ensure compliance with deadlines.

 PHASES OF THE EVALUATION AND REQUIRED DELIVERABLES

 The schedule for the evaluation process has been estimated for an approximate period of 14 weeks from the formalization of the contract and will adhere to the following distribution of phases, tasks, products, and length:

 1. Inception phase

• Initial meeting of the evaluation team with the Management Committee, first compilation of information and adjustment of the evaluation order.
• Identification and analysis of the documentation related to the Program
• Request and examination of additional information

 2 weeks

II. Field phase

• Field work in Ukraine: application of methodological techniques and tools designed for the collection of information

3 weeks

Product of Phase II: Presentation of preliminary findings. Once the field phase is over, the evaluation team will hold a presentation session in which the activities carried out and the preliminary findings will be described, which will be discussed among the members of the Management Committee.

Phase III. Analysis and interpretation of the information, preparation and presentation of the final report

• In-depth analysis and interpretation of the information collected.
• Drafting of a first draft of the final report, which will include the corresponding annexes.
• Integration of observations and comments to the report. The evaluation team will explain how these observations have been incorporated into the document and will present, where appropriate, the arguments that recommend their not being considered, safeguarding at all times the independence of the evaluation, in correspondence with the CAD evaluation standards of the OECD.
• Preparation of the final report.
• Presentation of the final report

3 weeks

 Products of Phase III: Final Report

It will consist of the following documents:

- The final report itself (maximum 70 pages without annexes), which must respond to the objectives of the evaluation and answer the evaluation questions.

- A list of annexes, which will include, among other documents, the detailed methodology, the list of techniques and instruments applied and a list of the main sources of information.

- A synthetic report (maximum 15 pages), which will be conceived in terms of its format and language for dissemination purposes and which will include, at least: an introduction to the object of evaluation and the methodology used, a synthesis of the main findings, a synthesis of the conclusions and a synthesis of the recommendations. The synthetic report should be prepared once the final report has been approved.

-A presentation of the evaluation results by the evaluation team. The evaluation team may also be required to participate in other activities for the presentation and return of results in Spain or Ukraine

All products will be delivered in digital format in Word to the Quality and Evaluation Unit which will be in charge of their distribution to the Monitoring Committee.

A guide content structure is proposed below for the final evaluation report. However, the index and the final outline of the report will be agreed between the Monitoring Committee and the evaluation team in the last phase of the process:

1. Introduction

1.1. Structure of the report.

1.2. Background, scope and objectives of the evaluation.

1.3. Questions and levels of analysis of the evaluation.

1.4. Theoretical-methodological framework.

1.5. Data collection and analysis process.

1.6. Conditioning factors and limits of the evaluation.

2. Analysis and interpretation of the data (findings).

3. Conclusions

4. Lessons learned

5. Recommendations (strategic and operational).

Annexes.

1.  EVALUATION PRINCIPLES 

 The evaluation will be guided by the OECD DAC Quality Standards for Development Evaluation and the Medecins du Monde Evaluation and Monitoring Framework. Throughout the evaluation process, the evaluation team must observe the following principles:

INDEPENDENCE: The monitoring and evaluation processes are expected to describe the strengths and weaknesses of the interventions truthfully and honestly. Therefore, evaluation should be independent to ensure credibility and maximize the benefits they can provide to the organization.

IMPARTIALITY AND ETHICS: To ensure impartiality, it must be guaranteed that the evaluation team is selected with due concurrence, have knowledge and sufficient experience for the performance of the tasks entrusted and are free from biases. In turn, the members of the units and evaluation teams must communicate previously possible conflicts of interest, they will provide respectful treatment and not discriminatory to all people and groups involved in the evaluation processes, and perform their duties with integrity and honesty

TRANSPARENCY: To guarantee the transparency of the evaluation function, it is required that the external evaluations are hired based on clear criteria and procedures and previously established, and that the main products of the evaluations are disseminated to all stakeholders through appropriate means.

CONFIDENTIALITY: During the development of the evaluation, respect must be guaranteed to privacy and the adequate treatment of personal data. Especially in contexts conflict or risk to personal integrity, measures will be taken to avoid possible

Damages derived from the identification of the people who participate in the evaluations.

CREDIBILITY: For the evaluation products to be credible and assumable by the different actors involved, evaluations should respond to the needs of information, must be carried out at the appropriate times, and must be carried out in a systematic and methodologically robust way such that the findings and recommendations are well founded. The evaluation teams will receive cooperation timely and access to all available information that is necessary to the exercise of their work. Doctors of the World is committed to ensuring evaluations are carried out according to international standards and based on reliable data and observations.

UTILITY: Evaluation should satisfy the information needs of the different audiences. In order to increase appropriation and the use of the results, the incorporation of the actors involved in the governance structures of the evaluations will be promoted, and the communication of findings and recommendations will be made in a clear, concise and concrete language, using formats adapted to different audiences

 REQUIRED SKILLS AND COMPETENCIES

 Education:

- Advanced degree in public health and evaluation is an asset.

Professional experience:

- At least 7 years’ experience in program evaluation in a development and humanitarian context and proven accomplishment in undertaking evaluations, including leading evaluations of multi-stakeholder programs. 

Knowledge and skills

- Knowledge in results-based programming

- Proven expertise in evaluating humanitarian health and mental health programs.

- Extensive knowledge of qualitative and quantitative evaluation methods

- Excellent written and spoken English and presentational capacities. Russian will be an asset

- Excellent inter-personal skills and communication skills

- Knowledge of the context of Ukraine is desirable.

The evaluator must explicitly declare his/her independence from any organizations that have been involved in designing, executing or advising any aspect of this program.

 The evaluation team responsible for this evaluation must be made up of people with demonstrable capacity, experience, and professional qualification to carry out an evaluation of this nature.

 It is recommended that the composition of the team be balanced between men and women.

 The person responsible for the evaluation team will always act as interlocutor and representative before the Management Committee.

 Annex I 

Preliminary list of the main actors involved in the evaluation process

 As the preliminary list of the key groups to be involved in the process of evaluation kindly consider next:

1. MdM mission in Ukraine:

• Coordination team
• Field office program teams
• STL sub-office
• NGCA team
• Admin and Logistic teams

1. State and local authorities:

• PMSACs in Popasna and Stanytsa Luhanska (Luhanska GCA)
• Health Department in Luhanska oblast GCA
• Targeted communities in Popasna and Stanytsa Luhanska ex-rayons (Luhanska GCA)
• Department of social protection of Luhansk oblast and of the STL district

1. Humanitarian community:

• ECHO Kyiv representative
• Health Cluster in Ukraine (including MHPSS working group)
• GBV sub-cluster
• PIN/ACTED / ACCESS Consortium
• WHO
• OCHA (GCA and NGCA representatives)
• Eventually to include other partners (PUI, ICRC, NRC)

Annex II

Reference documents

 As the reference documents kindly use next:

1. MdM Ukraine Strategy 2018 (to be shared based on demand)
2. Inter-sectorial documents:
3. HNO/HRP for 2015-2020
4. COVID-19 specific HRP, Ukraine 2020
5. REACH platform for Ukraine (both GCA and NGCA)
6. APLICATIONS PROCEDURE

 Proposals must be submitted to submit the following to Отправить резюме and 

Отправить резюме

 Deadline for Application: before midnight on 11 July 2021.

The Technical Proposal must include:

 Detailed methodological proposal.

• Workplan.
• Times of dedication to the evaluation of each of the members of the evaluation team, as well as the formal commitment to belong to the evaluation team during the term of the contract. Any change in the composition of the evaluation team must be previously agreed with the Management Committee.
• Experience in evaluation.

The criteria for evaluating the proposals submitted will be the following:

1. Profile of the evaluation team: maximum 50 points

2. Quality of the technical proposal: maximum 50 points