Evaluate

Powered by mxGraph

Introduction

Evaluation is an important activity to be carried out in SSR projects, programmes and policies. As defined by the OECD, an evaluation is "the systematic and objective assessment of an on-going or completed project, programme or policy, its design, implementation and results. The aim is to determine the relevance and fulfillment of objectives, development efficiency, effectiveness, impact and sustainability. An evaluation should provide information that is credible and useful, enabling the incorporation of lessons learned into the decision-making process of both recipients and donors." OECD (2002),Glossary of Key Terms in Evaluation and Results Based Management.

In this section, you will find a useful and operational methodology to undertake both mid-term/on-going/formative evaluations and ex-post/summative evaluations of individual SSR projects, complex SSR programmes and/or SSR policies.

This methodology is not appropriate for ex-ante prospective evaluations/assessments or for the design and implementation of a monitoring systems within a project/progamme.

Throughout this methodology, specific evaluation terms are used and are underlined. By mousing over these terms, their definition will appear. They all come from the OECD (2002),Glossary of Key Terms in Evaluation and Results Based Management.

Preliminary remarks on the specificities of evaluating SSR

A Saferworld study on evaluating SSR and the OECD SSR handbook, section 10 on evaluation and monitoring, highlight a number of specificities/challenges for evaluating SSR. Some are common to all development aid evaluation exercises, and some are more specific to SSR. Here below are the most relevant for this OGN. Further details can be found in the above-mentioned documents.Preliminary remarks on the specificities of evaluating SSR

  • SSR is often undertaken in fragile and conflict-affected environments:
    • Lack of data
    • Rapidly changing and insecure situation
    • Limited design and bad planning due to time pressure
    • Important to ensure conflict-sensitivity of the evaluation process and results
  • SSR has a multi-sectoral and integrated/holistic dimension:
    • The performance of one sub-sector is influenced by others. It is thus a challenge for attributing the performance when evaluating only one sub-sector.
    • Evaluating a whole-of-government approach and/or an integrated approach requires specific methodology
  • SSR is highly politically sensitive and has a culture of secrecy:
    • Difficult access to data (documents but also through interviews)
    • Official objectives may be different than unstated goals
  • SSR actors (e.g. soldiers, policemen, lawyers, judges and intelligence officers) often have limited understanding of evaluation processes:
    • Hostility towards evaluation processes
    • Difficult to reach agreement on what to measure and how
    • Difficult to integrate the lessons learned
    • Limited numbers of specialist SSR evaluators
  • Local ownership of SSR and of the evaluation process is crucial:
    • Difficult to evaluate local ownership
    • Results of the evaluation process have to be locally owned. Evaluation findings should be shared with local partners and recommendations adapted to their needs.

The evaluation methodology laid out above is a useful tool to tackle most of these issues. It offers a structured approach to deal with these difficulties. It allows informing SSR actors on the evaluation process in a simple manner and get them involved in each step of the methodology. And it provides evidenced-based findings and recommendations which are key for the success of your evaluation. Of course, the situation on the ground is different from the theory. The activities, sub-activities and tools provided here should thus be adapted to each evaluation mandate.

Please click on "Start Prezi" in the middle of the box below to view a presentation of the evaluation methodology. Use the arrows that will appear at the bottom of the box to move forward and backward through the presentation. If you want to see the presentation in full screen, click on the full screen icon at the bottom right hand corner.

For access to the Prezi Presentation, ISSAT Evaluation Methodology, kindly follow the link.  

Be the first to add a comment!

Short guide to Evaluate

Sub-activities

1. Plan the evaluation

What is it?

Planning is the first activity of an evaluation. The main actor of this phase is the evaluation team leader (or the ISSAT advisor dealing with the mandator’s request, if the team leader is yet to be identified).

NB: This activity and all the sub-activities are applicable for all types of evaluations (mid-term, ex-post) and interventions (project, programme, policy).

The following sub-activities should be carried out in parallel:

  • Support drafting of the Terms of References (ToR);
  • Compose the team and define the responsibilities of each expert;
  • Organize the work, detail the task, allocate an indicative number of working days per expert and formulate this into a work plan, ensuring realistic timeframe that allows for contingencies;
  • Develop the evaluation approach;
  • Hold an evaluation team kick-off meeting as soon as possible;
  • Start with the logistics for the field mission, if applicable.

At the end of this activity, the output should be an official written approval of the ToR. Moreover, the evaluation team should be composed, the budget decided and the work plan finalised.

Why is it important?

Planning is crucial for an evaluation exercise. It allows starting the evaluation on solid grounds with a clear mandate and objective, an adequate team and a realistic work plan. Evaluations can be sensitive, difficult to manage and long undertakings. Knowing where you are going and how you are going to get there allow you to manage the resources and the expectations efficiently.  

How do we do it?

  • Support drafting of the terms of reference: Engage with the mandator on the basis of the initial request and the initial version of the ToR.
  • Form the evaluation team: Choose the evaluation team leader and the rest of the evaluation team carefully. Have a balanced team with appropriate thematic, evaluation and local expertise. 
  • Plan the work: Detail to the maximum extent possible all the tasks to be carried out and allocate a number of working days to each task and per expert.
  • Develop the evaluation approach: Explain the evaluation approach by detailing the chain of reasoning from the building of the evaluation framework, to the collection of data and the analysis in order to formulate credible findings and useful recommendations. 
  • Hold team kick-off meeting: Organize a team kick-off meeting with all the members of the evaluation team (including support staff and local expert if possible) to discuss the ToR, the responsibilities of each member of the evaluation team and the work plan. 
  • Initiate logistics for field mission: Address all logistical aspects for the field phase, if applicable. This includes: booking flights, hotel, finance, insurance…

To help you go through these steps with success, you may want to carry out a “recce”/scoping mission.


2. Build the evaluation framework

What is it?

This is the second activity of an evaluation. The main actors are all the members of the evaluation team, headed by the team leader.The aim is to build the framework of the evaluation exercise that will accompany you throughout the whole process.

The following sub-activities should be carried out:

  • Analyze the subject of your evaluation: the country context (especially political and security context), the project, the programme and/or the policy.
  • Draft the project/programme/policy logic.
  • Draft and/or structure your evaluation questions in sub-questions and indicators.  
  • Prepare the field phase.

NB: This activity is applicable for all types of evaluations (mid-term, ex-post) and interventions (project, programme, policy). However, all the sub-activities are not applicable for all types of evaluations and interventions. For ex-post evaluations of complex programmes and policies, it is recommended to follow all the sub-activities in depth. For mid-term evaluations of simple projects, the main sub-activities are the logic, the evaluation questions and field phase preparation.

At the end of this phase, the output should be the validation of the evaluation framework by the mandator. It can take the form of an inception report or a meeting presenting the framework. Ideally, a written agreement from the mandator should be required.

NB: when the evaluation is already well structured by the mandator in the ToR (i.e. the context, logic, evaluation questions and approach are already well described), ensure that what is asked is feasible and realistic (one solution is to go through the work plan). Ensure that the entire evaluation team and main stakeholders have the same understanding of the entire approach.

Why is it important?

This activity is important because it provides you with a framework in which you will progress throughout the evaluation. Evaluation is not just about gathering some data and expressing an opinion on the performance of the project/programme/policy. It is a process that needs structure, rigor and methods to ask the relevant questions, collect the data needed and triangulate them, in order to express objective findings and provide useful recommendations adapted to the context and the needs.

With a structured evaluation exercise, you can easily justify your findings and recommendations with strong arguments. They are not just based on your opinions and expertise but on facts and evidence collected through robust tools and relevant questions.

Building that framework also structures the work of the team members. Experts can be the best in their field of expertise, they still need guidance to carry out an evaluation and to ensure the collection of useful data. At the end of the day, this will be crucial to ensure the coherence of the final report and the drafting of appropriate recommendations.

In the field of SSR with highly political issues and conflict-sensitivity, it is crucial to have a well structured approach that helps you justify your findings and recommendations on a solid basis. It helps you manage criticism and sensitivities. It allows the team to keep its distance from the perceptions of a donor-driven/political evaluation mandate and to undertake an evaluation based on an objective and a mutually reinforcing process. 

How do we do it?


3. Collect data in the field

What is it?

This is the third activity of an evaluation. The main actors are all the members of the evaluation team, led by the team leader. The aim is to collect data for building the answers to the Evaluation Questions (EQ) defined in the evaluation framework.

The following sub-activities should be carried out in the field:

  • Organise and hold a briefing with the entire evaluation team;
  • Organise and hold a briefing with the mandator and other stakeholders (e.g. national counterparts);
  • Collect data using various evaluation data collection tools;
  • Debrief with the team each day and write the preliminary findings by EQ;
  • Debrief with the mandator and other stakeholders on the preliminary findings.

At the end of this activity, the output should be a presentation of the preliminary findings by EQ to the mandator and other stakeholders in the field. They should be able to comment on these preliminary findings in order to rectify and complement them if necessary.

NB: in an evaluation process, going to the field can be seen as a tool, among others, to collect data. Some evaluations do not have field missions but carry out their data collection only on the basis of desk work. However, for ISSAT evaluation requests, the field mission is crucial. Therefore, it is considered as an activity in itself along with the three others within which several sub-activities should be followed.

NB: This activity and all the sub-activities are applicable for all types of evaluations (mid-term, ex-post, etc.) and interventions (project, programme, policy).

Why is it important?

Going to the field to collect data is crucial in a SSR evaluation. An evaluation team needs to take the measure of the work being done on the ground, understand the dynamics between the SSR programme and the beneficiaries, ask questions to the ultimate beneficiaries and see for itself the results of the project/programme/policy.

Presenting the preliminary findings at the end of the field mission is important in order to obtain feedback from the main stakeholders and to manage their expectations for the final report. They should not be surprised by the evaluation output at the end of the process.

How do we do it?

Field missions will be different according to each request. The following should thus be adapted to the context, the time available, the number of experts involved and the main stakeholders' role (donor and beneficiaries).


4. Analyse data and formulate findings and recommendations

What is it?

This is the fourth activity of an evaluation process. The main actors are the Team Leader and all the team members. The aim is to sum up all the information collected throughout the evaluation process and formulate objective and well structured findings according to the framework set up at the start of the evaluation. Those findings will support the recommendations that will be included in the final report.

The following sub-activities should be carried out:

  • Analyse the data collected throughout the previous evaluation phases.
  • Draft your findings by Evaluation Questions (EQ).
  • Formulate your recommendations.

At the end of this activity, the output should be i) the draft version of the final report and ii) the final version of the final report once the mandator and other main stakeholders (if appropriate) have had an opportunity to comment on the draft version. A validation phase can be also envisaged with the mandator and other stakeholders before the communication of the main findings and recommendations.

NB: This activity and all the sub-activity are applicable for all types of evaluations (mid-term, ex-post, etc) and interventions (project, programme, policy).

Why is it important?

This activity is important because it allows you to step back from all the information you have collected in the previous activities. It allows the analysis of the data collected in order to extract the main findings and to avoid providing only facts in the report. It is also important in order to draft the final report and communicate the results of your evaluation with some detachment from the field mission.

How do we do it?

SUB-ACTIVITY 1: Analyse the data collected.

SUB-ACTIVITY 2: Draft the findings by Evaluation Question.

SUB-ACTIVITY 3: Formulate the recommendations and draft the final report.

Here below is a figure that explains the “analysis chain” that we suggest in this methodology from the data collected to the recommendations. And how you can go back and justify your recommendations till the data collected. The boxes on the left side of the figure are the actions that we are carrying out to move from one layer to the other above.

  


5. Involve the mandator

What is it?

As stated in the OECD Glossary of key terms in evaluation, “an evaluation should provide information that is credible and useful, enabling the incorporation of lessons learned into the decision–making process of both recipients and donors”. Involving the recipients in an evaluation exercise is explained in the fiche on how to engage with local stakeholders. On the donor’s side, a number of actions throughout the evaluation exercise should be taken by the evaluation team in order to ensure that the findings and recommendations of the evaluation will be taken on board by the mandator and that it will be useful for improving the project/programme/policy’s performance.

Involving the mandator means involving all the main stakeholders of the project/programme on the donor’s side that is:

  • the staff managing the project/programme in the partner country,
  • the staff at HQ in the different services involved in SSR (at policy and project/programme level) and in the partner country for other related issues, e.g: staff from defence, foreign affairs, development cooperation, interior,and
  • the staff involved in evaluation within the organization, e.g. members of the evaluation unit/department.

NB: When ISSAT experts are part of a mandator team or when mandator staff are included in the evaluation team, the involvement is straight forward and there is a de facto interaction with the mandator. However, it is still important to implement some key actions in order to ensure the same level of involvement of all the key stakeholders on the mandator side (staff at HQ and in the field not involved in the evaluation team).

Why is it important?

  • To ensure the evaluation usefulness by adapting the evaluation approach and the Evaluation Questions to the mandator’s needs.
  • To create a climate of trust and not of rivalry. The evaluation should be perceived as an opportunity for improvement and not for control.
  • To ease the process of data collection.
  • To manage expectations. The mandator should not be surprised by the findings of the evaluations. S/he should be aware on the progress of the evaluation team perceptions and analysis of the data collected throughout the entire evaluation process. Moreover, s/he should be aware of the limitation encountered during the process.
  • To maintain buy-in and political support of the mandator throughout the evaluation process.

How do we do it?

To ensure the involvement of the mandator, you need to interact with her/him regularly and at least carry out the following actions at each activity of the evaluation:

  • When planning the evaluation:
    • Present the evaluation team (send CV and short biography of each expert) and obtain the mandator’s feedback.
    • Have a face-to-face or telephone meeting to discuss the ToR and the initial evaluation plan.
    • Present the evaluation approach and obtain the mandator’s feedback.
    • If applicable at this phase, define the Evaluation Questions and obtain the mandator’s feedback. It is very important that the mandator understands the Evaluation Questions and the overall evaluation approach because they are the backbone of the entire evaluation process.
    • Obtain an official approval on the ToR
  • When building the evaluation framework:
    • If not applicable in the planning phase, define the Evaluation Questions and obtain the mandator’s feedback. It is very important that the mandator understands the Evaluation Questions and the overall evaluation approach because they are the backbone of the entire evaluation process.
    • Present the lay out of the report and obtain the mandator’s feedback.
    • If applicable, present a report on the evaluation framework, obtain the mandator’s comment and then official approval.
  • When collecting the data in the field:
  • When analysing the data and formulating the recommendations:
    • Present the draft final report and obtain the mandator’s feedback before writing the final version of the final report.
  • Six month after the end evaluation:
    • Have an informal meeting (face-to-face or telephone) with the mandator to discuss the progress of the recommendations’ implementation.

6. Engage with local stakeholders

What is it?

As stated in the OECD Glossary of key terms in evaluation, “an evaluation should provide information that is credible and useful, enabling the incorporation of lessons learned into the decision–making process of both recipients and donors”. Involving the donors in an evaluation exercise is explained in the fiche on how to involve the mandator. On the recipients’ side, a number of actions also need to be taken in order to engage with local stakeholders throughout the evaluation process.

For an evaluation exercise, the most important local stakeholders to engage with throughout the entire process are, but not limited to:

  • The direct counterparts of the project/programme management team (e.g. the civil servants involved in the project/programme and/or the actors implementing the activities such as civil society organisations, parliament committees, etc.); and
  • The government official heading the sector supported by the project/programme (e.g. the Minister of defence, justice, interior, the Head of the police forces, etc.).

NB: The evaluation team will also interact with other local stakeholders for collecting data through, for example, interviews with the direct and indirect beneficiaries of the project/programme that is evaluated. But these stakeholders will not be engaged in the evaluation process as such. They will not, for example, provide comments on the evaluation’s deliverables or be targeted for the implementation of the recommendations.

Why is it important?

  • To ensure buy-in of the evaluation findings and recommendations.
  • To ensure that the findings and recommendations are adapted to the political context and the needs of local stakeholders.
  • To create a climate of trust and not of rivalry. The evaluation should be perceived as an opportunity for improvement and not for control (audit or internal control exist for that purpose).
  • To ease the process of data collection.
  • To raise awareness of evaluation processes and results based management.
  • To increase ownership of the project/programme and of the reform as a whole by encouraging the incorporation of the evaluation’s findings and recommendations into their decision-making process.

How do we do it?

To engage with local stakeholders, you need to interact with them regularly and at least carry out the following actions at each activity of the evaluation:

  • When planning the evaluation:
    • Present the evaluation purpose, the team and the initial evaluation plan.
    • Include them in the evaluation steering committee (if applicable)
    • Explain how the local stakeholders can be involved in the evaluation by:
      • Providing comments on the deliverables (if applicable);
      • Providing access to data (documents, interviews, site visits).
  • When building the evaluation framework:
  • Present the evaluation framework and ask their comments (if applicable). It is very important that the local stakeholders understands the Evaluation Questions and the overall evaluation approach because they are the backbone of the entire evaluation process.
  • When collecting the data collection in the field:
  • When analysing the data and formulating the recommendations:
    • Obtain the comments of the local stakeholders on the draft final report.
    • Send the final version of the final report to the local stakeholders.
  • Six month after the end evaluation:
    • Have an informal meeting (face-to-face or telephone) with the local stakeholders to discuss the progress of the recommendations’ implementation.

Resources