Monitoring and Evaluating Progress and Results

All the outputs foreseen in the ToR were achieved and the mandator was overall very happy with the work carried out by the review team. Some outcomes were also achieved (see LI form in annex here below) but some others were only partially achieved such as the clarification on the future scope of NORLAM after 2016, the status of a police component and the length of extension of NORLAM. It was recognised however that the ToR were very ambitious and Norway wanted to achieve too much in one review over a short period of time. Prioritisation of the expected outcomes could have been better and communication between ISSAT and Norway when drafting the ToR could have been more intense. The Oslo mission carried out during the planning phase of this mandate was very useful, but it took place a bit too late in the process. The ToR were already approved. However, amendments were still made to the ToR after the Oslo mission which were considered useful.

To help prioritise the ToR, the review team could have also communicated better and earlier on the methodology to be used for the review.

Norway and NORLAM are using the conclusions and the recommendations of the report. It is considered to be very important for them. The report played a major role in the discussion between the Norwegian Ministry of Justice and Ministry of Foreign Affairs to decide whether or not to extend NORLAM. Moreover, the NORLAM team organised an internal seminar in mid-September to review the report. They analysed the findings and conclusions and came up with a list of action points. This exercise was considered to be very useful for discussion and ownership of the report. A second internal seminar should take place soon to discuss more in depth the recommendations. In November, the Norwegian Ministry of Justice will visit Moldova to discuss the post-2016 NORLAM, on the basis, among others, of the conclusions and recommendations of the report.

Some recommendations are however considered to be too vague (e.g. what does it mean concretely to develop NORLAM Theory of Change, cf. Recommendation 4). Some others are difficult to follow such as the one on the legal status of NORLAM, because it needs political agreement and decision that are not easy to obtain. Nevertheless, having such recommendations in a written report, done by an external and international actor such as ISSAT, is considered to be very helpful and can be used as a political leverage for discussion with Moldovan authorities.

Lessons identified on output and outcomes:

• Ensure good communication with the mandator during the drafting of the ToR in order to help prioritise the expected outcomes and the scope of the work.
• Try to undertake a scoping mission earlier in the process so that discussions on the ToR can take place with the mandator. Overall the scoping mission is a very useful step in the planning phase of the mandate.
• Include in the ToR a section on the evaluation methodology to be used during the mandate.
• Develop as much as possible concrete recommendations. General recommendations could be unpacked with concrete actions to be considered by the mandator.
• Continue building ISSAT reputation in terms of being a recognised international actor with strong SSR expertise and evaluation methodology.

• M&E recommended

• The team used an informal evaluation process of the progress of the mission through  actively reserving time to discuss the mission process and approach and propose improvements.

- Even with the inherent weaknesses of the report (mentioned above), it is a good starting point for follow-up activities.

-  Build national capacity to undertake M&E where possible: As part of the programme options that were proposed in the report, a section of the report dealt with both the government ability to monitor and review the reform process (it recommended further investment in building this capacity) while also highlight how the Netherlands could ensure that Monitoring and Evaluation was built into the design of the future programmes. 

- The team was pretty good at monitoring its progress.

Three main interesting approaches regarding support to S&J were discussed during the AAR (the following reflects the main points discussed and, unfortunately, do not explore in depth these approaches because of the limited time allocated for the AAR discussion):

• The ASJP has a balanced approach to central institution capacity building and local level service delivery. This is not an innovative approach in itself, as it has already been applied in various DFID S&J programmes, BUT, the extent to which the balance has been incorporated in the design and the implementation of the programme is rather unique.
  • A key factor was the search for such “equal” balance during the design phase. Indeed, local actors and better service delivery to the local population were not just side aspects of the programme but considered as important as, or even more important than, building the capacity of state institutions. The ASJP support is directed to central administration, local districts, NGO, para legal / legal aid providers, traditional leaders / chiefs, etc.
  • The implementation of this support is also well managed by the programme thanks to the committed team who has technical advisors, a district coordinator and local support staff. This combination of staff enhances the implementation of such an approach.
• The ASJP uses public financial management (PFM) tools to increase S&J effectiveness and accountability.This has mainly been taken forward by an ASJP advisor with a PFM background. The ASJP has then been able to bring on board the Ministry of Finance, which was a great achievement.  Awareness on PFM is growing within security and justice ministries, which facilitates discussions for budget increases. It also enhances the accountability (at least on the financial side) of security and justice actors/institutions, even though this aspect has to be further developed by the ASJP. The involvement of the ASJP with various actors / ministries is considered by the participants in the AAR as a good practice for future support.
• The ASJP has a strong M&E component for measuring the programme performance and developing local partners M&E capacity. This has mainly been taken forward by the M&E advisor within ASJP. The M&E tasks are not only aimed at measuring the progress of the programme according to the logframe, but also have a strong focus on building the M&E capacity of local partners. This last point is crucial because without good data from the partners, it is harder to have a good M&E system for the programme itself. Although the approach is very sound, a lot of challenges are still present. For example, a lot of time is still invested in the programme logframe and indicators, which detracts from the time spent on building local M&E capacity; good data are still very difficult to collect which hinders the measurement of the progress; there is a need for more M&E advisors who understand the S&J sector, but they are difficult to find. 

The outputs of the mandate (Annual Review narrative report and report as per DFID template) were delivered in time and with quality. Considering the very short time allocated to this mandate, the mandator (DFID SL) and the main recipient of the review (the implementing partner) were satisfied with the reports.

In terms of outcomes, they were not explicitly mentioned in the ToR. However, the AAR discussion moved towards looking at whether the recommendations already had an effect on the main recipients of the Annual Review. Looking at the outcomes in those terms, it was a bit too early to assess them, but DFID SL already reported a positive shift in the relationship between the implementing partners and senior level staff from the government of SL (which was one of the recommendation made by the review team).

Lessons identified for improving the chances of implementing the recommendations:

• The “exit” presentation of the review team at the end of the field mission should be longer than two hours. Ideally, it should take the form of a workshop/brainstorming of one day with the main recipients of the Annual Review and the review team.  It should focus on the recommendations and concrete options for implementing them.
   
• The recommendations should be more specific and actionable.
   
• The review team should work more closely with the implementation team so that they can learn from the review team’s experience and expertise.
   
• (from ISSAT’s own experience: if possible, plan a follow-up field mission one or two months after the submission of the (draft) report to work specifically on the recommendations with the implementation team).

• Informal Evaluation: This mission’s success and impact is going to measured informally by ISSAT through the following three indicators:
  • Whether the mandator will continue to use ISSAT
  • Whether the evaluations recommendations were implemented
  • Feedback form reflects positive feedback on ISSAT Advisors’ performance

• Necessity to evaluate ISSAT engagements in countries where there has been medium to long term engagement for ISSAT: ISSAT has been involved in Burundi for the past two years with 5 missions being deployed to support various donor programmes. An evaluation of ISSAT’s contribution and impact on those donors and their programmes would be a positive step to undertake.

• Because of the openness of the participants in the workshop it was relatively straightforward for the facilitators to informally evaluate the value of the workshop to the participants.
• The workshop appeared to achieve its objectives and both participants and organizers noted their satisfaction, although the formal evaluation has not yet been finalized.

• The evaluation questionnaires during the training for participants used a lot of open ended questions, which will take time to review;
• It will be challenging to evaluate the mission results (both output & outcome). The plan is to contact the participants after 6 months to a year to check the results.

• At the end of each day, the training team sat together with the facilitators, in order to check how the day went, and what needed to be changed/ modified for the upcoming period;
• As it was the 2^nd pilot course, we did a lot of evaluations during the training (including questionnaires to participants) – especially related to the newly developed sessions.

• It might be good to involve the local POC even more in the monitoring/evaluation of the training when it is ongoing;
• More thinking is necessary on how to evaluate the results of the training;

• Suggestions for evaluation: peer feedback; participant feedback; video recording
• Co-facilitation presentation techniques: Co-facilitation can give real time feedback if facilitators have visual and word queus ready in advance 

• Proactive Monitoring of Participants: The advisor had the phone numbers of all workshop participants. This allowed him to send text messages mid-way through the independent work week to check how participants were progressing and reminding them they could receive assistance if they were having problems.
• Independent Work as a Form of Evaluation: The interest and commitment to the project by local NGOs could, in part, be gauged by how many participants returned to the final workshop to present their work.

ISSAT Long Term Focus: The decisions for this assessment, at the time of writing had yet to be taken and their possible implementation will be further down the line. ISSAT needs systematic follow up with its Governing Board members in order to realise the impact it is having.

• Work plan: The team’s work plan was an effective tool to review at the end of each day for monitoring purposes.
• Informal Quality Review: To supplement the review of the work plan, informal conversations were held with key stakeholders to judge the satisfaction of work undertaken and make adjustments accordingly.

Actions and words: The evaluation of impact is through a review of updated action plans for the military and police. The mission will be successful when we are able to observe evidence of SSR concepts from the workshop being incorporated into the updates. While this is one indicator that the concepts were learned, additional indicators need to be established to see if stakeholders can implement these concepts.

• Need for Clarity on Mission Purpose and Tasks: The relatively short lead time prior to mission commencement prevented a more thorough review as to how the different stages of the mission might proceed in practice. The Terms of Reference were detailed but discussions around more practical aspects of the work, as well as potential challenges throughout the project, would also be useful. Greater clarity around the nature of the assessment process itself would be important, particularly in terms of the mission and subsequent report being single or iterative ; this has a significant impact on the type of assessment process and resulting reports, and on methods for data collection and verification employed. Reporting methods are also impacted, as  single reports have entirely different structures and objectives than repeated reports. With respect to the final report, greater clarity regarding ultimate authorship would assist the drafting process, particularly with respect to content and tone. In addition, with several authors and draft versions in circulation, it is critical to agree a 'master' document that is the only version to be amended. 

• Comprehensive Assessment List: The interview list was thorough and allowed team members to gain a comprehensive sense of circumstances, progress and problems once on the ground.
• Team Composition: The collective expertise and knowledge represented by the assessment team helped to address the challenges noted earlier by allowing the overall approach to be modified, strengthened or re-sequenced as necessary and as work on the ground progressed.

• Determine Place of a Project in a Larger Programme: Confirm the nature of the overall assessment process and determine if possible the degree to which assessment results will inform future measurement frameworks and expectations. If an iterative role is intended then it is useful to know how many potential reports are required. This would allow the team to develop robust means of data gathering, verification and analysis.

Mitigation Strategies: Overall from Geneva and the ground ISSAT members have been capable of developing good mitigation strategies to address the challenges along the way.

• Untimely replacement of officers who end their tour of duty
• Unarmed police officers (self protection)
• The lack of expertise of police officers has an impact on the full capacity to deliver on:
  a. Reform and Restructuring
  b. DDR
  c. Mentoring, Coaching, Advising
  d. Training

• UN Technical Assessment Mission (TAM), came in twice. Their findings and recommendations were very useful for the improvement of the delivery of the mandate and the strengthening of UNPOL's delivery. We ultimatly were able to increase our personnel by 100 advisors and five Formed Police Units (650 officers). We started the mission on 4 April 2004 with an authorised strength of 350. In February 2006 the authorized strength was 450 advisors and five FPUs
• Turnover: The team that started the mission stayed throughout my mission; this allowed for a fluent follow-up in ideas and the strategy, due to a proper understanding of the forces in place

• None

• No monitoring and evaluation was done

• No monitoring and evaluation was done

• No monitoring and evaluation was done

• At the same time as the commission completed its report and presented its findings and recommendations to the intelligence minister, the minister resigned in a major government shake-up occasioned by the ruling party requesting President Mbeki to step down. The new intelligence minister was not reform-minded. He prioritised intelligence effectiveness over compliance with the constitution. He refused to present the commission's report to cabinet and parliament, and he has ignored the commission's recommendations.

None

None

• Information Coordination: When dealing with with multiple actors in one project it is important to have an integrated and centralised filing system.

None

• Reflect on Methodology: When an evaluation mission will undertake multiple deployments, be sure to review your own methodology to fine tune for future deployments in order to maximise efficiency. In this case ISSAT needs to reassess the methods it used to execute the first ISSAT team visit to the DRC for this DFID project, in order to enhance efficiency of the visit and it should agree on a clear procedure for reporting back during and after the team visits.

• Engagement in the whole program development process: This mission served to better familiarize the team with the capacity of the National Assembly so as to position the organization to respond to an upcoming call for proposals on parliamentary strengthening. While I believe my report was useful for the proposal (I was told so by the lead drafter), I was not involved in the process after the mission. By continuing to engage me in the program development process, they could have benefitted both from my expertise on parliaments, but more importantly on governance programming in that country.
• Maximise experts' contributions: Aside from daily notes, I was actually not expected to produce anything else. I felt this was a wasted opportunity, and wrote a 10-page document on recommendations for future programming. It was very welcomed, and apparently used extensively for the proposal itself.

• Timing of briefings and meeting notes: We divided up responsibilities for writing up meeting notes. We each took our own notes, but one person was responsible for circulating notes at the end of each day. We all reviewed and added/corrected information as needed; the program officer compiled the info into one document. This allowed us to stay current, updating ourselves while the information was still fresh. We were also able to see patterns emerge and identify gaps.

• Tap into participants direct views and experience: I would have designed a TOR that would have explicitly required each member of the team to provide his/her thoughts/recommendation before departing. These accounts could then have fed into the final mission report.
• Involvement of whole team in program development: To the extent possible, the organization could have continued to engage the team in the preparation of the proposal. Team members could have either been directly involved in the program development process, or at least asked to comment and provide feedback once the proposal was drafted. Continuing to build a relationship with team members would also have helped the organization recruit  experts if and when they win the bid.

• Early in the mission, we established expected deadlines for drafting of the proposals; this took into account the need to circulate drafts among staff and have completed proposals by the end of the mission. With an explicit time frame, there were no surprises, and I could organize my time according to agreed upon criteria. We discussed progress on a regular basis, which helped us all stay on schedule.
• Unlike many other missions, the outputs for this were clear: two proposals were to be drafted by the end of mission. Measuring progress was therefore straightforward.

• Built-in mechansims: Following approval of the White Paper by Cabinet and Parliament, the Department of Defence undertook a major Defence Review. The aim was to translate the high level policy pronouncements into more concrete force design arrangements, budgets, transformation programmes and organizational structures. What this process lacked was a built-in requirement for a formal assessment after a 5 or 7 year period.

None

• Formal review process: A White Paper and Defence Review should provide for a formal review and assessment process that includes the parliamentary defence committee; undertakes the review and submits a report to parliament within a specified period; and determines whether a new White Paper is required in the light of experiences to date and new security and other circumstances.

Monitoring and Evaluation in general suffer fron the "Stop and Go" approach. Meaning that a team is deployed to perform a monitoring mission for a specific amount of time, at the end of which they will have to go back to the office and momentum is lost.
This, in itself, is a bad practice. The monitoring component needs to be an on-going component. When ISSAT or another organisation sends a monitoring mission to the field, they need to be deployed throughout the span of a project.
This is the only way to be able to monitor the whole process without losing momentum.

Mission preparations by the WB made it possible that the team meets all significant stakeholders without missing out on anyone, including CSOs.

Monitoring processes need to be ongoing process covering the whole life-span of a project. Monitoring measure need to be taken at the very beginning of the design phase of a project.

Political monitoring: The political follow up and results, foreseen and unforeseen, could have been even more carefully monitored for adequate management response.

Creating national baseline: The assessment in itself was creating the basis for a national homicide baseline and a monitoring system.

Political monitoring: For similar processes, emphasizing the political dimension and necessary national ownership for change, the monitoring of political outcomes need to improve.

• Complexity: Some complex M&E methods for the curriculum could have benefitted from more explanations/support to the Technical Working Group, in particular methods of knowledge management.

• Manage expectations: Given tight time frame, the ‘curriculum was considered transitional.  A validation process was established for the following two months to review the content and conduct a training needs analysis to match the content more accurately to its audience.
• Knowledge Management: To keep track of feedback on the curriculum, the groundwork for a knowledge management system was discussed. Options were provided to ensure feasibility of longer term management according to current DGMMA capacity, enforced by some training.

• Mentoring and support: Due to uncompleted tasks leading to the workshop (identification of experts), monitoring through knowledge management will require hands on support to DGMMA staff on data collection to allow for a proper review of the curriculum in two years time.