Support to Evaluation of Security Sector Development Programme in Burundi

15/02/2012 - 15/02/2012

Target country

Mandator

The overall objective of the mission was to assist the Netherlands Embassy in Burundi to evaluate the results and impact of Phase I of their Security Sector Development Programme (SSDP). This will include an analysis of the efficiency of the project implementation structures, its design and relevance to its designated beneficiaries.

Mandate outputs / products

An Evaluation Report (with concrete recommendations on way forward).

Outcome objectives of mandate

This mission’s outcome is to contribute to a more relevant, coherent, effective and sustainable SSDP through sound, realistic recommendations that target all these aspects.

Start date

15/02/2012

End date

15/02/2012

Summary

  • Pre-deployment preparation for a team is crucial for the success of the mission and for the team to be flexible enough to deal with the challenges of the programme. Without a well-thought-through plan which is flexible enough to allow for adaptation, the team might lose precious time trying to make sense of all the information they are receiving without having an appropriate structure to absorb that into.
  • Key milestone during that phase are: desk research, identifying main challenges and issues to focus on, designing an overarching methodology and report structure.
  •  Team working sessions at the end of each day for the period of at least one hour, are key to the success of a mission allowing the advisors to go through the information they have received in a more structured way and facilitating cross learning as well as ensuring that the team maintain common objectives in sight and move toward common objectives.
  • Translation of terms between different languages is tricky enough; this can be made even more complex if you add the technical terminology of an evaluation and that of the armed forces institution or the police institution. Advisors deployed to evaluation mission and need to conduct interviews with stakeholders should be very knowledgeable of the technical terminology, as well as the cultural context so as not to create confusion.
  • Involve ISSAT team more during missions: Under the guidance of team leader involve ISSAT resource people, before, during and after a mission through getting their input on mission design and asking for their support during implementation. After mission completion, informal debriefings, templates for reports, in-house comments on reports, and usage of the CoP for innovative solutions would be a positive way forward for ISSAT.

Other Comments

  • Bernard mentioned the possibility of doing a video with him tackling the differences between the Police institution and the military institution and how the latter is more open and transparent than the former. Aspects that he believes are very much universal to these institutions due to specific reasons he could detail in that video.

Specific Lessons Identified

Challenges

  • M&E OGN: The weakness of the OGN’s is that they are only useful to those who have never done a certain process. For experienced practitioners the OGNs are of too basic nature.

Successes

  • M&E OGN: ISSAT OGN on M&E does not exist. This led to ISSAT Advisor conducting a working session with ISSAT M&E expert to design the framework and approach for the evaluation.

Recommendations

  • M&E OGN: It is the team’s opinion that one of ISSAT’s added values is the role it plays in methodology design for practitioners. An M&E OGN would be much needed and appreciated by SSR practitioners.

Tags

Challenges

  • Local programme staff limited resources and time to undertake the additional workload of international mission preparation: The schedule of meetings and interviews was not prepared and shared with ISSAT experts before their field deployment despite the explicit requests from the –Geneva team.  This is due in part to the knowledge of local programme staff of the flexibility of local stakeholders and the possibility of confirming interviews at the last minute. It was also due to the fact that the programme staff’s limited resources and time to dedicate to such an evaluation. They had to juggle their own working schedules with the additional responsibility of organizing such a mission.
  • Balance between planning imperatives and the context imperatives: This gap in planning meant that the team had to spend some of its time on logistics. However, this did not have a big effect on the mission’s effectiveness since local stakeholders were easily accessible and scheduling appointments for interviews was relatively rapid. The international team’s continued requests for a schedule of interviews as an important milestone for their planning led to certain moments of tension with the local counterparts. The latter interpreted the international team’s concern as lack of confidence in their capacities and knowledge. The international team however, was uncomfortable arriving to the country without a list of planned interviews in hand. 
  • First engagements for experts from ISSAT roster: One of the experts on this mission was from the ISSAT Expert Roster embarking on a first mission with ISSAT. Substantive planning with this expert had already started when they discovered that the ISSAT operating procedures were not suitable for them. This was challenging for the team since the mission-planning process had already started and losing one team member at this stage would have problematic.
  • Security Briefings for ISSAT staff: Security briefings for ISSAT staff are standard imperatives for all missions. However, in the case of this country, the Dutch Embassy has not provided security briefings for its own staff members, who work and live in country with their families. Requesting security briefing for ISSAT inspite of this fact might cause exasperation as a result of unrealistic expectations on ISSAT’s side.

Successes

  • Flexibility of local stakeholders: The stakeholders implicated in the interviews, were very flexible in terms of their availability and the evaluation team could organise meetings with a fair degree of facility and relatively short notice.
  • Project Assistant supporting missions: The evaluation team received the support of a Project Assistant whose contribution to the mission was to perform the necessary desk review, collate reports and papers, review their relevance with the available advisor and then ensure that the team is able to access them through Basecamp. This worked very well as it ensured that everyone had a harmonized database of documents and all team members had access to the same documents saving them time and effort that they could then invest in the planning and methodology design. This good practice was a lesson learnt from previous missions where ISSAT Advisors used to undertake the above role at the expense of time that should be invested in the mission methodology design and other preparative activities.

Recommendations

  • Balance between planning imperatives and the context imperatives: e mission team tried to push for more rigorous planning regarding the interviews with stakeholders. However, when the situation in the field became clearer in terms of the programme staff’s resources and time, in addition to reassurances from them on the expected flexibility of stakeholders, the international team pushed less on their criteria and accepted the context’s imperatives. A balance needs to be struck between structured planning imperatives, the reality of the field, time limitations and the local programme staff’s capacity, resources and availability. This could be done in being realistic about the crucial issues to be ready before field deployment and the issues that permit more flexibility. This balance is usually facilitated by the team leader as in most cases team members have different visions of what should be urgent and what is not. 
  • First engagements for experts from ISSAT roster: Having a consolidated power point presentation explaining ISSAT’s structure and functions should be made available to all Expert Roster Members on the CoP. Sharing ISSAT contracting procedures and conditions with first-time Expert Roster members before they embark on the mission planning is of capital importance.
  • Security Briefings for ISSAT staff: In cases where the mandator does not provide security briefings to its own staff members, higher flexibility from ISSAT might be required. It might need to look for alternatives whether online or through contacting relevant embassies.
  • Joint collaboration and consultations between ISSAT mission teams and In-house resource people: Consultations and/or joint working sessions between mission team members and other ISSAT members on specific aspects (governance, gender, strategic planning, etc.) before, during and after field mission can be of high value on the team’s work. Proper channels need to be put in place for such exchanges. Some suggestions can be:
    • Before missions: Working sessions and/or discussions with advisors as appropriate
    • During mission: Email interactions between mission team and Geneva team on targeted issues
    • After mission: Post-mission informal briefings by the mission team to ISSAT team (upon interest). This will help the process of thought of the team before  the report typing phase.

Tags

Successes

  • Informal Evaluation: This mission’s success and impact is going to measured informally by ISSAT through the following three indicators:
    • Whether the mandator will continue to use ISSAT
    • Whether the evaluations recommendations were implemented
    • Feedback form reflects positive feedback on ISSAT Advisors’ performance

Recommendations

  • Necessity to evaluate ISSAT engagements in countries where there has been medium to long term engagement for ISSAT: ISSAT has been involved in Burundi for the past two years with 5 missions being deployed to support various donor programmes. An evaluation of ISSAT’s contribution and impact on those donors and their programmes would be a positive step to undertake.

Tags

Challenges

  • Ownership of programme staff: In theory, the Steering Committees of this programme (Political Committee and high-level Technical Committee) should have led this exercise. However, in reality, the Political Committee met only twice since the beginning of the programme (representation is at level of Ministers who are rarely available) and the Technical Committee (Burundians and programme staff) were not interested enough by this initiative. As a result the Dutch Embassy drove this evaluation. This fact is not however indicative of the national ownership levels for the programme. The Steering Committees and the level of representation were ineffective due to their flawed conception. 
  • Ownership of local stakeholders: See comments on Burundian team members’ contribution under methodology. 

Recommendations

  • Ownership of programme staff: Involve steering committees.
  • Ownership of local stakeholders: See recommendations on Burundian team members’ contribution under methodology

Tags

Challenges

  • Newly formed teams: The challenge of newly formed teams is that members are not familiar with each other, their skills, competencies and methods of work.
  • See comments on Burundian team members’ contribution under methodology

Successes

  • The size and skills of the team: The team as it was defined worked very well in terms of its size and type of experience. Having 3 experts deployed for the evaluation of a big programme was the appropriate number as each advisor had one pillar to focus their efforts, while collaborating with the others on the other pillars. In terms of the skills and competencies deployed, this worked very well also. There was an expert on the thematic area (evaluation) and one expert on the technical questions (police and military) and one team leader who also dealt with governance aspects.
  • Pre-deployment preparation: All the team met in Brussels before deploying to Burundi to harmonize information, approaches and work plans.
  • Time dedicated for team work factored in daily: This team met once every day at the end of each day to recap with the team main findings, challenges etc this helped harmonise the picture for all team members. These working sessions were of crucial value to ensure a certain coherence of work between members thus allowing for building on each other’s work as opposed to duplicating or collating information without knowledge of its value to the objective of the evaluation.
  • The Mandator’s role in evaluations: The mandators were very much hands-off when it came to the team’s work. They did not interfere with the design or planning of the mission and left the team entirely free in undertaking the evaluation.

Recommendations

  • Collaboration with the Burundian team members should have been managed better to get better results on both sides (see section on methodology and approach).

  • Time dedicated for team work factored in daily: This is absolutely key to the success of a mission. These meetings should be at least an hour long. These meetings were missing in a similar mission in Congo which had a big negative impact on the team’s work).

  • The Mandator’s role in evaluations: A hands-off approach by the mandator in terms of the substantive part of the evaluation and assuming the role of a partner resourceful of the historical background of the programme is a very positive framework to encourage. 

  • Collaboration with the Burundian team members should have been managed better to get better results on both sides (see section on methodology and approach)

  • Time dedicated for team work factored in daily:  This is absolutely key to the success of a mission. These meetings should be at least an hour long. These meetings were missing in a similar mission in Congo which had a big negative impact on the team’s work).

  • The Mandator’s role in evaluations: A hands-off approach by the mandator in terms of the substantive part of the evaluation and assuming the role of a partner resourceful of the historical background of the programme is a very positive framework to encourage.

Tags

Challenges

Joint teams (international/national representatives): The Programme manager designed a theoretically very useful and positive approach for the evaluation which consists of deploying a national Security Sector representative to be associated to each advisor during the evaluation process. The objectives of such an approach were to –on the one hand- allow the Burundian counterparts to learn through actively participating in the exercise; on the other hand, to allow the evaluation to be anchored in solid knowledge of the context and the realities of the country’s security sector.

This however had varied implications in practice. Due to the absence of a clear structure for the national team members identifying their roles and deliverables, their contribution to the evaluation depended very much on variables such as: their relationship with the international advisor, personal dynamics, their knowledge of the process etc.

As a result in some cases they contributed by providing a local view and value to the evaluation and in other cases, they played the roles of logistics coordinators rather than contributing substantially to the evaluation.

This was due to the following factors:

  •  
    • The Terms of Reference for the mission, do include the designation of the Burundian team members and describe their role in a rather superficial way, not detailing what is to be expected from them after the desk review is done and the stakeholders identified;
    • After the three Burundian team members were designated by their respective institutions (police and armed forces), the team was not aware of any follow-up meeting with these individuals to brief them on their role and prepare them to proactively contribute to the design of the methodology.
    • The three Burundian team members were associated to an advisor in a way to allow the police officer to be on the pillar evaluating the army and vice versa. This did not work well in practice as the army officers were sometimes not comfortable speaking infront of the police representatives and neither were the police officers to the armed forces’ representatives. This led to instances where the Burundian team member was prevented to come to the meeting, or in some cases, to leave the room, while the international advisor carried on the interview regarding more sensitive issues.
    • The Burundian team members did not feel at ease or have the sufficient confidence in their role to be proactive in contributing to the evaluation. One of the Burundian officers did contribute slightly to asking questions but that was since he knew the international advisor well and felt comfortable enough to put questions forward.
    • Since the Burundians were not implicated in the process from the start, their learning during the interviews was very limited as they did not have the overall vision of the underlying approach and the long term objective of these questions.  

Successes

  • Joint collaboration and consultations between ISSAT mission teams and In-house resource people: For this mission ISSAT M&E Expert supported directly the design of the evaluation’s methodology. This exercise was useful in two ways: It helped build the capacity of the ISSAT Advisor who had thematic expertise (Police Reform) but not the M&E knowledge and skills. The second benefit was related to the team acquiring an overarching guiding framework that facilitated their mission considerably. Without this preparatory exercise, the team could not have been able to deal with the complexities of the mission in a coherent manner which gave them the ability to respond to the challenges of the programme (absence of criteria for success, indicators, etc.) without losing sight of their objectives and without losing too much time in attempting to clarify how these considerations fit together. 
  • Design of methodology and structure of the report before deployment: Having designed the structure of the report before deploying was also very helpful to the team as they could very rapidly and with ease collate the information they got out of their interviews and analyse them. This structure also facilitated the exchange of information amongst team members and their collaboration in terms of which piece of information fits where on the skeleton of the evaluation’s report.
  •  In-depth exchanges with field programme leads and staff:The team leader for this mission reserved a considerable amount of time to work with the programme lead in the country. The objectives of these working sessions were to ensure that the mission team and the programme team had a coherent understanding and the same expectations from the evaluation. The draft ToRs were used as basis for discussion.  

Recommendations

  • Joint collaboration and consultations between ISSAT mission teams and In-house resource people:  Consultations and/or joint working sessions between mission team members and other ISSAT members on specific aspects (governance, gender, strategic planning, etc.) before, during and after field mission can be of high value on the team’s work. Proper channels need to be put in place for such exchanges. Some suggestions can be:
    • Before missions: Working sessions and/or discussions with advisors as appropriate
    • During mission: Email interactions between mission team and Geneva team on targeted issues
    • After mission: Post-mission informal briefings by the mission team to ISSAT team (upon interest). The purpose of such informal debriefing is to share the first observations and key lessons identified. An exchange and questions can follow during the debriefing. This exercise should be able to help the mission team to brainstorm on their findings and put their ideas together as they are writing their report.
    • It might be useful (depending on the missions) to consider spending one more week on every report and circulating among the ISSAT team for commen
  • Design of methodology and structure of the report before deployment:  For evaluation missions, it is recommendable that the methodology and the report structure be designed before the team deploys to the field. However, depending on the programme, the methodology should be an overarching one proposing themes to be tackled and overarching questions to be asked without going too much into detail (such as designing indicators, or sub-questions). Necessary margins of flexibility should be left for the team to adjust the methodology as they arrive to the field and are briefed by the programme staff. Having a methodology before hand, helps the team conserve the guiding values of an evaluation (coherence, efficiency, etc.) and not lose sight of them.
  • Joint teams (international/national representatives): In terms of the initiative of creating evaluation teams that combine international advisors and national actors, it is the recommendation of this team to approach this matter in a much more structured manner through the following:
    • Once ToRs for the assignment are drafted, there should be a section dedicated to the international advisors role and the Burundian actors’ roles, detailing the responsibilities and roles of each group.
    • The ToRs should also include the value of including the national actors as a cross-learning process. The international advisors could pass on the comparative, international view and the modern technical know-how of evaluations and the Burundians could bring in the technical and cultural know-how of their security institutions and country. This would facilitate more realistic analysis and adapted methodologies to the reality of the country and programme. 
  • In-depth exchanges with field programme leads and staff: Even if the ToRs are clear, the mission team should always dedicate time to discussing them with the programme team and other relevant stakeholders so expectations are harmonized amongst the group. 
  • ISSAT templates:Disposing of an ISSAT templates for reports is very much needed. A report template per process would be very beneficial the teams (assessment report template, evaluation report template, etc.). This could include in each case a paragraph on what is ISSAT at the end which can serve as a useful reminder to mandators as well as help place ISSAT’s recommendations in their bigger picture (ISSAT’s mandate for reinforcing good practice).

Tags