Security Sector Development Programme for Burundi (SSDP)

Methodology design

Scoping mission

Implementation

Working with Comité Directeur

The Evaluation report

Political Coverage of ISSAT on evaluations

This mandate fully delivered the expected outputs, including an audit report with recommendations in accordance to the mandator’s needs. The recommendations were taken already in consideration and two subsequent missions from Polfed to Burundi worked on how to implement some of them. The reaction to the recommendations was immediate and positive also from the IGSP, which already took suggested points of action in consideration and welcomes the clarification of roles and responsibilities between the Inspectorate and SSD. During the mandate, the position of Inspector-general was vacant but the new head of IGSP took office in the same day of the end of the field mission. A working group was set up just after the restitution session to discuss a set of activities aimed at obtaining quick-wins on the recommendations and that ISSAT proposed to be carried out until Q4 2014.

To this date, the IGSP still didn’t act on most of these actions and further steps depend on the completion of the new legal framework for the IGSP. The Organic Law and the Internal Statute of the IGSP have been drafted and provide finally a legal framework for organising  the work of the Inspectorate, although a decree on the function of internal audit is still to be drafted. Until the legal framework is approved and fully in effect, the scope of activities of IGSP  is nonetheless very limited, specially when it comes to investigations and filing of cases.

The extent to which recommendations are fully implemented will need to be checked in one or two years, since a previous audit to IGSP was basically left in the drawer in terms of acting on suggested points. The review report delivered by the team provides a strong base for the design of SSD projects in support of the IGSP but it also implies and needs that the mandator follows up on the results by pushing for tangible actions (see below).

Throughout the report, attention is given to technical and procedural notions that help unpack the concept of internal control and that lay out a comprehensive set of principles, mechanisms and tools. This corpus can be easily taken as part of training materials for IGSP staff and the PNB overall, as acknowledged by the SSD/MSP coach visiting ISSAT in October to discuss further areas of collaboration related to training in police integrity.

As part of the second/main field mission, in March 2014, a one-day workshop was organised by the team in Bujumbura to conduct a SWOT analysis of the IGSP with a group of fifteen staff of the Inspectorate. The SWOT exercise was considered to provide simple yet powerful analytical tools, with gains in terms of ownership of the review itself.

The evaluation team was assisted in Burundi by two members from IGSP that facilitated contacts and participated in several meetings. Future will tell what will result  from this exposure of IGSP staff to ISSAT’s evaluation methodology and to intense discussions that articulated the principles of internal control with the realities on the ground and the political and social context in Burundi.

For ISSAT, the fact that the evaluation team included two experts working actually in police internal control in their own country provided an opportunity to gain a more technical view of the questions involved. It is considered a relevant gain of capacity for future reviews and an important enhancement of existing in-house knowledge on police reform, integrity and responsibility.

It is the perception of the IGSP that the mandate resulted in opening some doors to potential collaboration of the institution with civil society organisations.

• A good preparation of the mandate, starting with a one-day meeting in Geneva aligning approaches and planning work among Polfed and ISSAT experts on team, starting from scratch and designing the mission.
• Scoping visit from the team leader to Burundi.
• Wealth of local/contextual knowledge in the team.
• Mixed team composition bringing complementary skills and approaches to democratic control mechanisms: technical/Polfed and SSR/ISSAT.
• Trust build over time between ISSAT and SSD in Burundi.
• Good local interaction between SSD, CTB (Belgian development agency) and GIZ (German aid agency).
• ISSAT methodology.
• Facilitation from IGSP staff.

Lessons identified for improving the chances of success:

• • Recommendations can be discussed with local stakeholders in a way that allows for some knowledge transfer on programme design, in the case restricted to the elaboration of an action plan for ‘quick-wins’. Such an exercise can bridge recommendations with whichever projects that SSD decides to carry forward following the review.
  • SSD needs to act on the audit in terms of pushing to have some of the recommendations followed on; this implies that the Programme has to keep representing to the Burundian authorities that crucial decisions have to be taken and operationalised soon to have IGSP fulfilling its important role.

Concepts and processes– Uneven understanding among different stakeholders, within the IGSP and across the MSP of the role, importance and mechanisms entailed by internal control in a functioning democracy.

Political engagement– As noticed in the AAR, even the most immediate quick-win activities recommended in the report need that IGSP has a legal framework in place, but political commitment from Burundian Government to move forward on this has been vague, in fact compromising IGSP critical role.

Programme support– The audit sets clear directions and suggests an action plan to address fundamental gaps in internal control, but any progress on this depends on a robust and immediate support from SSD to democratic oversight and accountability.

Programme overstretch– SSD is a complex, far-reaching SSR programme spanning for eight years and supporting the main security institutions in Burundi (Police and Defence Forces), but there is the risk that whatever is not included or prioritised and funded under SSD is not purposely sustained by the Burundian counterparts.

Uneven and unused capacity– IGSP badly lacks capacity across the board when it comes to main and secondary/procedural functions of an internal control body, from legal expertise to strategic management, while some of the actual existing capacity among staff is lost by the way IGSP functions.

Lessons identified:

• Civil society organisations are usually keen to engage in discussions about internal control mechanisms of police services. Such occasions offer entry-points to bring-in external stakeholders, otherwise kept fairly outside relevant processes and activities, as it happened this time, in spite of clear demands from ISSAT for such opening from the Burundian authorities.
• Training in internal control and democratic oversight, linking up internal audit bodies with the role of external stakeholders (Parliament, Ombudsman, etc.), are the natural and crucial follow-up of this mandate. These activities can be a contribution from ISSAT to pursue support in areas identified as major capacity gaps, and keeping the momentum for a reinforcement of the SSD Governance pillar. It’s worth reminding that “Internal Control” goes on being  one of the focus areas under the action plan for ‘Police’ pillar of SDD Phase III.
• Dedicate the first day of the team in the field to set up interviews and organise agenda, since gaps in pre-arranging interviews are recurrent and almost unavoidable in such contexts.

Integrating Concepts Mid-Workflow: The workshops took place after the action plans for the military and the police had been written. For internationals external to the programme to come in after the fact to recommend new concepts for inclusion can come across as critical of national efforts and undermine local ownership. A very good communication strategy amongst the team, coupled with strong high level political support, is needed to mitigate the negative impacts of coming in late.

Partnering and Transferring Responsibility: The overall structure of the eight-year programme is designed to start with a shared responsibility between the Dutch and the Burundians, and progressively have the Dutch play more of a technical advisory role.

National Coordinator: The Burundian co-coordinator of the programme is very competent and has good relationships with both the Ministers of Interior and Defence. This allowed the programme to have substantive impact.

Politics of SSR: Conveying the politics of SSR proved to be a concept that was difficult to transfer as not all uniformed participants viewed themselves as having a political role, rather they are neutral. While encouraging, there are still political aspects for all actors involved in SSR. This includes trainers wishing to convey the concept.

ISSAT Training: Portions of ISSAT’s Level 1 training were used to convey the key principles of SSR as well as fundamental components of supporting an SSR programme. The materials were well suited to be adapted to a specific audience.  

New Concepts: There will always be a challenge in instilling a new concept or methodology in people unfamiliar with the SSR approach. Make sure to be aware of their willingness and ability to take on a new concept. These two indicators can help in planning an initial engagement as well as follow on actions.

Politics of SSR: Be aware that discussing the politics of SSR is in itself political and plan accordingly.

Clarity: There was a very good working relationship among the team as roles, responsibilities, and objectives were all made clear ahead of the mission.

Uniform Culture: When working with security actors, different uniforms (i.e. police, military) hold different traditions and institutional mentalities. If you can be aware of these factors well in advance, respect them, and work with them as opposed to working against them, then the collaboration with partners from these groups will greatly improve.

Actions and words: The evaluation of impact is through a review of updated action plans for the military and police. The mission will be successful when we are able to observe evidence of SSR concepts from the workshop being incorporated into the updates. While this is one indicator that the concepts were learned, additional indicators need to be established to see if stakeholders can implement these concepts.

Informal assessment: Ahead of the workshops, the training team met with Dutch staff to discuss expected attitudes towards the SSR concepts among the Burundian team. The informal nature of this discussion was quite useful to get a quick assessment of training needs and absorbsion capacity, and enabled the training team to tailor the training material to the audience.  Meeting informally, in this instance, as opposed to formal interviews also allowed the military and police officials to feel more relaxed and speak more openly about their thoughts.

Get to know the audience: As useful as the informal discussions with the Dutch team were, it would have been even more beneficial to have met informally with the Burundian counterparts. This would have given an even better idea of how to adapt the training, as well as established an initial rapport. 

Joint teams (international/national representatives): The Programme manager designed a theoretically very useful and positive approach for the evaluation which consists of deploying a national Security Sector representative to be associated to each advisor during the evaluation process. The objectives of such an approach were to –on the one hand- allow the Burundian counterparts to learn through actively participating in the exercise; on the other hand, to allow the evaluation to be anchored in solid knowledge of the context and the realities of the country’s security sector.

This however had varied implications in practice. Due to the absence of a clear structure for the national team members identifying their roles and deliverables, their contribution to the evaluation depended very much on variables such as: their relationship with the international advisor, personal dynamics, their knowledge of the process etc.

As a result in some cases they contributed by providing a local view and value to the evaluation and in other cases, they played the roles of logistics coordinators rather than contributing substantially to the evaluation.

This was due to the following factors:

•  
  • The Terms of Reference for the mission, do include the designation of the Burundian team members and describe their role in a rather superficial way, not detailing what is to be expected from them after the desk review is done and the stakeholders identified;
  • After the three Burundian team members were designated by their respective institutions (police and armed forces), the team was not aware of any follow-up meeting with these individuals to brief them on their role and prepare them to proactively contribute to the design of the methodology.
  • The three Burundian team members were associated to an advisor in a way to allow the police officer to be on the pillar evaluating the army and vice versa. This did not work well in practice as the army officers were sometimes not comfortable speaking infront of the police representatives and neither were the police officers to the armed forces’ representatives. This led to instances where the Burundian team member was prevented to come to the meeting, or in some cases, to leave the room, while the international advisor carried on the interview regarding more sensitive issues.
  • The Burundian team members did not feel at ease or have the sufficient confidence in their role to be proactive in contributing to the evaluation. One of the Burundian officers did contribute slightly to asking questions but that was since he knew the international advisor well and felt comfortable enough to put questions forward.
  • Since the Burundians were not implicated in the process from the start, their learning during the interviews was very limited as they did not have the overall vision of the underlying approach and the long term objective of these questions.  

• Joint collaboration and consultations between ISSAT mission teams and In-house resource people: For this mission ISSAT M&E Expert supported directly the design of the evaluation’s methodology. This exercise was useful in two ways: It helped build the capacity of the ISSAT Advisor who had thematic expertise (Police Reform) but not the M&E knowledge and skills. The second benefit was related to the team acquiring an overarching guiding framework that facilitated their mission considerably. Without this preparatory exercise, the team could not have been able to deal with the complexities of the mission in a coherent manner which gave them the ability to respond to the challenges of the programme (absence of criteria for success, indicators, etc.) without losing sight of their objectives and without losing too much time in attempting to clarify how these considerations fit together. 
• Design of methodology and structure of the report before deployment: Having designed the structure of the report before deploying was also very helpful to the team as they could very rapidly and with ease collate the information they got out of their interviews and analyse them. This structure also facilitated the exchange of information amongst team members and their collaboration in terms of which piece of information fits where on the skeleton of the evaluation’s report.

•  In-depth exchanges with field programme leads and staff:The team leader for this mission reserved a considerable amount of time to work with the programme lead in the country. The objectives of these working sessions were to ensure that the mission team and the programme team had a coherent understanding and the same expectations from the evaluation. The draft ToRs were used as basis for discussion.  

• Joint collaboration and consultations between ISSAT mission teams and In-house resource people:  Consultations and/or joint working sessions between mission team members and other ISSAT members on specific aspects (governance, gender, strategic planning, etc.) before, during and after field mission can be of high value on the team’s work. Proper channels need to be put in place for such exchanges. Some suggestions can be:
  • Before missions: Working sessions and/or discussions with advisors as appropriate
  • During mission: Email interactions between mission team and Geneva team on targeted issues
  • After mission: Post-mission informal briefings by the mission team to ISSAT team (upon interest). The purpose of such informal debriefing is to share the first observations and key lessons identified. An exchange and questions can follow during the debriefing. This exercise should be able to help the mission team to brainstorm on their findings and put their ideas together as they are writing their report.
  • It might be useful (depending on the missions) to consider spending one more week on every report and circulating among the ISSAT team for commen
• Design of methodology and structure of the report before deployment:  For evaluation missions, it is recommendable that the methodology and the report structure be designed before the team deploys to the field. However, depending on the programme, the methodology should be an overarching one proposing themes to be tackled and overarching questions to be asked without going too much into detail (such as designing indicators, or sub-questions). Necessary margins of flexibility should be left for the team to adjust the methodology as they arrive to the field and are briefed by the programme staff. Having a methodology before hand, helps the team conserve the guiding values of an evaluation (coherence, efficiency, etc.) and not lose sight of them.
• Joint teams (international/national representatives): In terms of the initiative of creating evaluation teams that combine international advisors and national actors, it is the recommendation of this team to approach this matter in a much more structured manner through the following:
  • Once ToRs for the assignment are drafted, there should be a section dedicated to the international advisors role and the Burundian actors’ roles, detailing the responsibilities and roles of each group.
  • The ToRs should also include the value of including the national actors as a cross-learning process. The international advisors could pass on the comparative, international view and the modern technical know-how of evaluations and the Burundians could bring in the technical and cultural know-how of their security institutions and country. This would facilitate more realistic analysis and adapted methodologies to the reality of the country and programme. 
• In-depth exchanges with field programme leads and staff: Even if the ToRs are clear, the mission team should always dedicate time to discussing them with the programme team and other relevant stakeholders so expectations are harmonized amongst the group. 
• ISSAT templates:Disposing of an ISSAT templates for reports is very much needed. A report template per process would be very beneficial the teams (assessment report template, evaluation report template, etc.). This could include in each case a paragraph on what is ISSAT at the end which can serve as a useful reminder to mandators as well as help place ISSAT’s recommendations in their bigger picture (ISSAT’s mandate for reinforcing good practice).

• Newly formed teams: The challenge of newly formed teams is that members are not familiar with each other, their skills, competencies and methods of work.
• See comments on Burundian team members’ contribution under methodology

• The size and skills of the team: The team as it was defined worked very well in terms of its size and type of experience. Having 3 experts deployed for the evaluation of a big programme was the appropriate number as each advisor had one pillar to focus their efforts, while collaborating with the others on the other pillars. In terms of the skills and competencies deployed, this worked very well also. There was an expert on the thematic area (evaluation) and one expert on the technical questions (police and military) and one team leader who also dealt with governance aspects.
• Pre-deployment preparation: All the team met in Brussels before deploying to Burundi to harmonize information, approaches and work plans.
• Time dedicated for team work factored in daily: This team met once every day at the end of each day to recap with the team main findings, challenges etc this helped harmonise the picture for all team members. These working sessions were of crucial value to ensure a certain coherence of work between members thus allowing for building on each other’s work as opposed to duplicating or collating information without knowledge of its value to the objective of the evaluation.
• The Mandator’s role in evaluations: The mandators were very much hands-off when it came to the team’s work. They did not interfere with the design or planning of the mission and left the team entirely free in undertaking the evaluation.

• Collaboration with the Burundian team members should have been managed better to get better results on both sides (see section on methodology and approach).
• Time dedicated for team work factored in daily: This is absolutely key to the success of a mission. These meetings should be at least an hour long. These meetings were missing in a similar mission in Congo which had a big negative impact on the team’s work).
• The Mandator’s role in evaluations: A hands-off approach by the mandator in terms of the substantive part of the evaluation and assuming the role of a partner resourceful of the historical background of the programme is a very positive framework to encourage. 
• Collaboration with the Burundian team members should have been managed better to get better results on both sides (see section on methodology and approach)
• Time dedicated for team work factored in daily:  This is absolutely key to the success of a mission. These meetings should be at least an hour long. These meetings were missing in a similar mission in Congo which had a big negative impact on the team’s work).
• The Mandator’s role in evaluations: A hands-off approach by the mandator in terms of the substantive part of the evaluation and assuming the role of a partner resourceful of the historical background of the programme is a very positive framework to encourage.

• Ownership of programme staff: In theory, the Steering Committees of this programme (Political Committee and high-level Technical Committee) should have led this exercise. However, in reality, the Political Committee met only twice since the beginning of the programme (representation is at level of Ministers who are rarely available) and the Technical Committee (Burundians and programme staff) were not interested enough by this initiative. As a result the Dutch Embassy drove this evaluation. This fact is not however indicative of the national ownership levels for the programme. The Steering Committees and the level of representation were ineffective due to their flawed conception. 
• Ownership of local stakeholders: See comments on Burundian team members’ contribution under methodology. 

• Ownership of programme staff: Involve steering committees.
• Ownership of local stakeholders: See recommendations on Burundian team members’ contribution under methodology

• Informal Evaluation: This mission’s success and impact is going to measured informally by ISSAT through the following three indicators:
  • Whether the mandator will continue to use ISSAT
  • Whether the evaluations recommendations were implemented
  • Feedback form reflects positive feedback on ISSAT Advisors’ performance

• Necessity to evaluate ISSAT engagements in countries where there has been medium to long term engagement for ISSAT: ISSAT has been involved in Burundi for the past two years with 5 missions being deployed to support various donor programmes. An evaluation of ISSAT’s contribution and impact on those donors and their programmes would be a positive step to undertake.

• Local programme staff limited resources and time to undertake the additional workload of international mission preparation: The schedule of meetings and interviews was not prepared and shared with ISSAT experts before their field deployment despite the explicit requests from the –Geneva team.  This is due in part to the knowledge of local programme staff of the flexibility of local stakeholders and the possibility of confirming interviews at the last minute. It was also due to the fact that the programme staff’s limited resources and time to dedicate to such an evaluation. They had to juggle their own working schedules with the additional responsibility of organizing such a mission.
• Balance between planning imperatives and the context imperatives: This gap in planning meant that the team had to spend some of its time on logistics. However, this did not have a big effect on the mission’s effectiveness since local stakeholders were easily accessible and scheduling appointments for interviews was relatively rapid. The international team’s continued requests for a schedule of interviews as an important milestone for their planning led to certain moments of tension with the local counterparts. The latter interpreted the international team’s concern as lack of confidence in their capacities and knowledge. The international team however, was uncomfortable arriving to the country without a list of planned interviews in hand. 
• First engagements for experts from ISSAT roster: One of the experts on this mission was from the ISSAT Expert Roster embarking on a first mission with ISSAT. Substantive planning with this expert had already started when they discovered that the ISSAT operating procedures were not suitable for them. This was challenging for the team since the mission-planning process had already started and losing one team member at this stage would have problematic.
• Security Briefings for ISSAT staff: Security briefings for ISSAT staff are standard imperatives for all missions. However, in the case of this country, the Dutch Embassy has not provided security briefings for its own staff members, who work and live in country with their families. Requesting security briefing for ISSAT inspite of this fact might cause exasperation as a result of unrealistic expectations on ISSAT’s side.

• Flexibility of local stakeholders: The stakeholders implicated in the interviews, were very flexible in terms of their availability and the evaluation team could organise meetings with a fair degree of facility and relatively short notice.
• Project Assistant supporting missions: The evaluation team received the support of a Project Assistant whose contribution to the mission was to perform the necessary desk review, collate reports and papers, review their relevance with the available advisor and then ensure that the team is able to access them through Basecamp. This worked very well as it ensured that everyone had a harmonized database of documents and all team members had access to the same documents saving them time and effort that they could then invest in the planning and methodology design. This good practice was a lesson learnt from previous missions where ISSAT Advisors used to undertake the above role at the expense of time that should be invested in the mission methodology design and other preparative activities.

• Balance between planning imperatives and the context imperatives: e mission team tried to push for more rigorous planning regarding the interviews with stakeholders. However, when the situation in the field became clearer in terms of the programme staff’s resources and time, in addition to reassurances from them on the expected flexibility of stakeholders, the international team pushed less on their criteria and accepted the context’s imperatives. A balance needs to be struck between structured planning imperatives, the reality of the field, time limitations and the local programme staff’s capacity, resources and availability. This could be done in being realistic about the crucial issues to be ready before field deployment and the issues that permit more flexibility. This balance is usually facilitated by the team leader as in most cases team members have different visions of what should be urgent and what is not. 
• First engagements for experts from ISSAT roster: Having a consolidated power point presentation explaining ISSAT’s structure and functions should be made available to all Expert Roster Members on the CoP. Sharing ISSAT contracting procedures and conditions with first-time Expert Roster members before they embark on the mission planning is of capital importance.
• Security Briefings for ISSAT staff: In cases where the mandator does not provide security briefings to its own staff members, higher flexibility from ISSAT might be required. It might need to look for alternatives whether online or through contacting relevant embassies.
• Joint collaboration and consultations between ISSAT mission teams and In-house resource people: Consultations and/or joint working sessions between mission team members and other ISSAT members on specific aspects (governance, gender, strategic planning, etc.) before, during and after field mission can be of high value on the team’s work. Proper channels need to be put in place for such exchanges. Some suggestions can be:
  • Before missions: Working sessions and/or discussions with advisors as appropriate
  • During mission: Email interactions between mission team and Geneva team on targeted issues
  • After mission: Post-mission informal briefings by the mission team to ISSAT team (upon interest). This will help the process of thought of the team before  the report typing phase.

• M&E OGN: The weakness of the OGN’s is that they are only useful to those who have never done a certain process. For experienced practitioners the OGNs are of too basic nature.

• M&E OGN: ISSAT OGN on M&E does not exist. This led to ISSAT Advisor conducting a working session with ISSAT M&E expert to design the framework and approach for the evaluation.

• M&E OGN: It is the team’s opinion that one of ISSAT’s added values is the role it plays in methodology design for practitioners. An M&E OGN would be much needed and appreciated by SSR practitioners.