Approach / Methodology Used

• Time constraints:while prior knowledge of the MoI from ISSAT team and assistance from Sida/SPAP greatly contributed to carry out the field mission, the team had only two weeks in Belgrade, which restricted the number of meetings and activities possible.
• Culture of secrecy: like if by default, all kinds of documents and information are classified in the MoI and it is difficult to navigate a mandate of this nature when the prevailing institutional culture and legal/policy framework is one of secrecy and opacity.
• Empowering while doing no harm– Capacities are uneven among mid- and senior management at the MoI and between units. The imbalance of capacity across units is compounded by uneven attention given to different units by senior managers and donors. In this mandate, the risk was that those units needing the most support would be undermined further by being highlighted as deficient in the report due to the inherent blame culture (the Analytics Directorate being an example). Asymmetrical support to some units while others take on expanding responsibilities posed another challenge: how to reinforce the most performing units (e.g. BSP) whilst not creating too much of a gap in capacity with the rest of the MoI, making them the object of jealously by their peers.
• MoI in transition– the MoI was looking ‘very inwards’ (sic) until Brussels gave its feedback to the screening under Chapter 24, and then preparing Serbia’s follow-up on the results; afterwards, the new Cabinet has been focusing attention in drafting the new Police Law. Discussion on gaps identified by ISSAT will thus depend on the Cabinet being more free of those processes to think on strategic management. The report ‘is a seed that can grow, but in other conditions’, as pointed out in the AAR. (See below also).

• Objectivity: ISSAT mandate was carried out in a way that safeguarded the team and the report against considerations of political bias and confirmed the reputation of the organisation within the MoI as an independent body.
• Contextual knowledge: previous ISSAT mandate with the MoI and in Serbia, and in the region provided a robust knowledge of the realities and issues at stake in this review.
• Balance and relevance:  the report was perceived as addressing all the relevant issues against a benchmark that was rational, balanced and that made justice to the current capacities and overall context of the MoI. This meant in turn that recommendations are credible in that they contain attainable outcomes, when the opposite is quite often true in the experience of the MoI: evaluations against  an ideal scenario with no relation to realities on the ground and existing baselines, thus immediately discarded by those to whom they are addressed.
• Language/Tone: the mandate touched sensitive topics and the report raised sensitive questions, including transparency, stakeholder engagement etc, in a tone that was “diplomatic but straight to the point” to convey the important messages. The subtle but sharp relevant language also contributed to empower without undermining (see below).

Lessons identified for improving the chances of success:

Link-up the review with building the capacity of the institution(s) being assessed:ISSAT team was requested to share its review methodology with the BSP, in order to share additional tools and practices that could provide the Bureau with ideas to inform their role in the context of the Accession process. While the use of such methodology by the Bureau needs further confirmation, the fact that it is perceived as useful to other programmes within the MoI is a relevant ‘collateral’ gain. (See also below).

The mid-term evaluation report delivered by ISSAT was considered very useful, pertinent and objective  by both the mandator and the MoI. Both stakeholders noted nonetheless that the report has yet to produce it’s full impact due to circumstances external to the mission: first, the catastrophic floods that put the country in a status of national emergency for several months; second, the general elections held in March that redrew Serbian political landscape and brought a new party to power. At the MoI, both events took attention and resources away from discussions on strategic management and disrupted ordinary activities. This was compounded by an overriding priority given by the new Government in its first months in office to the revision of the National Programme for the Adoption of theacquis communautaire(NPAA) and the follow-up to the EU screening on Chapter 24 (Serbia recently submitted a new action plan to meet the goals set by Brussels on Justice, Freedom and Security).

Only recently life returned to normal in the Ministry. The action plan for Chapter 24 and the new plan for the Development Strategy provided an opportunity for BSP to use ISSAT’s report within the MoI, at this stage to raise awareness of issues raised in the document and the importance of building capacities in areas like budgeting and strategic analysis. ISSAT findings and recommendations are likely to have a bigger impact long after the mandate finished, into 2015. Indeed, the review seems to be recognised at senior level in the MoI as a reference source to inform planning at the Ministry, in two dimensions that seem interconnected. First, the mandate report is used as a stand-alone document considered for its sensitive analysis and recommendations - even if the latter were not yet or not entirely acted upon. Second, the mandate (report, methodology and ISSAT’s overall approach) left a set of tools and good practice, meaning that ISSAT contribution is clearly understood to have high value beyond the review conducted. New management in the Directorate for Analytics has also enhanced opportunities for the recommendations to actually be taken in consideration in terms of addressing some of the imbalances identified in the report.

The item of capacities for budgetary inputs into strategic planning was not included on the initial Terms of Reference for this review but it was considered relevant during the field mission and included in the report. The resulting remarks and evaluation were credited as being very useful to the MoI, well beyond the new Action Plan recently approved. Adding budgeting to the mandate contributed to raise awareness of the importance of this focus-area at the MoI, as previously mentioned.

ISSAT produced a comprehensive report that evolved from the original request of doing a baseline report to one that looked at the various needs and made substantial recommendations. Note that the term ‘baseline’ suggests that programme areas have been already identified and the study is to see where the process evolved from. This mandate actually needed the programme areas to be identified.

In order to capture a broader array of views, an electronic survey was developed by the team and distributed to key personnel in the Central Apparatus/MIA and the heads and deputy heads of a number of departments within the General Inspectorate of Police, Border Police, and the “Stefan cel Mare” Academy (SCM) following the first main mission.

The findings of the review were considered robust, sharp and timely both by the mandator and the MIA. They point to the exceptional progress made in 18 months in strategic management at the MIA  under the current minister, while identifying gaps in different areas. The report was used by the mandator (Sweden) to develop a programme by DCAF with the MIA, with a suggestion made in the AAR that findings from the baseline assessment can be carried forward by DCAF.

The recommendations are meant to quickly shore up gains obtained so far with the need to sustain reform in the longer term, also against the prospect of a change of tide in Moldovan politics. The question raised in the AAR is whether those recommendations can be implemented in the existing conditions, due to political constraints and limited human and financial resources.

Early on the mission, the team  had discussions with Swedish embassy staff on what the role of ISSAT was regarding capacity building and the question remains if ISSAT’s ‘reinforce not replace’ role was effectively delivered. There were plans to have an afternoon capacity building session with the embassy staff, which never materialised due to time constraints and uneven engagement. From the perspective of the mandator, though, the results were tangible in terms of apprehending methods and tools.

Scoping and survey - The development of a template specific to the mission was vital in order to maintain a coherent framework for the analysis. The template questions were also shared widely so we were better able to manage expectations. The work carried out earlier on in the mandate allowed ISSAT to get to a point towards the end of the mandate where the team had a fairly comprehensive overview of key findings that could be tested out with key interlocutors (e.g. the EU High Level Advisor in the ministry). It also gave the team the confidence to be able to support the idea that detailed design of the subsequent programme could be initiated based on the draft report in order to save time.

Access to information/transparency– The degree of openness at the MIA is remarkable. If anything, there is too much information : over 100 strategies/concept papers across the MIA, all available to be shared. The progress made in transparency is quite extraordinary if we take in consideration that in 2012 the MIA was still perceived as “a closed system that lacked transparency, with Soviet-totalitarian governing practices” (MIA Functional Analysis).

Team composition- The team composition was important. ISSAT experts on team could count on the extensive knowledge of David Clarke, drawing on his understanding of the political undercurrents. This was a great example of ISSAT being able to bring together different resources.

Local support– Excellent support was provided by the advisor to the Minister.

Local knowledge– The mandate gained from the insight gathered by ISSAT during the NORLAM Moldova mission that had started slightly earlier.

Relevance of the report– As mentioned above, dozens of concept papers exist across the MIA, adding to the three functional assessments previously carried out by various donors (including EU). The risk for ISSAT baseline to be redundant was avoided by rooting the findings in very comprehensive information collected from a range of sources and using different tools (including an electronic survey, interviews, document analysis, etc.). From the MIA, the report is perceived as adding depth and scope to the existing analysis.

Credibility– The findings are not disputed in the MIA and the report was ‘generally accepted’, including by those who don’t accept that a review can be performed by a foreign organisation and experts from abroad. Such actors tend, if anything, to question the legitimacy of the approach – not the credibility and validity of the actual findings.

• A good preparation of the mandate, starting with a one-day meeting in Geneva aligning approaches and planning work among Polfed and ISSAT experts on team, starting from scratch and designing the mission.
• Scoping visit from the team leader to Burundi.
• Wealth of local/contextual knowledge in the team.
• Mixed team composition bringing complementary skills and approaches to democratic control mechanisms: technical/Polfed and SSR/ISSAT.
• Trust build over time between ISSAT and SSD in Burundi.
• Good local interaction between SSD, CTB (Belgian development agency) and GIZ (German aid agency).
• ISSAT methodology.
• Facilitation from IGSP staff.

Lessons identified for improving the chances of success:

• • Recommendations can be discussed with local stakeholders in a way that allows for some knowledge transfer on programme design, in the case restricted to the elaboration of an action plan for ‘quick-wins’. Such an exercise can bridge recommendations with whichever projects that SSD decides to carry forward following the review.
  • SSD needs to act on the audit in terms of pushing to have some of the recommendations followed on; this implies that the Programme has to keep representing to the Burundian authorities that crucial decisions have to be taken and operationalised soon to have IGSP fulfilling its important role.

This mandate fully delivered the expected outputs, including an audit report with recommendations in accordance to the mandator’s needs. The recommendations were taken already in consideration and two subsequent missions from Polfed to Burundi worked on how to implement some of them. The reaction to the recommendations was immediate and positive also from the IGSP, which already took suggested points of action in consideration and welcomes the clarification of roles and responsibilities between the Inspectorate and SSD. During the mandate, the position of Inspector-general was vacant but the new head of IGSP took office in the same day of the end of the field mission. A working group was set up just after the restitution session to discuss a set of activities aimed at obtaining quick-wins on the recommendations and that ISSAT proposed to be carried out until Q4 2014.

To this date, the IGSP still didn’t act on most of these actions and further steps depend on the completion of the new legal framework for the IGSP. The Organic Law and the Internal Statute of the IGSP have been drafted and provide finally a legal framework for organising  the work of the Inspectorate, although a decree on the function of internal audit is still to be drafted. Until the legal framework is approved and fully in effect, the scope of activities of IGSP  is nonetheless very limited, specially when it comes to investigations and filing of cases.

The extent to which recommendations are fully implemented will need to be checked in one or two years, since a previous audit to IGSP was basically left in the drawer in terms of acting on suggested points. The review report delivered by the team provides a strong base for the design of SSD projects in support of the IGSP but it also implies and needs that the mandator follows up on the results by pushing for tangible actions (see below).

Throughout the report, attention is given to technical and procedural notions that help unpack the concept of internal control and that lay out a comprehensive set of principles, mechanisms and tools. This corpus can be easily taken as part of training materials for IGSP staff and the PNB overall, as acknowledged by the SSD/MSP coach visiting ISSAT in October to discuss further areas of collaboration related to training in police integrity.

As part of the second/main field mission, in March 2014, a one-day workshop was organised by the team in Bujumbura to conduct a SWOT analysis of the IGSP with a group of fifteen staff of the Inspectorate. The SWOT exercise was considered to provide simple yet powerful analytical tools, with gains in terms of ownership of the review itself.

The evaluation team was assisted in Burundi by two members from IGSP that facilitated contacts and participated in several meetings. Future will tell what will result  from this exposure of IGSP staff to ISSAT’s evaluation methodology and to intense discussions that articulated the principles of internal control with the realities on the ground and the political and social context in Burundi.

For ISSAT, the fact that the evaluation team included two experts working actually in police internal control in their own country provided an opportunity to gain a more technical view of the questions involved. It is considered a relevant gain of capacity for future reviews and an important enhancement of existing in-house knowledge on police reform, integrity and responsibility.

It is the perception of the IGSP that the mandate resulted in opening some doors to potential collaboration of the institution with civil society organisations.

DFID n’a pas de réelle capacité opérationnelle pour mener ce type de mandat. DFID n’a sans doute pas non plus la volonté d’obtenir cette capacité en interne. DFID externalise toujours ce type de mandat et dès lors utilise ISSAT comme un contractant. Ce mandat a donc principalement été pour remplacer les mandataires et non pas les renforcer.

Néanmoins, le renforcement s’est fait au travers du travail rapproché de l’équipe d’ISSAT avec le staff de DFID et UE, individuellement.

De plus, DFID est satisfait d’utiliser ISSAT comme contractant pour certains mandats en RDC, puisque ISSAT a une bonne réputation, provenant de ses compétences et son expérience des évaluations annuelles du programme SSAPR. ISSAT est respecté auprès des interlocuteurs internationaux et nationaux. ISSAT est reconnu comme indépendant et donc utile pour mener à bien des évaluations. Néanmoins, le présent mandat n’était pas une évaluation indépendante, mais bien une étude préalable pour identifier les options pour les futurs programmes soutenus par DFID et UE. L’aspect d’indépendance est moins essentiel, mais DFID a vu ce mandat comme les autres. Il n’y avait donc que très peu d’options pour réellement renforcer et travailler main dans la main avec DFID et l’UE.

Leçons identifiées:

-         Mentionner clairement dans les TdR qu’ISSAT fait partie intégrante de l’équipe du mandatairesur le terrain pour la durée du mandat.

-         Discuter avec DFID HQ sur leur volonté d’obtenir d’ISSAT un renforcement de leur capacité, de quelle manière et dans quelle situation. Pour un mandat d’évaluation, l’indépendance peut être un aspect crucial. Mais pour d’autres types de mandat, ceci ne doit pas spécialement être le cas.

-         Travailler main dans la main avec le staff des mandataires sur le terrain. Mentionner dans les TdR certaines activités de renforcement des capacités lorsque l’équipe est déployée sur le terrain.

Capacity building activities were mainly carried out individually, through the participation to the field mission of one staff from Norad. It was clear from the start that someone on the mandator side had to participate to the mandate and it proved to be very useful. The evaluation methodology was not new to the mandator but applying it made the terms and concepts more understandable. The mandate took a structured and systematic approach in collecting and sharing the information, with day-to-day team briefings, planning the interview questions ahead of time, etc. This was considered to be very good. The involvement of Norad staff was limited to the field mission. It could have been better if he would have been involved also during the planning and reporting phases.

The role of ISSAT in terms of capacity building was not clear. It should have been clarified since the beginning in the ToR. NORLAM staff could have benefitted from capacity building activities on the methodology and the use of terms such as output, outcome, impact, etc. Moreover, they do not have the in-house expertise to implement some of the recommendations on strategic planning, project design, etc.

NORLAM staff considered that the ISSAT field mission was very time consuming in terms of planning interviews and logistics. They did not expect that at all and were very busy with their day-to-day work. The mission added a lot of work on top of everything else. Even though the ToR clarifies that “Overall responsibility for co-ordination of the team’s activities on the ground will be provided by NORLAM. i.e. making appointments, providing transport, etc.”, this investment should have been better communicated and someone should have been dedicated to that. There seems to be some confusion on the role of the local expert, part of the review team, in terms of planning and logistics.

It is recognised that time was a big constraint for this mandate. The field mission was very intense with a lot of interviews and only 10 days in Moldova. But this mandate was very well organised and very well led by the team leader. Team members were impressed by that. An additional 2 days on the ground could have been foreseen in order to collect additional information or meet again with some interlocutors to clarify certain issues.

Lessons identified on capacity building:

• Include a staff member from the mandator side since the beginning of the mandate, from the planning to the reporting phase. Clarify her/his role in the ToR as well as capacity building activities to be undertaken during the mandate.
• Propose a workshop on the evaluation methodology to the mandator staff in the field. This could be done in half a day at the beginning of the field mission.
• Clarify and communicate as much as possible on who should plan interviews and take care of the logistics. Recognise that it takes time and plan that well in advance (3 weeks before the deployment of the team in the field).
• If possible, leave two free days at the end of the field mission in order to be flexible to investigate further certain issues and plan additional interviews.

All the outputs foreseen in the ToR were achieved and the mandator was overall very happy with the work carried out by the review team. Some outcomes were also achieved (see LI form in annex here below) but some others were only partially achieved such as the clarification on the future scope of NORLAM after 2016, the status of a police component and the length of extension of NORLAM. It was recognised however that the ToR were very ambitious and Norway wanted to achieve too much in one review over a short period of time. Prioritisation of the expected outcomes could have been better and communication between ISSAT and Norway when drafting the ToR could have been more intense. The Oslo mission carried out during the planning phase of this mandate was very useful, but it took place a bit too late in the process. The ToR were already approved. However, amendments were still made to the ToR after the Oslo mission which were considered useful.

To help prioritise the ToR, the review team could have also communicated better and earlier on the methodology to be used for the review.

Norway and NORLAM are using the conclusions and the recommendations of the report. It is considered to be very important for them. The report played a major role in the discussion between the Norwegian Ministry of Justice and Ministry of Foreign Affairs to decide whether or not to extend NORLAM. Moreover, the NORLAM team organised an internal seminar in mid-September to review the report. They analysed the findings and conclusions and came up with a list of action points. This exercise was considered to be very useful for discussion and ownership of the report. A second internal seminar should take place soon to discuss more in depth the recommendations. In November, the Norwegian Ministry of Justice will visit Moldova to discuss the post-2016 NORLAM, on the basis, among others, of the conclusions and recommendations of the report.

Some recommendations are however considered to be too vague (e.g. what does it mean concretely to develop NORLAM Theory of Change, cf. Recommendation 4). Some others are difficult to follow such as the one on the legal status of NORLAM, because it needs political agreement and decision that are not easy to obtain. Nevertheless, having such recommendations in a written report, done by an external and international actor such as ISSAT, is considered to be very helpful and can be used as a political leverage for discussion with Moldovan authorities.

Lessons identified on output and outcomes:

• Ensure good communication with the mandator during the drafting of the ToR in order to help prioritise the expected outcomes and the scope of the work.
• Try to undertake a scoping mission earlier in the process so that discussions on the ToR can take place with the mandator. Overall the scoping mission is a very useful step in the planning phase of the mandate.
• Include in the ToR a section on the evaluation methodology to be used during the mandate.
• Develop as much as possible concrete recommendations. General recommendations could be unpacked with concrete actions to be considered by the mandator.
• Continue building ISSAT reputation in terms of being a recognised international actor with strong SSR expertise and evaluation methodology.

The approach to developing the curriculum, although condensed and would have benefited from a more drawn out process was quite systematic in the areas it covered, focusing on the body of knowledge that could be developed into course content, as well as tools for implementing the curriculum.

Preparation time on the part of facilitators and political will on the part of the SPLA were critical to the workshop meeting its goal. 

The value of political will on the part of national partners cannot be underestimated in the success of a mission. The higher the buy-in from senior officials for a mission, the more engaged nation counterparts who are participating in a mission will be.

A large body of work such as a curriculum should be done in a multi-staged process, with different leads and working sessions to address the topics that were descried under the Approach section in Step 4. Topics such as the development of a taxonomy and knowledge management can be complex and require time for national stakeholders to build up their capacity in these technical areas. The workshop which was conducted would have profited more from being an opportunity to amalgamate and consolidate the work of previous stages.

Number of participants: The large number of participants affected group work and it took a little longer for some of them to become comfortable enough to contribute fully.  

Length of working schedules: There were two lunchtime speakers, whose contribution impinged upon the participants’ downtime, and contributed to some long days.  The individual sessions were normally for 90 minutes, and for a non-native English speaker, this was a long time to concentrate.  

The donor’s SSR knowledge: The donor’s considerable background knowledge of SSR, as well as, strong commitment to the concept, ensured that local support to the course was willing and responsive.

Balanced groups: Much time was spent on ensuring that there were several well-balanced groups for the exercises.  In addition, several ad hoc groups were also formed in order to cut down on the time spent in reporting back in plenary.  This combination worked extremely well and helped break down barriers and increase interaction.  

Facilitation strategies: To counter-balance the disadvantage of the big number of participants, the team leader broke down each of the sessions into a combination of talking from the front and exercises, and used up to four different individuals in order to provide variety in both voice and style.  This worked well for most of the time. 

Group ownership: The final exercise involved the participants breaking away in their thematic to be an extremely valuable experience.  Not only did they have responsibility (and thus buy-in) for the final product, it also helped to cement relationships amongst individuals within the group and thus networking.  

Distribution of Course Material:  The distribution of course material, including all the presentations and the ISSAT Handbook, to all the participants at the end of the training event was an excellent idea.  

Lunch breaks: It is recommended that a full hour is allocated for lunch so that the participants have an opportunity to re-charge their batteries.

• The mandate was complex owing to the sheer scale/volume of countries to be visited and the insistence of the EU on “regional” priorities over “national” priorities. It was exceedingly difficult in many cases to identify regional priorities in EU country offices since most of them are understandably preoccupied with “national” concerns. Moreover, EU investments in regional programs are often among the weakest and least impressive in terms of performance. The EU’s relatively limited investment in security and justice also meant that the level of awareness/understanding was comparatively low. 

• The missions often involved reviewing both regional and national programs. There was a risk of inflating expectations of EU counterparts, but this could not be avoided. Likewise, the mission reports were shared with EU counterparts for their review and comment – seen as essential in building up good will between Brussels and country offices. As such the mission was concerned with both the tangible output (a report) as well as process outcomes (improved relations and sensitization of JSSR priorities in the LAC country offices). 

• Greater negotiation with counterparts to ensure that there is an adequate balance of HQ and country office concerns in the ToR. The risk is that external consultants can have their methodology undermined if the mission agenda is not at all aligned with country office priorities and preoccupations. Also, a short “fact sheet” on JSSR could be usefully circulated to country offices in advance so they are prepared for visiting consultants. This would obviate the need for extensive explanations of what was/was not JSSR.

• The report from the previous assessment mission to Kenya should have been shared before the design mission deployed (as indicated in team ToRs). This would have better informed the process and better clarified the objective of the mission. This was not done because of SIDA’s bureaucracy measures and the person in charge being too busy with other issues. Also team is not certain that SIDA understood the importance of sending out the report on time.
• This design mission could have been better informed by the previous assessment mission had the assessment mission been focused on mapping out and analyzing the needs and challenges. Due to the political process at that time, the assessment was anchored around finding out if Sweden should continue its engagement in Kenya without going into the details of such an engagement. In reality, the team had suggested sending out a scoping mission in between the assessment and design missions to identify the main challenges that would then inform the design process. This suggestion was not adopted by SIDA. As a result the already limited time dedicated to the design process had to include an assessment of challenges.
• Some team members would have appreciated a proper methodological framework for this mandate similar to the evaluation methodology.
• The time SIDA is taking to approve the programme document goes beyond the overall timing given to the team to have the document ready for implementation.
• One of the criticisms SIDA has formulated concerning the design document is about its local ownership when in reality the lack of local ownership is a result of the limited timing SIDA allowed for this process.

• The workshop was a big success. Participants were engaged throughout the whole period and gave their very best efforts to complete the document despite of limited timing. However, participants were mostly people who were already on board in terms of understanding community policing concept and its application in Kenya.  Participation from police officials would have been a better indicator of their buy-in into the process.

• ISSAT is working on Programme Design methodology

• There was insufficient ownership by the Ministry resulting in them not taking responsibility and not showing up. Non-state actors were entirely missing.
• Unclear what follow-up action was taken and how the process continues. There is lack of communication from UNSMIL on follow-up and next steps. 

• The review teams were made up of a big number of people whose time was not dedicated 100% to the process. As a result, ISSAT had to hold the pen and lead the process forward; although they were supposed to be facilitators of the process with the DRSG office and the Justice and Security Board acting as overall monitoring bodies.
• What also contributed to that fact was that ISSAT was full time committed to this process and was able to attend all meetings which was not the case of the others.
• ISSAT had the technical responsibility of delivering the product and so could not risk not taking the lead on the process.
• The big size of the team was not operational in some cases. Too many people were involved with no clear indication as to how to move forward. Action plans for each sub-team should have been done as mentioned in the ToRs.
• UNDP/UNMIL did not spend enough time ironing out the linkages between police-justice and prosecution.
• Preparation by the UN should have been more in-depth, the overall objective of the mission should have been better clarified.

• ToRs were extremely exhaustive and clear. However they kept to the overall framework which was their added-value once team arrived to the field and gave them enough flexibility to adapt them to their work.
• ToRs were shared with Sollicitor General and he commented on them.
• ToRs were also approved by the Justice and Security Board which had national counterparts on it which secured some level of commitment since the outset.
• There were three teams handling the different aspects of the review: police, prosecution and judiciary.

• Should have organised one session (1/2 day) for approach planning to have everybody on the same page (some team members were not even aware of what was governance of oversight).
• The process of selection of team members was unclear.
• Each team should have had one person helping on schedule planning. The police team had one and the prosecution partly had one but the judiciary team had none.

• Did this mission have an explicit methodology? 
  • Yes

• The questionnaires and surveys were built based on the ToRs. Once team arrived to the field, they had to adapt considerably those tools. These tools were very much focused on the quantitative aspect of things and as the team understood the situation better, they were able to adapt to the qualitative aspects of the evaluation.

• The ToC developed by the team was very useful to capture in a nutshell the logic of the programme, in particular for the team members who were not the in the loop of the programme since the start.
• Having developed the methodology beforehand and designed the tools helped maintain a firm sense of objectivity for the evaluation. It helped also maintain a strong scientific framework for the evaluation ensuring that judgement and recommendations only be made based on concrete proof.
• Team ensured regular end-of-day team meetings to compare notes and debrief. This allowed re-adapting the approach, balance evidence-gathering, ensure the absence of gaps and work on contradictions.
• Team prepared a document that would be filled with preliminary findings gradually after every team meeting.
• Team did first draft of report before they left the country. During a team meeting they finalized the answers to most of the evaluation questions. They harmonized everybody’s views on all issues. The following day, team worked on editing and tightening the language.
• The methodology designed for this evaluation helped design the structure for the report.
• The external member of the team, led the team debriefs which reinforced the objectivity of points put across and registered as preliminary findings. 

• Did this mission have an explicit methodology?
  • Yes
• Where linkages between security, justice and governance looked into? 

• PCRs are a heavy bureaucratic process
• A wide Security and Justice Reform strategy for UK in South Sudan does not exist

• The PCR is an explicit process for DfID. This process was articulated to the team by DfID and was fairly straight forward. 

• The main deliverable from this event kept shifting in terms of its shape and scope. Initially it was meant to be a report, then AfDB requested to have published as a publication, then more issues were added to the original document. It was challenging to finalize this document and get consolidated clarity on how it should look like and what issues it should cover.
• The fact that the training was happening in parallel with the HLP, some participants started alternating between the various sessions that they saw interesting between one event and the other.
• AfDB did not have concise clarity about what they wanted out of this training. As a result, ISSAT had to be extremely flexible in catering for AfDB’s shifting ideas on what to include and what not to include in the training.
• The challenge for AfDB was that the Nairobi training was a next step after the Tunis training which was also a L1. Half the participants from Tunis were participating in the second training. AfDB did not want that workshop to be repetitive for those participants.
• However, eventually only 2 or 3 people from the Tunis training ended up returning to the Nairobi training.
• Some of the objectives that the training was supposed to achieve were not thoroughly achieved, for ex: Discussing and establishing clarity on AfDB SSR approach, or looking at operationalizing SSR tools by the AfDB.
• Training team had a hard time identifying linkages between the AfDB’s work and teh training objectives on SSR.
• As a result of some of the factors listed above, this workshop felt less of a training and more of a SSR seminar.
• ISSAT should have controlled better what was going to be said in the opening statement for this training and for how long (ISSAT should have written the speech) since it kicked off the training on a negative note.
• Training support material should be available in French.

• Participants found it very interesting to have the HLP running at the same time. The HLP allowed for a timely initial induction for participants so when the training actually started, they already had an idea about the issues at hand. It also allowed for a window to discuss big picture issues and expand the participants’ understanding of SSR.
• The HLP raised the level of debate for the training and participants were able to debate on issues that they would have been able to do had the HLP not happened.
• It also gave a lot of legitimacy to the trainers and they did not have to look for examples on everything they presented.
• In terms of sensitization/awareness-raising, ISSAT did achieve some progress on this training vis à vis AfDB.
• Case studies and facilitation skills worked well in AfDB PoC’s point of view (Christina).

• A discussion has to happen between ISSAT (Mark D., Eleanor) and AfDB (Christina) to clarify clear objectives or milestones for future ISSAT-AfDB trainings.
• Governance and justice are the key thematic areas for AfDB this should be used when talking about SSR with them. So is conflict prevention, World development report, civil society...so something that is closer to preventive SSR.

• ToRs: The Mandators hardly contributed to the design of the ToRs; the design process was completely left for ISSAT to undertake. 

• Joint mandate:This was a joint-mandate mission given to ISSAT by two mandators.
• ToRs: Team consulted in ToRs design

• ToRs: This mission contributes to a long register of After Action Review highlighting the importance of ToRs as a “framework of protection” for ISSAT Advisors once on the field. Clear and detailed  ToRs with specific roles and deliverables are key for ISSAT Advisors, especially when the mission is in highly politically-charged contexts (as per the mission in Guinea Conakry in March 2012) or when the mission team is deployed within teams with very poor leadership (as per this mission and the Evaluation mission in Congo in November 2010).
• Team meetings:End-of-day team debriefings are a key practice that has proved its success in all previous missions. This is something that needs to continue to be implemented. However, they need to be structured and times so as not to become a useless waste of time. Also, team leader should make sure that structured internal discussions within team need to be achieved before meeting with external partners. The objective here is to harmonize teams understanding of objectives, expectations and challenges of the interactions with the mission’s partners.

• Process and timing of methodology development:
  • In theory, an overall methodology had to be designed by the lead expert for this multi-country review. In reality, no process was put in place to support methodology design and identify the timing and synchronisation of the field missions, in spite of the fact that it was initially envisaged in the mandate request. As a result, whilst the methodology was still not developed, field missions were already underway usingadhocapproaches. This meant that no specific stakeholder was identified to be systematically interviewed across the countries. In the case of Liberia, the choice of meetings was problematic: A few key meetings were not arranged whilst meetings with individuals who were only marginally relevant were. This was due to meeting requests being made with too short a notice and the support team not explaining the meeting objectives in a very compelling way.
  • This links back to short notice given to field-based mandator office (Liberia PBO) and its limited capacity to respond to requests with tight timelines (PBO’s capacity is very limited beyond its Director who was out of the country at the time).
  • ISSAT Advisor received an initial question set while en-route to Liberia. The methodology of the mission was finalised when ISSAT Advisor was on the way back from the mission. Luckily, the list of questions that the Advisor used to collect data was not altered.
  • Initial draft program for ISSAT Advisors meetings was received from PBO on the morning of departure for Monrovia.
  • The mission programme of meetings was problematic. Sequence of meetings was inappropriate including sometimes over-lappings and unstrategic meetings. Appointments requested by ISSAT Advisor were not taken into account and key respondents did not appear on it. In spite of a request by ISSAT Advisor seeking some adjustments, no changes were made by the time ISSAT Advisor landed in Monrovia. This was due to the support-team’s lack in human resources to be able to respond in effective and timely manner to last-minute requests. They were not given the adequate time to understand the purpose and objectives of the mission.

• Background documents selection and stakeholders to involve in substantive mission planning:
  • While PBSO uploaded a large number of Liberia/PBF related documents on Basecamp, some important documents for the SSR Thematic Reviews were not posted (such as the Liberia PBF mid-term review). This may have been due to the fact that PBSO personnel familiar with Liberia were only involved very marginally and at a very late stage in the preparation of the mission.

• Process and timing of methodology development:
  • ISSAT Advisor had to mitigate the absence of a process for methodology development by assuming that the question set would remain unchanged even though it could have been improved.
  • Support from ISSAT Project Officer in preparation of one of the annexes to complement the Burundi Case Study was useful in keeping the agreed timelines for this product.

• Process and timing of methodology development:
  • The need for solid pre-mission planning is of capital importance since it allows the possibility of developing a sound methodological framework for the mission and thus directly impact its degree of effectiveness.

• Mission conception and ToRs design:
  • There was probably a disconnect between ISSAT and SSDO on the understood core expectations of the mission. The team is in need for more fundamental capacity strengthening that a two-week mission cannot address.
  • The initial ToRs that were developed by the mandator for this mission emphasized to large degree the concrete product of developing Quick Impact Projects (QIPS). ISSAT did not have a clear idea pertaining to the lack in capacities of the mandator’s staff and the need for higher emphasis for longer term capacity development to ensure ownership of the QIPs and sustainability of such an initiative.
  • ISSAT was informed of the above gap once the Advisor was deployed to the field. As a result, the ToRs were amended to separate the capacity development and the QIPs as two distinct objectives with different timelines.
  •  Through continuous communication between the mandator on the field, ISSAT and the Advisor, expectations were managed to allow for more realistic objectives for this mission and the definition of future support to the mandator on the longer term.

• Delivering on outputs and objectives with very little capacity and resources from mandator’s side:
  • ISSAT and the Advisor deployed to support this mission had to strike a balance between delivering the products of this mission (finalised QIPs) and the raising awareness on the importance of process-oriented approaches for strengthening capacities and allowing for ownership and sustainability.
  • Mandator staff lacked basic strategic planning and project management skills. ISSAT Advisor spent her time trying to get more information on what processes, templates, funds exist for designing the QIPs, which was very difficult to obtain and sometimes nonexistent.

• Mission conception and ToRs design:
  • Flexibility from ISSAT’s side to discuss and rephrase the ToR according to the actual situation on the ground was crucial for being able to deliver needed outputs.  

• Delivering on outputs and objectives with very little capacity and resources from mandator’s side:
  • This balance was found by facilitating towards a collaborated approach to design frameworks for the QIPs as a first step. This allowed for as much as possible capacity building for the staff on the field as well as the production of concrete  products that they could develop in the future
  • Working with the gender team to develop their QIPs was relatively easier than the other teams. This might have been because the gender team was composed of civilians who were more comfortable in project management approaches, in addition to their interest in obtaining funding for their initiatives. This was a department where ISSAT’s support was effective.

• Mission conception and ToRs design:
  • A recce mission before the development of the ToRs and agreement on mission conception would have been very useful in this case. When ISSAT is not very familiar with challenges, capacities and resources on the ground, it is preferable to conduct areccemission before agreeing to any support, in order to get a realistic understanding of the situation at hand. Through mere emails exchanges, it is difficult to convey more than the tip of the iceberg where as the major part remains hidden until field deployment.
  • Working with the mandator’s team may be challenging for various personality-related and institutional reasons. However, it is also fascinating from a SSR perspective, probably making it worthwhile for ISSAT to remain engaged. 

• Delivering on outputs and objectives with very little capacity and resources from mandator’s side / ISSAT’s future role in Somalia:
  • Any future support from ISSAT to the mandator’s office should take into consideration the huge challenge related to capacity and thus must focus on awareness raising and capacity development of relevant parties to allow for change before stepping up any delivery of concrete products since this would be equivalent to replacing the capacities on the ground as opposed to reinforcing them which is contrary to ISSAT’s mandate.
  • The mandator’s office is in need of capacity building on strategic planning which is a huge gap where ISSAT could add value. This is itself could be a challenge given that the staff turnover rate is relatively high.
  • ISSAT might have an added value in lobbying to raise awareness of parties such as its Governing Board’s UN members or UN HQ agencies for strategic interventions on their side to change the situation of UN Office for Somalia which is in need of proper staffing and resources before engaging in any other support operations. This could be complemented by a contribution from ISSAT’s side in the form of a mapping or a study of the capacities and resources available to UNPOS, their mission and objectives and highlighting the gaps and the needs for an effective delivery of expected objectives and outputs.

• ToRs design and consolidation: The ToRs for this mission were put together in a hasty manner, a week before the mission, which did not permit the team to properly own them before departure on mission.
• Clarity on objective and outcome of a mission: The team didn't have much clarity as to the objective and overarching expected outcomes of their engagement with DfID on their strategic engagement with South Sudan, including linkages with national processes and engagement with other international regional actors and donors. Clarity on the “big picture” was somehow lacking in this mission.
• Lack of holistic approach to donor’s programming:
  • One of the team’s main concerns was a lack of an overarching HMG Security & Justice Strategy for South Sudan.  The existence of this would have made it easier to understand the impact required from a continuing intervention.  In general, it is important that donor strategy is evident before commencing on such a project.
  • As a result of the fragmented approach to SSR by HMG’s programmes in South Sudan (separating between defence, justice and policing aspects), this mission was conducted separately from another assessment focused on access to justice. Given the links between the two, it would have been useful to link up the missions and either have a joint mission or at least to have conducted them concurrently.
• Donor’s processes and templates (Business Case approach):
  • The current BC process is cumbersome.  This current version was updated by DFID in 2011 and has done little to make the process more logical.  The process demands that a Logframe is submitted along with the BC which seems to be pointless because, at the point of producing the BC, there is no programme.  This follows later and, at that point, a Logframe will have to be produced anyway.  The BC process as it stands does not allow for a logical flow of information and, to a certain extent, one is forced to provide information to fit the “box” structure.  Also, there is no obvious way in which to present, and argue the case for, the preferred option.
  • The Business Case approach is a very specific structure that requires some training as to how to use it and adapt it to the needs. Neither of the Advisors had previously worked with that format. As a result, they would have appreciated clearance on the approach and access to a set of examples (such as the Malawi Business Case) to understand better how to work with this model. Unclear section complicated progress for the team (such as the unclear difference between the sections: “Strategic Case” and the “Appraisal Case”).

• ToRs design and consolidation: Despite of the missed opportunity for the team to feed into the ToRs design, as the process moved forward, the team was able to revisit them and adapt them to the context.
• Operational collaboration with justice assessment team: The Business Case team on defence and security proactively engaged itself in collaborating with the access to justice assessment team. They communicated to them any relevant information or data that would be useful for their Business Case. The team shared contacts and knowledge and tried to minimise the repetition of meetings in country that was very likely.
• Feedback to decision-makers on holistic approach: The assessment team did feed back to HMG its observations on the challenges of separating army, justice and police aspects into three different programmes, highlighting the importance of holistic approaches.
• Donor’s processes and templates (Business Case approach): The Business case format highlights specific sections that need attention and pushes users to answer directly the issues put forward by the template. This approach allows for assessment reports that are straight to the point and that answer all the aspects of interest to the donor as opposed to putting information that is of weak relevance or does not answer a specific need, thus maximising the level of consistency of the report (for example, it obliges the team to go through the programme’s Theory of Change, it calls for evidence based analysis etc.).
• Using contextual knowledge and experience to find out what works: The team had significant recent cross-sector country and regional experience which they used to bring up related positive examples of what had worked in South Sudan in terms of SSR.
• Sessions for joint team work: These sessions were most valuable to the team and allowed it to achieve important progress intellectually. The team needed to know enough collectively since each member came from a different background and harmonization of understating amongst them was key taking into consideration the limited time for planning they had.
• SSR in general: SSR programming and approaches have come a long way since 10 years ago. On the policy level, things are on the right track in terms of considering linkages between security, justice and governance issues. Programmatically and logistically more efforts are needed to achieve the needed balance among the constituents of the security sector. DfID has also achieved a lot in that direction; it has now one unit that deals with security and justice issues (the Security and Justice Unit). 

• ToRs design and consolidation: The ToRs design for such missions should be initiated at least a month before the mission to allow all relevant stakeholders to feed into them and to tailor them to the needs of the mission.
• Clarity on objective and outcome of a mission: The team would have appreciated gaining more clarity on the overarching objectives of their mission and contribution to DfID’s partnership with South Sudan. These issues could have been highlighted in the ToRs or in a separate document sharing the “statement of intent” about this engagement.
• Donor’s processes and templates (Business Case approach):A briefing session should have been organised for the assessment team before their deployment on the approach, terminology and usage of the Business Case. Since the “Business Case” approach and format were not known to any of the team members. It would have also been useful for them to have access to reliable examples of Business Cases available at the beginning of the mission.
• In addition to the above, a briefing session for DfID’s partners in country on the Business Case approach would also be of high importance.It was noted during the mission that each stakeholder had a different conception of this approach and the terminology was not always accessible to them. Re-examine Business Case structure and guidance notes. Ensure that all stakeholders are involved from the beginning of the process and that everybody knows what the objective is.  It is advisable to refrain from engagement until national and international rationale for potential engagement is clearly stated and understood at all appropriate levels “Sequencing” is important– there is always a preferred order to engagement and, in this case, (if we had enough time) it would have made sense to start with the relevant parts of HMG in London.
• Lack of holistic approach to donor’s programming: Earlier discussions in donor capital with the relevant departments always help clarify visions and approaches when an overall framework for the donor’s interventions does not exist or is unclear.
• Sessions for joint team work: It would have been useful to have designed, in advance, blocks of time to work collectively as a team and other blocks of time reserved for interviews and consultations. This would have identified certain milestones and the team would have been obligated to stop gathering information and start analyzing and synthesizing what they gathered.

- The number of interviews was too limited for a thorough analysis. The report mainly led to a first overview of the problems but lacks depth.

- Excellent approach. Well prepared. Presentations on the right level. 

Have realistic timelines, report writing always takes longer than anticipated: Having adequate time available both for the planning phase and on the ground is important, but an area that was under estimated was the time taken to write and review the report. There is also a need to recognize that such assessment activities are costly and best carried out as part of ‘joint’ approach between the government and the international community.

- Go beyond the elite’s to try and understand the people needs: It is always difficult and a challenge to understand ‘people’s security and justice’ needs in such a short period of time. Use of focus groups including events outside the capital, national researchers, having access to national polling data can all help.

- Create a multi-disciplinary team: The team brought together expertise that included policing, justice and management experts. The team also included a national team member, who helped with context, contacts and an in-depth understanding of the national reform process. This was invaluable.

- Understanding the context, political decision making, social structure is key: The approaching taken focused initially on understanding the structure and nature of the state and the dynamic of power relations across government and within the police and justice sectors. It focused on how the public access security and justice services and where there was room for improvement.

- Include national members of the team where possible: A lesson from other missions that could have been utilized was the inclusion of liaison officers from the police and justice sectors into the team. 

- Sustaining the early disposition to accept the methods agreed to.

- As far as methodology, as lead expert, my proposals were accepted, in general.

- It would help to assign to the task of liaising with expert team someone who has some understanding of SSR generally.

- The approach was well adapted to the circumstances: that of a country and a parliament which are mostly still unaware of the possibility of security sector reform. 

- All reports were circulated through the line ministry. However, it was felt that its role in circulating information within the State administration could have been much more proactive. The ASP were much more enthusiastic in sharing their views, but were hampered by protocol as they needed to wait for the official version of the documents to be sent to them by the ministry.

- The final programme document specified the desired outcomes and impact of the programme, and activities, outputs and outcomes of the component activities. It did not, however, specify how those activities should be carried out.

- The briefings from an anthropologist specialising on Albania were considered to be very worthwhile, as it gave insight on more sensitive subjects (Inter-agency relationships).

- Care was taken to ensure that as many groups as possible were involved in the design project, as it started the process to build relationships and partnerships, which is an integral part of Community Policing.

- Arranging meetings was througha combination of formal and informal approaches. At times formal approaches were needed, although the arrangements had already been made through informal channels.

- All of the workshops were run in Albanian. Translation was used where necessary for the plenary activities, but all the working groups were facilitated by Albanian members of the team

- When circulated, the official version of the reports were Albanian, with the English version provided as an addition.

- Assessments should not just be considered as extracting information, but can also provide a capacity-building experience, for example where stakeholders meet, discuss and learn from each other.

- Care must be taken to prevent potential implementing parties from influencing the final version of the programme proposal so that it fits with what they can deliver. Turning a demand-driven process into a supply-driven process at the last minute undermines the integrity of design, national ownership and efforts to meet the genuine needs of the population.

-ToRs for programme design must set out clear parameters and milestones, but at the same time remain flexible to allow the team to develop a process that takes into account how information is being processed, analysed and decided upon, in order to take into account contextual developments.

- Enhance awareness of the local context and culture through engaging an anthropologist during the process.

- Consider the process used to develop the programme as both capacity building and an opportunity to build partnerships.

- Wherever possible, produce documents supporting the design process in the local language in order to allow the national decision-makers to take the lead in selecting options etc. Factor in the need for translation services into the planning and budget.

- The programme document must specify clear results required, and detail supporting activities. Potential implementers are responsible for determining how the activities would be carried out; hence principles of implementation must be included (e.g. wide participation, national ownership/partnership, monitoring and reporting, etc.). 

The outputs of the mandate (Annual Review narrative report and report as per DFID template) were delivered in time and with quality. Considering the very short time allocated to this mandate, the mandator (DFID SL) and the main recipient of the review (the implementing partner) were satisfied with the reports.

In terms of outcomes, they were not explicitly mentioned in the ToR. However, the AAR discussion moved towards looking at whether the recommendations already had an effect on the main recipients of the Annual Review. Looking at the outcomes in those terms, it was a bit too early to assess them, but DFID SL already reported a positive shift in the relationship between the implementing partners and senior level staff from the government of SL (which was one of the recommendation made by the review team).

Lessons identified for improving the chances of implementing the recommendations:

• The “exit” presentation of the review team at the end of the field mission should be longer than two hours. Ideally, it should take the form of a workshop/brainstorming of one day with the main recipients of the Annual Review and the review team.  It should focus on the recommendations and concrete options for implementing them.
   
• The recommendations should be more specific and actionable.
   
• The review team should work more closely with the implementation team so that they can learn from the review team’s experience and expertise.
   
• (from ISSAT’s own experience: if possible, plan a follow-up field mission one or two months after the submission of the (draft) report to work specifically on the recommendations with the implementation team).

Joint teams (international/national representatives): The Programme manager designed a theoretically very useful and positive approach for the evaluation which consists of deploying a national Security Sector representative to be associated to each advisor during the evaluation process. The objectives of such an approach were to –on the one hand- allow the Burundian counterparts to learn through actively participating in the exercise; on the other hand, to allow the evaluation to be anchored in solid knowledge of the context and the realities of the country’s security sector.

This however had varied implications in practice. Due to the absence of a clear structure for the national team members identifying their roles and deliverables, their contribution to the evaluation depended very much on variables such as: their relationship with the international advisor, personal dynamics, their knowledge of the process etc.

As a result in some cases they contributed by providing a local view and value to the evaluation and in other cases, they played the roles of logistics coordinators rather than contributing substantially to the evaluation.

This was due to the following factors:

•  
  • The Terms of Reference for the mission, do include the designation of the Burundian team members and describe their role in a rather superficial way, not detailing what is to be expected from them after the desk review is done and the stakeholders identified;
  • After the three Burundian team members were designated by their respective institutions (police and armed forces), the team was not aware of any follow-up meeting with these individuals to brief them on their role and prepare them to proactively contribute to the design of the methodology.
  • The three Burundian team members were associated to an advisor in a way to allow the police officer to be on the pillar evaluating the army and vice versa. This did not work well in practice as the army officers were sometimes not comfortable speaking infront of the police representatives and neither were the police officers to the armed forces’ representatives. This led to instances where the Burundian team member was prevented to come to the meeting, or in some cases, to leave the room, while the international advisor carried on the interview regarding more sensitive issues.
  • The Burundian team members did not feel at ease or have the sufficient confidence in their role to be proactive in contributing to the evaluation. One of the Burundian officers did contribute slightly to asking questions but that was since he knew the international advisor well and felt comfortable enough to put questions forward.
  • Since the Burundians were not implicated in the process from the start, their learning during the interviews was very limited as they did not have the overall vision of the underlying approach and the long term objective of these questions.  

• Joint collaboration and consultations between ISSAT mission teams and In-house resource people: For this mission ISSAT M&E Expert supported directly the design of the evaluation’s methodology. This exercise was useful in two ways: It helped build the capacity of the ISSAT Advisor who had thematic expertise (Police Reform) but not the M&E knowledge and skills. The second benefit was related to the team acquiring an overarching guiding framework that facilitated their mission considerably. Without this preparatory exercise, the team could not have been able to deal with the complexities of the mission in a coherent manner which gave them the ability to respond to the challenges of the programme (absence of criteria for success, indicators, etc.) without losing sight of their objectives and without losing too much time in attempting to clarify how these considerations fit together. 
• Design of methodology and structure of the report before deployment: Having designed the structure of the report before deploying was also very helpful to the team as they could very rapidly and with ease collate the information they got out of their interviews and analyse them. This structure also facilitated the exchange of information amongst team members and their collaboration in terms of which piece of information fits where on the skeleton of the evaluation’s report.

•  In-depth exchanges with field programme leads and staff:The team leader for this mission reserved a considerable amount of time to work with the programme lead in the country. The objectives of these working sessions were to ensure that the mission team and the programme team had a coherent understanding and the same expectations from the evaluation. The draft ToRs were used as basis for discussion.  

• Joint collaboration and consultations between ISSAT mission teams and In-house resource people:  Consultations and/or joint working sessions between mission team members and other ISSAT members on specific aspects (governance, gender, strategic planning, etc.) before, during and after field mission can be of high value on the team’s work. Proper channels need to be put in place for such exchanges. Some suggestions can be:
  • Before missions: Working sessions and/or discussions with advisors as appropriate
  • During mission: Email interactions between mission team and Geneva team on targeted issues
  • After mission: Post-mission informal briefings by the mission team to ISSAT team (upon interest). The purpose of such informal debriefing is to share the first observations and key lessons identified. An exchange and questions can follow during the debriefing. This exercise should be able to help the mission team to brainstorm on their findings and put their ideas together as they are writing their report.
  • It might be useful (depending on the missions) to consider spending one more week on every report and circulating among the ISSAT team for commen
• Design of methodology and structure of the report before deployment:  For evaluation missions, it is recommendable that the methodology and the report structure be designed before the team deploys to the field. However, depending on the programme, the methodology should be an overarching one proposing themes to be tackled and overarching questions to be asked without going too much into detail (such as designing indicators, or sub-questions). Necessary margins of flexibility should be left for the team to adjust the methodology as they arrive to the field and are briefed by the programme staff. Having a methodology before hand, helps the team conserve the guiding values of an evaluation (coherence, efficiency, etc.) and not lose sight of them.
• Joint teams (international/national representatives): In terms of the initiative of creating evaluation teams that combine international advisors and national actors, it is the recommendation of this team to approach this matter in a much more structured manner through the following:
  • Once ToRs for the assignment are drafted, there should be a section dedicated to the international advisors role and the Burundian actors’ roles, detailing the responsibilities and roles of each group.
  • The ToRs should also include the value of including the national actors as a cross-learning process. The international advisors could pass on the comparative, international view and the modern technical know-how of evaluations and the Burundians could bring in the technical and cultural know-how of their security institutions and country. This would facilitate more realistic analysis and adapted methodologies to the reality of the country and programme. 
• In-depth exchanges with field programme leads and staff: Even if the ToRs are clear, the mission team should always dedicate time to discussing them with the programme team and other relevant stakeholders so expectations are harmonized amongst the group. 
• ISSAT templates:Disposing of an ISSAT templates for reports is very much needed. A report template per process would be very beneficial the teams (assessment report template, evaluation report template, etc.). This could include in each case a paragraph on what is ISSAT at the end which can serve as a useful reminder to mandators as well as help place ISSAT’s recommendations in their bigger picture (ISSAT’s mandate for reinforcing good practice).

• ISSAT Advisors had a five-hour briefing session with the donors regarding the context, the programme and the activity in question.
• The Advisors had a working session with the translators to go through the material and harmonize understandings.
• The ISSAT Advisors kept their Power Point presentations very flexible in terms of the quantity of information allowing for optimal adaptation once they gained more knowledge of the audience, their level of expertise and needs. Question and answer sessions were particularly worthwhile.
• The approach adopted for this workshop was developed through the collaboration of temporary international expertise (ISSAT Advisors), locally-based international program management representatives, local experts and civil society actors, as well as the main police audience. This worked very well in building up a dynamic momentum owned by all concerned and allowing for an optimisation of skills and knowledge of the human resources deployed.
• The mix between local and international expertise to support the objectives of the workshop proved well balanced and productive. It allowed for the identification of the structures in place that are performing, as well as highlighting ways and entry points to improve them from and international perspective.
• Informal discussions with local resource people (in this case: Police commanders) informed international experts on the systems in place and how they performed. As a result, the issue became a matter of finding how to improve those systems instead of imposing international good practice on the nationals.
• The Working Groups was a very positive approach that worked well in terms of reinforcing the issues that came up during the presentations and allowing participants to analyse them further.
• The themes for the Working Groups were chosen to target specific issues relevant to each department of the Police institution. The groups were identified along the departmental divisions as well.
• Translation was simultaneous and worked very well. The interpreters were well briefed and the result was fluid and dynamic interpretation without any gaps in communication.

• Experts should always focus on smaller and well targeted focus groups to discuss specific issues and come out with concrete recommendations (e.g.: Development of discipline regulations, managing the complaints process, central recording, etc.).
• A way forward to sustain the momentum could consist of an approach whereby, “mentors” or “buddies” are assigned to the PPF on the longer term and act as links between the international experts and the PPF. A “mentor” or “buddy” could be assigned per issue (for e.g: development of a complaints system, development of discipline regulations, standard operating procedures, developing an inspection capability, codes of ethics, etc.). These mentors could then sustain the momentum and work out on developing the details in between international expert visits who then would be working on the conception of frameworks (through short term engagements).

• The content of the training package needs to be further adapted to an audience which does not include donors but participants from countries undergoing SSR processes;
• The exercises in the training package need to have clearer guidelines for participants;
• The thematic areas that were not much represented in the participants (justice and police) were compensated for through the examples that were given by the trainers.

• The training team was designed to include trainers from the Mali Peacekeeping Training Centre which allowed for direct exchanges between two African Peacekeeping training centers. Also two Rwandan trainers, who participated in the first PSO training in Mali, were on the team which provided a positive mix of country experience and thematic knowledge. It also allowed the Rwandan trainers to confirm their recently acquired knowledge in a concrete training;
• The training team members were selected taking into consideration their backgrounds. The team included facilitators with backgrounds ranging from peacekeeping, SSR, military, SSR and development sectors;
• To facilitate work allocation and management, sub-teams were identified by team leader with one lead for each session. This meant that one person was responsible for each session and would be the focal point on preparations and progress. This approach proved to be highly beneficial in particular since team members were present in three different countries;
• The benefits of having already completed a similar pilot exercise in Mali led to the following positive changes for the Rwanda experience:

•  
  • New session added on International Actors to replace the old coordination session, even if changes are still necessary;
  • Real-life  examples in session on Non-state actors, that participants could relate to and found informative and useful;
  • Sessions better adapted to an audience with background in Peace-Support ;
  • From the first pilot PSO training in Mali, it became clear that most participants will not be designing an entire module/training, but they will be focusing on sessions & exercises. As a result, the ToT was designed in a way to allow participants to design and plan sessions & exercises factoring in sufficient time for feedback/discussion.

• Some sessions were organised by Rwanda Peace Academy on Rwanda and were very useful and insightful on their national experience;
• A session on E-learning was added with intention to use Internet. Due to lack of Internet, we used group discussions on several SSR questions as a substitute exercise. This turned out to be very useful and the team will keep it in mind for future trainings;

• The opportunity to use the e-learning as a means to confirm the knowledge acquired by participants is a good way to go forward.

• ToRs as a binding document: there was a tendency by part of the mission team to use the ToRs as a binding document refusing to scope new possibilities, imposed by the realities of the field.
• Team task allocation: the team members’ tasks were allocated according to their competencies, experience and character traits. A strategy was used to involve the “difficult” team members positively by giving them important responsibilities. This impacted negatively the quality of some of the focus areas. The team mitigated this challenge by assigning support capacity from the team on each focus area.

• Existing relations between mandator and country: Methodology for the mission was based on ISSAT good practice in assessments, ISSAT’s OGNs and Norway’s existing exchanges with the Government of South Sudan (GoSS). The team’s work (interviews, access to key people, access to information) was very much facilitated by Norway’s positive engagement in South Sudan, as well as, ISSAT’s professional reputation.
• ISSAT’s engagement in South Sudan:  ISSAT’s engagement with the Swiss and UNMISS fed into the mission supporting Norway, enriching it with additional comparative experience.
• Participatory methodologies: the methodology included the design of a master list of guiding questions for the assessment. This exercise was done including all team members. It worked very well, resulting in strong ownership of the methodology.
• Sharing and validating data among team on daily basis: the team factored in necessary time for team meetings everyday inorder to share and validate information collected during the day. When it was not possible on daily basis, weekends were reserved for team work. This also served as time needed to plan for future steps of the mission.
• Disclosing list of recommendations to national partners: the team did not give full disclosure of the recommendations to the South Sudanese partners in order to manage expectations. A sense of where things were going was shared but the recommendations in detail were sent to the Norwegian authorities for validation and final decision-making. Based on the latter, communication will take place with the Sudanese government.

• ISSAT to be involved in drafting the ToRs: ISSAT should support the ToR drafting process making sure that the political objectives and project concept/rationale are well understood. ISSAT should draft them directly but support the process through advice on international best practice and providing comparative experience.
• ToRs as guidance to be validated by mission team: After drafting the ToRs, they should remain in draft form until discussed and validated by the project team on the ground. Mission team should be given the opportunity to review the ToRs after a few key meetings. A suggestion could be to add to the ToR document that it is a draft subject to review once team is deployed to the field.
• “Who is Who” directory as an ISSAT Networking Tool: ISSAT Advisers deployed to the field could benefit  from an accessible compilation of resource people that ISSAT is in contact with in different countries.This knowledge product could be a directory of “Who is Who” and “Who knows Who”. It could be divided into two levels –one strategic and one operational. The names of key people per country and institution could be listed next to their contact information. In addition, it would be useful to have each ISSAT Adviser linked to the key people they know.

Impact of absence of TNA :  

• Design of the course was driven by what was available in the L1 as opposed to what was needed
• Training team did not have a complete insight into the Bank's approach, existing knowledge and strategic needs in terms of mainstreaming SSR within their programmes
• The course was actually a workshop with the intended output of recommendations for action. Had this been understood in advance, more time would be been dedicated to group facilitation rather than lectures
• The facilitation/teaching balance in terms of delivery of knowledge was not properly managed. Had the team known more about the experiences of the audience they would have made more of an effort for allowing the audience to present their own experiences
• The choice of groups would have been easier, more relevant and meaningful had they been established along the expertise of participants

Balance between material density and participants’ expertise:

• ISSAT L1 is a heavy course especially for field staff who have their own experiences and would need more space for discussion

 Number of case studies used:

• the use of case studies should be handled with care. In this training too many case studies might have been introduced. ISSAT might want to consider for the future presenting one or two cases and giving the participants the time to seriously absorb them

Building on participants’ knowledge and experiences:

• Session 8 had participants discuss how the AfDB could incorporate JSSR into its everyday work. As the principles were generated by participants in an early session, this created highly motivated engagement in thinking through how to take them forward.

• Need for structured TNA: There is a need to establish a structured and systematic TNA which can be used for each event to help identify background of audience, objectives of the mandatory and use these variables to tailor trainings to needs
• Integration of non-donors perspectives: ISSAT training material needs to be revisited to take into account perspectives and needs of non-donors and southern countries needs and perspectives
• Building on partnerships: ISSAT to build on its relationship with ASSET and ASSN in supporting the delivery of trainings and workshops
• Accommodating discussion time: More time is required for audience discussion and participation. Consider either shortening the materials presented in ISSAT L1 or adding in an extra half day to schedule
• Training mission requests processing: In ISSAT's recently established mandator request processing, there are two ways to accept training requests. It is either a standard training and it will go directly to the training team, or it is a different  training request and it goes through the SMT. Having a training request go through the SMT first should be considered as the milestone to indicate that in-depth tailoring of the material needs to be done
• Establish an ISSAT repository for exercises: A recommendation for future implemention could be a repository of exercises that a trainer could chose from and adapt to different audiences and levels of SSR knowledge

• Process:

1. Frame questions
2. Discuss how they can be answered (sources, approaches, etc.)
3. Participants spend a week doing research and preparing answers
4. Re-group to share answers and the basis for the answers
5. Review what they did to find the answers
6. Share and discuss in the group
7. Refine the answers 

• Focus on Research Process: Participants were not only asked to provide substantive answers to questions, but also to provide the method which they used to answer the questions, thus building up sustainable research capacity.
• Learn By Doing: Capacity was built up through workshop participants conducting their own research as opposed to being lectured.
• Workshop Language: The whole project was conducted in Tetun which helped establish local ownership, facilitated the rapport between the external advisor and workshop participants, and kept a continuous flow of discussion and work through the workshops.

Transferring Knowledge: Particularly in ISSAT’s case, capacity building of team members and local counterparts is always an interwoven objective to the overall programme, if not the objective. This could have been pushed for more and clearly communicated so that there were not some lost opportunities for all parties to learn from each other.

Absorption Capacity: Along with donor’s assessing their own ability to deliver a programme, be sure to assess a partner’s ability to absorb a programme. This is a sensitive situation as resources to distribute may already be known or discussed which creates incentives to be agreeable to programmes that potentially cannot be absorbed. Remember, expectations have been raised once an assessment is under way and this needs to be managed.

Desk Review:  A comprehensive review of material enabled a good understanding of the situation. Many donors have been or are involved in military justice in DRC, and there is much valuable knowledge already on the books to be considered.

First Hand Knowledge: The three members of the team had extensive experience in the DRC which enabled the work to be focused and productive in a short period of time.

Review Objectives: Every morning the team would to think through the interviews of the day and discuss the objectives and how they fit into the program so that focus was not lost. This also allowed team members to know who was best placed to ask certain questions.

Programme History: Many countries undergoing an SSR process will have met with a number donors already for various assessments. Many of these assessments may have not gone anywhere and did not build capacity while taking place. This can create fatigue on the side of the national stakeholders. Without taking into consideration what has come before, and not realising the current expectations a donor could wind up harming their own reputation, the reputation of the assessment team members, and stall chances of success for further donors. All of these considerations are important to consider and need to be managed.

• Sufficient coaching for ownership of methodology: The mission’s methodology included coaching and working sessions with national stakeholders in order to allow for local ownership through a pedagogical approach. Due to the short time frame before the seminar, the coaching sessions delivered were not enough compared to the needs.
• Concept confusion: A clear confusion was observed as to the significance of the concept of “defence” as opposed to that of “security”. At times, both were used interchangeably without any emphasis on the human factor. On other occasions, “security” was used to talk about the military aspect and “defence” to talk about the holistic aspect. Part of this confusion stems from the adoption of SSR in the francophone context where there is not a tradition of ‘National Security Strategies’, but rather ‘National Defence Strategies’ (politique de defence ) which conceive different relationships of security, state, institutions, and population than Anglophone concepts (or Russian, or Arabic, or African, etc.)

• Focus on key actors: In an attempt to mitigate time limitations, particular focus was paid to the UN SSR Focal Point on the field since he had enough knowledge and capacities to absorb the new tools and concepts. The mission’s strategy was to focus on building his capacity so that a there would be a longer to presence who could transfer acquired skills to the local partners.
• The National Technical committee’s need for longer term coaching and capacity development however remained on table and was put down in the mission’s recommendations for the immediate term.
• Political Will: In trying convey a new concept and embed it into a process (in this case ‘human security’), the highest optimal political authority is needed to assist in achieving this task. In this context the National Seminar was opened and closed by a speech from the President of Guinea which emphasised the human component of security.

• Continuous discussion on priorities: Often, support missions tend to have an internal conflict between expectations and time constraints. It was key during this mission to be able to discuss openly and continuously with all concerned (especially the mandators and ISSAT management) of the risks of limited time and the real needs on the field. Jointly, we adjusted priorities as we went, keeping the focus on the main objective (successful national seminar) while finding appropriate slots for other activities on other agendas, always keeping ownership at the core of every process.
• Agree on common terms acceptable nationally and internationally: Capacity development right after assessment to at least clear out key terms and arrive to common appellations.
• Contextualising SSR: There needs to be investment in understanding the traditions of different cultural state and security concepts and structures (e.g. French, German, Spanish, Portuguese, Russian, Chinese, Arabic, African, etc.). Often SSR is viewed from an Anglophone perspective and advisors can placed into a system they are unfamiliar with and presume it is wrong and work against a structure rather than trying to understand how SSR principles can be integrated. Not every advisor can know every cultural model; models and concepts will be different in each case, but awareness raising in trainings and developing skills for inquiry can help. International standards and tools such as the OECD DAC Handbook also need to be updated to reflect this reality.

Politics of SSR: Conveying the politics of SSR proved to be a concept that was difficult to transfer as not all uniformed participants viewed themselves as having a political role, rather they are neutral. While encouraging, there are still political aspects for all actors involved in SSR. This includes trainers wishing to convey the concept.

ISSAT Training: Portions of ISSAT’s Level 1 training were used to convey the key principles of SSR as well as fundamental components of supporting an SSR programme. The materials were well suited to be adapted to a specific audience.  

New Concepts: There will always be a challenge in instilling a new concept or methodology in people unfamiliar with the SSR approach. Make sure to be aware of their willingness and ability to take on a new concept. These two indicators can help in planning an initial engagement as well as follow on actions.

Politics of SSR: Be aware that discussing the politics of SSR is in itself political and plan accordingly.

Citizen Protection: Needs to be considered further durign workshops

Workshop: Format was engaging and gave good insight

Security

• Team Input: More time at the beginning of the mission to allow team members to get a clearer sense of strategic and tactical objectives, and particularly the drafting and finalization of the mission report, would be useful. The list of interviews and the interview schedules and sequences were largely determined prior to mission commencement without significant input by team members. Whereas many of the interviews clearly involved substantive discussions of issues and concerns to inform the assessment, others appeared to be more introductory in nature, or scheduled as a courtesy.

• Iterative Team Input: The challenges noted earlier were addressed largely by working closely with other team members to define and adjust the overall approach to the evaluation and assessment work as the mission progressed. This also allowed for more substantive and focused interviews in the latter stages of the mission. A clearer understanding of the nature of the team's involvement once in country also helped to focus efforts and lines of enquiry in the DRC.

• Early Team Input on Questions: Allow more time at the beginning of the mission for team members to prepare key lines of enquiry and verification. Team members would benefit from having the ability to develop and set out the list of questions or lines of enquiry that might be pursued during upcoming missions. This would allow for a more effective use of time in country. Finally, team members should be able to determine the list and sequencing of key interviews, particularly if an assessment mission is intended to be iterative as each mission represents more focused scrutiny of issues noted previously.

• Consistency of Methodology: A team needs to be clear on an approach or methodology being used. Differences of view points should be reconciled well in advance of being deployed. Without this there is the possibility of team members acting independently which will complicate the final analysis of information.
• Preparatory Reading: Initial desk research is important and the right documents need be provided by a mandator well in advance with those documents being read by the whole team.

• Slow recruitment and deployment of UNPOL
• Selection of officers: Very often UNPOL officers would not meet minimum selection criteria 
• Officers' competencies: The lack of UNPOL officers with specific expertise
• Language skills: Difficulty to attract officers with knowledge skills in French for francophone missions

• Immediate deployment of the Police Commissioner and a group of five senior officers to start up the mission
• Capacity building: To overcome deficiencies in the capacity to deliver the mandate, we developed a UNPOL training for incoming UNPOL officers with lack of knowledge in use of computer, driving, human rights, training,mentoring, advising, report writing, etc.

• Selection criteria: Need for a better selection of officers by Police Contributing Countries (PCCs)

On MICAH/UN's side:

• Recruitment: It took too long to recruit and fill the technical advisors' position
• Mission duration: The mission was too short (11 months)

On the Police Nationale Haitienne's (PNH) side:

• Weakness in the chain of command
• Weakness in the leadership
•  Lack of human and material resources
• Weakness in the capacity in general policing and criminal investigations

On PNH's side:

• A five year strategic plan jointly developed
• Imporvement in the organisational structure
• Internal control mechanisms/oversight reinforced but not accepted by all
• Improvement in the operational capacity and in the dily delivery of activities. The improvement is direcly linked to the leadership in place. Results varied from location to location
• Administrative and operational directives organised and accessible to all

On UN/MICAH's side:

• The right advisors were selected
• All advisors were fluent in French and many could communicate in French Creole and English
• Good coordination among international actors (USA, France, Canada, UN/MICAH and UNDP)
• Elections security plan developed and implemented  
• MICAH rule of law pillars were integrated and worked in harmony

• Recruitment: UN recruitment process should be faster
• Longer term planning: There should be a medium to long term planning established by the assessment mission. It would prevent missions like MICAH to stop after 11 months without anybody to fill the gap. Hopefully, in the future SSR principles, if respected, would prevent repetition of the MICAH experience elswhere

• The commission issued a call for public inputs on intelligence reform but very few submissions were received. Most of the submissions that were made were not based on a good understanding of the intelligence dispensation. The commission had planned to conduct public hearings but there was insufficient public interest to justify to proceed with the plan.
• The parliamentary committee responsible for intelligence oversight was antagonistic to the establishment of the commission. This was a result of political tensions within the ruling party and the committee's resistance to intelligence reform.

• The heads of the intelligence services, the inspector-general for intelligence and representatives of relevant government departments made detailed written submissions to the commission and also made oral presentations at which the commission was able to solicit further information and documentation.
• All the operational policies of the intelligence services were made available to the commission. The commission was therefore able to make an informed judgement and set of recommendations.

None

• Coordination of Actors: When contracting out the implementation of a programme to multiple actors, be sure that they are harmonised in actions and capacity, such as level of deployment.
• Communications among Actors: When bringing in an external evaluator, make sure that there is a clear understanding of their purpose. In this case there were misunderstandings that the evaluation was on the feasibility of the project's goals and milestones, not simply the performance of implementing actors.
• Documentation Consensus: Key documents, such as the project logframe, need to be agreed upon and understood by all stakeholders to avoid unnecessary deficiencies in carrying out a project.

• Second Start: If required, a second start to an evaluation may be required. The long-term benefits to righting admitted wrongs can far outweigh acknowledging, but not correcting mistakes. In this case the ISSAT team needs to return 3 months after the deployment of all 3 contracting agencies, to reassess its first findings of November 2010.Afterwards, one yearly team visit (twice per year for the Police Support Program) will suffice.

• Coordination amongst actors: ISSAT should recommend to DFD that it bolsters its role in organizing the internal coordination and planning of the project components with the 3 contracting agencies.

• In the beginning of the mission, meetings were not as focused as they could have been. There was a tendency in the first couple of days to veer off topic and cover tangential issues; while this changed toward the end of the mission (as is often the case) , we lost some time in the early days.
• As a private sector company, the main focus of the organization is to win program bids. To some extent, strategy to win the next program bid overshadowed the substance of how best that program should be carried out.

• Prior to departure, the team organized numerous discussions on the purpose and scope of the mission. This was very helpful in understanding in detail what the organization was hoping to achieve. I think it contributed to the success of the mission.
• I was asked to provide contacts in parliament with whom we could meet; this was part of the value I added to the mission. Given that I had personal relationships with those individuals (among others who we met with), the meetings were focused and on target. I briefed the team before each meeting, focusing on why that particular meeting was being held. A program officer organized a number of other meetings without understanding the need for context; as a result, many of them were less useful.
• The mission included a comprehensive range of meetings: most were with Afghans, while the rest were with internationals from the diplomatic community, NGOs, media, etc. For parliament, we made sure to meet with government and opposition, representative from different ethnic groups, women and youth.

• It would have been useful to have developed a semi-structured set of interview questions prior to the initial meetings. This would have allowed us to be on the same page from day one.

• There was a noticeable absence of input from national stakeholders in the proposal development process. This was due largely to the expat staff in both countries who respectively managed the process very closely. In both cases, expat staff had a clear sense of what they wanted to do, and failed to use the program development process to either solicit feedback or secure national ownership; in place of input from national stakeholders, extensive polling was used. While this provided a valuable source of information, such data cannot replace insights and priorities of stakeholders and beneficiaries.
• Designing a regional program to cover a dozen countries poses many challenges. Before committing to such a program, careful consideration should be given to whether such a program is realistic given resources and expected outcomes.

• There was a significant focus on developing a "theory of change" for each of the two proposals. We included a theory of change for each, explicitly linking how outputs lead to outcomes. Too often results matrices are an afterthought, revealing leaps in logics between activities and expected impact. Theories of change are a critical component of the program development process that is often overlooked or ignored. This organization made it a priority, which I believe  ultimately helped make the proposals competitive.
• I was the lead on developing one proposal, and contributed to a second. This was an useful approach to dividing responsibilities and worked well.

• Theories of change are a critical part of any program development and should be incorporated as a core step in the process. It serves as a filter to ensure that activities are not simply conducted for their own sake, but that there is a logical progression toward impact.
• Developing programs that are realistic in scope and expected impact is critical. A multi-country, regional program in which a great deal of resources are spent on travel for participants may not be the most cost effective approach to solving a regional issue. Using a theory of change approach may help mitigate against overly ambitious program proposals.

• Decentralized intervention scopes: In addition to training in the capital, our staff in the provinces provided ongoing support. It appeared that in an effort to help the councilors succeed, our staff may have been too "hands-on" and in certain situations had a heavy hand in follow-up activities.
• Political dimension of participation: We asked each of the participating councils to select one "rapporteur" on behalf of the council. The most qualified were often chosen, however, we did encounter a number of individuals who were chosen for political reasons and not their interest or capacity to participate in the program.

• Use of acquired skills: Unlike most training activities, we designed a program in which the training sessions featured as only a small component. Training was held to provide participants with the skills necessary to carry out the remaining components of the program. We designed a three-phase program in which local councilors were trained on specific tasks for each phase; they, in turn, used those skills to carry out activities during that phase. As such, the real value of the program was not the training, but the use of acquired skills.
• On-the- job implementation: In phase one, councilors were trained on how to solicit input from their constituents on the five most pressing problems, and potential solutions, in their communities. Training comprised how to organize public meetings, taking notes, drafting agendas, among other topics. They then returned to their provinces and with our support, each held a dozen such events.
• Follow-up: We brought them back for phase two, and worked with them on how to prioritize the information they collected and turn it into a development plan. With our support, each of the councils produced a development plan.
• Communication training: Phase three comprised communication training; with development plans in hand, we worked with councilors on methods of communication with stakeholders and citizens. We then sponsored half a dozen communication activities for each council in their respective provinces.
• Theory to  Practice: In this manner, councilors received training for a specific purpose and were supported to carry out follow up activities. Measuring the impact of training is very difficult when trainings are stand alone events. The real impact was seen in how participants put the training to use.

• Follow-up between training sessions was critical to the success of the program; we should have made it clearer to our staff that we would have preferred the councilors to try and fail than have our staff save them when activities faced challenges.
• Political aspect of exercise: To the extent possible, councils should have been guided more clearly on the profile of individuals selected to serve as rapporteurs. Given the political nature of this work, however, this may have only had a marginal impact. Another possibility would have been to have each council put forward two candidates, which we could have evaluated and chosen the most qualified.

None

• Building consensus: During the drafting process the challenge of building consensus was addressed by 1) consulting all stakeholders thoroughly and repeatedly in an iterative drafting process; 2) keeping scrupulous records of all inputs and the reasons for accepting or rejecting the inputs; 3) using the Constitution as the fundamental legal and policy framework for the White Paper; 4) undertaking comparative research on civil-military relations, military doctrine and regional security arrangements in order to identify a range of policy options; 5) a willingness by all major stakeholders to be flexible and adapt their positions in the interests of reaching a consensus; 6) broad recognition of the different responsibilities and authority of the Chief of Defence Force, the Secretary for Defence, parliament and the Executive; and 7) providing sufficient resources and time (over two years) for the consultation and drafting to be inclusive and collectively satisfactory.

None

• Prior to the mission, the team was not fully aware on what to expect during the implementation. No full clarity on expectations was established as to the exact objective of the mission that needs to be achieved along with a detailed understanding of the specific issues that the team would confront. Tighter planning processes would have helped shape realistic expectations and prepare adequate actions, as opposed to adhoc reactions.

• Logistics were cleared out ahead of the mission itself.The World Bank's initiative to send representatives on the field, prior to the mission was a successful approach that contributed very much to the efficiency of the mission.

• Specific and precise mission planning: Mission planning needs to be much more specific, giving the team a realistic understanding of the specific tasks at hand.

• Handling of data: There was a tendency of excessive inductive interpretation of quantitative data without a theory behind the interpretation and some initial lack of systematization. This was overcome by strengthening the social science expertise (methodology) of the initial team by the Embassy and external reviewers. Also, national statistics used had varying quality, but the project in itself could help improve some of these national limitations.

• Choice of experts: The investigation team was carefully selected as to represent both technical and political capacity with these positions all being filled by regional expers - for example, the former Chief of Police of Nicaragua participat
• Rigourous scientific methodology:  By using a strong methodological  approach, roughly 10% of annual average homicide rates were analysed following the cases from teh crime scene to the concluding sentence, if any.

• Precision in capturing data: If some hypothesis are made initially, the data can colledted with a higher degree of precision while still beign open for non-expected finding.
• Political strategy: The political strategy for launching the results and the followign process of formenting change could be slightly better planned and managed. Thus, a communication strategy could be a way to further improve.

• Timeframe constraint: The timeframe for output delivery, was too short. It was noted to the external team by the technical working committee that a draft academy curriculum typically takes anywhere from one to two years to develop.
• Subject matter experts:  there was no consultation in advance or involvement during the workshop making the development of the curriculum difficult.

• Structured Approach: This compensated for the tight time frame, allowing the construction of the curriculum and an appropriate focus on the content and the tools required to implement the curriculum.
• Taxonomy Development: The approach began with a review of critical documents for southern Sudan and the SPLA to establish defence training and education priorities and a taxonomy structure. The taxonomy was based on Bloom’s Taxonomy Model and developed into a body of knowledge which could be drawn upon to develop course outlines.
• Support Mechanisms: Two key areas of discussion were a knowledge management strategy to support implementation and keep the curriculum content updated; and how donor assistance could be applied to the curriculum.

• Expand timeframe: The development of the curriculum should be done in a multi-staged process, with different leads and working sessions to address the previous topics described. Topics such as the development of a taxonomy and knowledge management can be complex and require time for national stakeholders to build up their capacity in them.
• Breakup workload: The workshop would have profited more from being an oppotunity to amalgamate and consolidate the work of previous stages.
• Subject matter experts: More involvement of subject matter experts to fill details of the taxonomy and course content would have been of great value.