Review of the Implementation of the Action Plan for the Development Strategy of the Ministry of Interior of Serbia
- Time constraints:while prior knowledge of the MoI from ISSAT team and assistance from Sida/SPAP greatly contributed to carry out the field mission, the team had only two weeks in Belgrade, which restricted the number of meetings and activities possible.
- Culture of secrecy: like if by default, all kinds of documents and information are classified in the MoI and it is difficult to navigate a mandate of this nature when the prevailing institutional culture and legal/policy framework is one of secrecy and opacity.
- Empowering while doing no harm– Capacities are uneven among mid- and senior management at the MoI and between units. The imbalance of capacity across units is compounded by uneven attention given to different units by senior managers and donors. In this mandate, the risk was that those units needing the most support would be undermined further by being highlighted as deficient in the report due to the inherent blame culture (the Analytics Directorate being an example). Asymmetrical support to some units while others take on expanding responsibilities posed another challenge: how to reinforce the most performing units (e.g. BSP) whilst not creating too much of a gap in capacity with the rest of the MoI, making them the object of jealously by their peers.
- MoI in transition– the MoI was looking ‘very inwards’ (sic) until Brussels gave its feedback to the screening under Chapter 24, and then preparing Serbia’s follow-up on the results; afterwards, the new Cabinet has been focusing attention in drafting the new Police Law. Discussion on gaps identified by ISSAT will thus depend on the Cabinet being more free of those processes to think on strategic management. The report ‘is a seed that can grow, but in other conditions’, as pointed out in the AAR. (See below also).
- Objectivity: ISSAT mandate was carried out in a way that safeguarded the team and the report against considerations of political bias and confirmed the reputation of the organisation within the MoI as an independent body.
- Contextual knowledge: previous ISSAT mandate with the MoI and in Serbia, and in the region provided a robust knowledge of the realities and issues at stake in this review.
- Balance and relevance: the report was perceived as addressing all the relevant issues against a benchmark that was rational, balanced and that made justice to the current capacities and overall context of the MoI. This meant in turn that recommendations are credible in that they contain attainable outcomes, when the opposite is quite often true in the experience of the MoI: evaluations against an ideal scenario with no relation to realities on the ground and existing baselines, thus immediately discarded by those to whom they are addressed.
- Language/Tone: the mandate touched sensitive topics and the report raised sensitive questions, including transparency, stakeholder engagement etc, in a tone that was “diplomatic but straight to the point” to convey the important messages. The subtle but sharp relevant language also contributed to empower without undermining (see below).
Lessons identified for improving the chances of success:
Link-up the review with building the capacity of the institution(s) being assessed:ISSAT team was requested to share its review methodology with the BSP, in order to share additional tools and practices that could provide the Bureau with ideas to inform their role in the context of the Accession process. While the use of such methodology by the Bureau needs further confirmation, the fact that it is perceived as useful to other programmes within the MoI is a relevant ‘collateral’ gain. (See also below).
Outputs and Outcomes
The mid-term evaluation report delivered by ISSAT was considered very useful, pertinent and objective by both the mandator and the MoI. Both stakeholders noted nonetheless that the report has yet to produce it’s full impact due to circumstances external to the mission: first, the catastrophic floods that put the country in a status of national emergency for several months; second, the general elections held in March that redrew Serbian political landscape and brought a new party to power. At the MoI, both events took attention and resources away from discussions on strategic management and disrupted ordinary activities. This was compounded by an overriding priority given by the new Government in its first months in office to the revision of the National Programme for the Adoption of theacquis communautaire(NPAA) and the follow-up to the EU screening on Chapter 24 (Serbia recently submitted a new action plan to meet the goals set by Brussels on Justice, Freedom and Security).
Only recently life returned to normal in the Ministry. The action plan for Chapter 24 and the new plan for the Development Strategy provided an opportunity for BSP to use ISSAT’s report within the MoI, at this stage to raise awareness of issues raised in the document and the importance of building capacities in areas like budgeting and strategic analysis. ISSAT findings and recommendations are likely to have a bigger impact long after the mandate finished, into 2015. Indeed, the review seems to be recognised at senior level in the MoI as a reference source to inform planning at the Ministry, in two dimensions that seem interconnected. First, the mandate report is used as a stand-alone document considered for its sensitive analysis and recommendations - even if the latter were not yet or not entirely acted upon. Second, the mandate (report, methodology and ISSAT’s overall approach) left a set of tools and good practice, meaning that ISSAT contribution is clearly understood to have high value beyond the review conducted. New management in the Directorate for Analytics has also enhanced opportunities for the recommendations to actually be taken in consideration in terms of addressing some of the imbalances identified in the report.
The item of capacities for budgetary inputs into strategic planning was not included on the initial Terms of Reference for this review but it was considered relevant during the field mission and included in the report. The resulting remarks and evaluation were credited as being very useful to the MoI, well beyond the new Action Plan recently approved. Adding budgeting to the mandate contributed to raise awareness of the importance of this focus-area at the MoI, as previously mentioned.
Strategic Management - Baseline Study Moldovan MIA
Outputs and outcomes
ISSAT produced a comprehensive report that evolved from the original request of doing a baseline report to one that looked at the various needs and made substantial recommendations. Note that the term ‘baseline’ suggests that programme areas have been already identified and the study is to see where the process evolved from. This mandate actually needed the programme areas to be identified.
In order to capture a broader array of views, an electronic survey was developed by the team and distributed to key personnel in the Central Apparatus/MIA and the heads and deputy heads of a number of departments within the General Inspectorate of Police, Border Police, and the “Stefan cel Mare” Academy (SCM) following the first main mission.
The findings of the review were considered robust, sharp and timely both by the mandator and the MIA. They point to the exceptional progress made in 18 months in strategic management at the MIA under the current minister, while identifying gaps in different areas. The report was used by the mandator (Sweden) to develop a programme by DCAF with the MIA, with a suggestion made in the AAR that findings from the baseline assessment can be carried forward by DCAF.
The recommendations are meant to quickly shore up gains obtained so far with the need to sustain reform in the longer term, also against the prospect of a change of tide in Moldovan politics. The question raised in the AAR is whether those recommendations can be implemented in the existing conditions, due to political constraints and limited human and financial resources.
Early on the mission, the team had discussions with Swedish embassy staff on what the role of ISSAT was regarding capacity building and the question remains if ISSAT’s ‘reinforce not replace’ role was effectively delivered. There were plans to have an afternoon capacity building session with the embassy staff, which never materialised due to time constraints and uneven engagement. From the perspective of the mandator, though, the results were tangible in terms of apprehending methods and tools.
Strategic Management - Baseline Study Moldovan MIA
Scoping and survey - The development of a template specific to the mission was vital in order to maintain a coherent framework for the analysis. The template questions were also shared widely so we were better able to manage expectations. The work carried out earlier on in the mandate allowed ISSAT to get to a point towards the end of the mandate where the team had a fairly comprehensive overview of key findings that could be tested out with key interlocutors (e.g. the EU High Level Advisor in the ministry). It also gave the team the confidence to be able to support the idea that detailed design of the subsequent programme could be initiated based on the draft report in order to save time.
Access to information/transparency– The degree of openness at the MIA is remarkable. If anything, there is too much information : over 100 strategies/concept papers across the MIA, all available to be shared. The progress made in transparency is quite extraordinary if we take in consideration that in 2012 the MIA was still perceived as “a closed system that lacked transparency, with Soviet-totalitarian governing practices” (MIA Functional Analysis).
Team composition- The team composition was important. ISSAT experts on team could count on the extensive knowledge of David Clarke, drawing on his understanding of the political undercurrents. This was a great example of ISSAT being able to bring together different resources.
Local support– Excellent support was provided by the advisor to the Minister.
Local knowledge– The mandate gained from the insight gathered by ISSAT during the NORLAM Moldova mission that had started slightly earlier.
Relevance of the report– As mentioned above, dozens of concept papers exist across the MIA, adding to the three functional assessments previously carried out by various donors (including EU). The risk for ISSAT baseline to be redundant was avoided by rooting the findings in very comprehensive information collected from a range of sources and using different tools (including an electronic survey, interviews, document analysis, etc.). From the MIA, the report is perceived as adding depth and scope to the existing analysis.
Credibility– The findings are not disputed in the MIA and the report was ‘generally accepted’, including by those who don’t accept that a review can be performed by a foreign organisation and experts from abroad. Such actors tend, if anything, to question the legitimacy of the approach – not the credibility and validity of the actual findings.
Strategic Management - Baseline Study Moldovan MIA
- In recent years, the MIA increased dramatically the level of participation in its activities, which in turn was reflected in substantial gains in transparency, which makes for a unique case in the region. Crucial to civil society participation is the Civil Society Council of the Minister, an informal body made up of key persons from civil society that convenes every three months. The Council provides an avenue for substantive inputs from civil society to the MIA, for instance in undertaking monitoring.
- The Stefan cel Mare training Academy, one of the administrative authorities subordinated to the MIA, plays an important political role, as many politicians and members of Parliament are former alumni. The Academy, though, is seen as averse to reform and its influence might play against the momentum created under the current leadership.
Assessment to inform potential options for future international support on Police reform in DRC
From Internal AAR:
La période des vacances n’a pas du tout été propice à la bonne planification du mandat. Ceci depuis la demande initial des mandataires à ISSAT jusqu’au déploiement de l’équipe sur le terrain. Les personnes de contact du côté des mandataires (DFID et UE) étant en vacances durant la phase de planification. L’équipe a donc observé un manque de communication, de soutien logistique et administratif. Il y a eu également un manque de référence aux documents clés à analyser avant la mission de terrain. De plus, les membres de l’équipe d’ISSAT, issus du roster, n’ont pas reçu toute l’information nécessaire sur les tenants et aboutissants de la négociation du mandat afin de bien se préparer pour leur mission de terrain. ISSAT ayant soutenu de manière continue DFID RDC depuis 2012, il aurait été intéressant de briefer l’équipe d’experts sur les résultats des précédents mandats.
Les TdR ont probablement été copié-collé d’un modèle de l’UE, assez peu détaillés et pas finalisés ni approuvés officiellement avant le déploiement de l’équipe sur le terrain. Les objectifs officiels, détaillés dans les TdR, étaient fort différents des objectifs officieux qui n’ont été communiqués à l’équipe qu’une fois sur le terrain.
Les partenaires locaux n’ont que très peu été impliqué lors de la phase de planification. De manière générale, les partenaires locaux ne reçoivent pas les documents de DFID qui ne sont pas traduits en français. Ceci est le cas par exemple pour les rapports d’évaluation du SSAPR rédigés en 2012 et 2013 conjointement par ISSAT et la SU. Ceci dit en passant, ces rapports étant très (voir trop) long selon certains, il est difficile et fort coûteux de les traduire. Il serait intéressant d’essayer de réduire la longueur des rapports auxquels ISSAT contribue.
- Besoin de TdR clairs, détaillés, identifiant les responsabilités de chacun, ainsi qu’une personne de contact pour la logistique du côté du mandataire. Les TdR devraient être approuvés avant le déploiement de l’équipe sur le terrain.
- Mener une mission préalable(scoping mission) de quelques jours est une des leçons apprises par ISSAT afin de mener à bien la phase de planification, voir de finaliser les TdR et obtenir une approbation officielle. Ceci n’a pas été possible pour ce mandat à cause de raisons diverses et variés (temps, disponibilités des experts, volonté des mandataires).
- Traduire les documents clés pour les partenaires locaux. Intégrer le coût de la traduction dans le budget du mandat afin de mener à bien ce travail (car ceci n’est jamais fait par DFID). ISSAT et le mandataire doivent convenir ensemble de qui s’occupera de la traduction et partager les contacts de bons traducteurs (qui ne sont pas facile à trouver).
- Communiquer avec les membres de l’équipe d’ISSAT, issus du roster, sur les tenants et aboutissants des négociations du mandat avant leur déploiement sur le terrain. Dans le cas où le mandat est la suite d’un mandat précédent mené par ISSAT, s’assurer de bien communiquer les résultats de ce dernier à la nouvelle équipe d’expert. Si cela est opportun, faire venir le Team Leader chez ISSAT à Genève pour 1-2 jours afin de discuter avec les membres d’ISSAT ayant suivi les précédents mandats.
Review of the Rule of Law Advisory Mission on Moldova (NORLAM)
Outputs and Outcomes
All the outputs foreseen in the ToR were achieved and the mandator was overall very happy with the work carried out by the review team. Some outcomes were also achieved (see LI form in annex here below) but some others were only partially achieved such as the clarification on the future scope of NORLAM after 2016, the status of a police component and the length of extension of NORLAM. It was recognised however that the ToR were very ambitious and Norway wanted to achieve too much in one review over a short period of time. Prioritisation of the expected outcomes could have been better and communication between ISSAT and Norway when drafting the ToR could have been more intense. The Oslo mission carried out during the planning phase of this mandate was very useful, but it took place a bit too late in the process. The ToR were already approved. However, amendments were still made to the ToR after the Oslo mission which were considered useful.
To help prioritise the ToR, the review team could have also communicated better and earlier on the methodology to be used for the review.
Norway and NORLAM are using the conclusions and the recommendations of the report. It is considered to be very important for them. The report played a major role in the discussion between the Norwegian Ministry of Justice and Ministry of Foreign Affairs to decide whether or not to extend NORLAM. Moreover, the NORLAM team organised an internal seminar in mid-September to review the report. They analysed the findings and conclusions and came up with a list of action points. This exercise was considered to be very useful for discussion and ownership of the report. A second internal seminar should take place soon to discuss more in depth the recommendations. In November, the Norwegian Ministry of Justice will visit Moldova to discuss the post-2016 NORLAM, on the basis, among others, of the conclusions and recommendations of the report.
Some recommendations are however considered to be too vague (e.g. what does it mean concretely to develop NORLAM Theory of Change, cf. Recommendation 4). Some others are difficult to follow such as the one on the legal status of NORLAM, because it needs political agreement and decision that are not easy to obtain. Nevertheless, having such recommendations in a written report, done by an external and international actor such as ISSAT, is considered to be very helpful and can be used as a political leverage for discussion with Moldovan authorities.
Lessons identified on output and outcomes:
- Ensure good communication with the mandator during the drafting of the ToR in order to help prioritise the expected outcomes and the scope of the work.
- Try to undertake a scoping mission earlier in the process so that discussions on the ToR can take place with the mandator. Overall the scoping mission is a very useful step in the planning phase of the mandate.
- Include in the ToR a section on the evaluation methodology to be used during the mandate.
- Develop as much as possible concrete recommendations. General recommendations could be unpacked with concrete actions to be considered by the mandator.
- Continue building ISSAT reputation in terms of being a recognised international actor with strong SSR expertise and evaluation methodology.
Annual Review of DFID Sierra Leone Access to Security and Justice Programme
- Great team work, good division of labour between members of the team due to:
- Very good expertise in terms of thematic (justice and security) and context (SL and DFID working methods);
- Team members knowing each other well;and
- Good ToR, discussed beforehand and circulated to all team members.
- Great support from DFID SL and the implementing partner.
Lessons identified for improving the chances of success:
- Have a mixed team with:
- strong and relevant thematic expertise;
- knowledge of the country and programme context;
- knowledge of mandator’s methods and procedures;and
- who knows each other.
Circulate and provide comments on the ToR before hand.
- Ensure strong support from mandator’s field office/Embassy and programme implementers.