In this SSR Methodology section, Community of Practice members collaborate on the continuous development of practical guidance for carrying out and supporting Security and Justice Sector Reform programmes. The guidance is organized according to the phases associated with the reform process and includes explanations of key concepts, recommended activities, and a constantly growing collection of case studies, tools, examples, lessons and other resources drawn from the collected experience of the community.
As a starting point in the development of the community SSR Methodology, ISSAT has focused on creating practical guidance related to the phases of a typical programme cycle, as illustrated in the diagram on this page. Please explore the guidance, tools and other resources contained within this methodology, try them out in your work, and share your feedback with the community so that they can be refined and made even more useful. We look forward to hearing your insights!
An assessment is a process of data gathering and information analysis carried out in support of a predetermined purpose. Purposes of assessments can include the following, and may be multiple:
- Inform policy towards a particular country, sector or grouping;
- Support the design of programmes to assist the development of security and justice systems;
- Identify suitable entry points to support longer-term security and justice processes;
- Identify and mitigate against risks and pitfalls of engagement;
- Identify individuals, communities, networks, organisations or institutions to champion security and justice processes;
- Identify possible suitable local, national, regional or donor partners for future engagement;
- Inform funding priorities for support to SSR;
- Create a baseline from which to evaluate programme outcomes and impact.
Preliminary remarks on what makes security and justice assessments different
While many of the principles behind conducting an assessment are the same in security and justice development as in other development activities, the nature of donor support to security and justice initiatives requires a different framework. Security and justice development is inherently a highly sensitive and political process. The extent of political will to undertake (or to avoid) security and justice development must be understood. Assessments must not just focus on technical issues and institutions, but include political dynamics, individual influences, informal networks and customs. In addition, a realistic understanding is required of the extent of the impact a donor can have (both positive and negative). It is important to unpack and acknowledge the relationships within the partner country, as well as donor capabilities to support effective development given the constellation of partner country relationships.
In addition to a good mix of sectoral technical experts, security and justice assessment teams need an understanding of security and justice development as being a holistic and political process that requires the combination of effectiveness with accountability. There is also a requirement for project management skills, cultural and local political awareness, knowledge of good practice in assessment processes and change management experts.
These challenges are exacerbated in the sorts of fragile or conflict-affected state environments in which security and justice support is often undertaken. The security and justice system in a post-conflict society is characterised by political volatility, collapsed structures, institutional fluidity and general uncertainty: the nature and number of personnel, the levels of organisational capacity, the effectiveness of the security and justice institutions, the activities of non-state security and justice providers, and the security and justice needs of the population, are often not clearly known.
Fragile or conflict-affected societies are often confronted by legacies of massive abuse generating a lack of trust and a crisis of legitimacy of the security institutions. Typical other post-conflict processes such as DDR, elections, mine action, and return of refugees are highly resource intensive and politically demanding. The tension between the numbers of top priorities (everything is urgent) and limited resources often calls for a careful identification of suitable entry points, rather than just focusing on long-term reform.
Entry points are limited programme activities that are feasible and not threatening, but build confidence, open doors for long-term involvement, and contribute to a holistic reform vision.
Entry points should, therefore, be able to be implemented straight away and can be completed in a relatively short timescale.
Overview of ISSAT's approach to undertaking assessments
Assessments have traditionally focused on the analysis of security and justice in a country from the perspective of the provider – using institutional analyses that look for gaps when comparing the situation in the host country against an ‘ideal’ model. This, however, risks creating recommendations that fail to impact on the end-user of security and justice services and can end up being supply driven. They also fail to take into account the realities of the country and how security and justice are actually provided.
Experience has also shown that comprehensive assessments that aim to cover all actors in depth are ambitious, complex, resource intensive and time consuming. Comprehensive assessment efforts often falter with poor results that assess a limited number of actors unevenly. As a result, comprehensive assessments should generally be avoided, and focus given to areas where there is likely to be a greater positive impact.
Based on lessons identified and evolving good practice, ISSAT has selected a five phase process:
Phase 1. Determine the feasibility and appropriateness of an assessment
This phase covers the main questions that need to be asked on whether an assessment is the most appropriate action (looking at relevance and feasibility); checking against the main principles for supporting SSR and reviewing how these influence the decision-making process; formulating the assessment request (including defining the objectives and scope); and identifying resources.
Phase 2. Plan the assessment
This phase concentrates on ensuring the ground work has been made to incorporate the Paris Declaration and Accra Agenda for Action principles, as well as undertaking preparatory analysis covering past assessment reports, and awareness of existing political dialogue and the history of SSR in the country. The procedural aspects of the assessment mission that are covered at this stage include details on the assessment team, terms of reference and beginning to develop a context-specific methodology, including commissioning surveys where possible and appropriate.
Phase 3. Build an assessment framework
This phase suggests a framework within which an assessment team can work and that combines a ‘Needs Assessment’ approach (looking at the institutional gaps) with a ‘What is Used’ approach (whereby focus is placed on the end user, where they actually go to get security and justice, how security and justice services are delivered and by whom). The process is founded on four identified good practices:
- Evaluating the nature and structure of the State, particularly taking into account the governance structures and cultural norms that historically exist. This allows evaluations of justice and security institutions to be placed into the appropriate context, rather than gauging effectiveness against a Westphalian state model.
- Recognising the importance of personalised politics: the team need to map individual networks and examine leadership abilities, rather that just looking at institutional and capacity analyses. Unpacking the dynamic balance of power between different actors, communities and organisations – as well as their self-interests – is key, albeit extremely difficult. Particular care is taken where possible to vary the sources and examine opinions and judgements according to their power interests.
- Recognising the importance of the relationship between security and justice service providers and their customers. These relationships are examined to see how these can be progressively made more accountable and effective, taking into account the myriad of ways (state and non-state) that people currently use to find security and justice.
- Examining which constituencies are likely to support what initiatives, combined with an analysis of likely sources and levels of resistance.
A comprehensive risk management approach is adopted throughout due to the fact that knowledge will inherently be partial and incomplete, and therefore unintended consequences are virtually a certainty that need to be mitigated against.
Phase 4. Conduct the assessment in the field
This phase provides some guidance for conducting the assessment mission in the field and collect data according to the framework set in the previous phase.
Phase 5. Analyse the data and formulate recommendations
This is a specific stage of the process that provides the link between the assessment and any subsequent SSR support activities. It also provides the opportunity to collate and review lessons learned for future assessments. The exact process to determine recommendations will have been worked out with the mandator during the initiating/planning phases: for example, recommendations could be produced by the assessment team or could be explored through a stakeholder workshop. Four main parameters are considered: the life-span of the recommendation, the cost, potential spin-offs, and the political impact. Furthermore, recommendations need to be focused and feasible, with preconditions and consequences made explicit. Next steps are considered to clarify how the process will move from diagnosis to action and this will invariably require close liaison between the assessment team, the mandator and the host government. If national stakeholders are not party to the development of recommendations, bilateral or open meetings are highly recommended to validate findings, enhance national ownership of the process and build trust with key implementers.
Programme design is a process through which a programme is identified and defined. It draws upon insights gathered during the assessment phase to define a temporary, flexible organisational structure to coordinate, direct and oversee the implementation of a set of related projects and activities that aim to deliver a specific set of outcomes and benefits for key stakeholders.
The starting point for programme design is a programme mandate, which describes a high-level vision for change and required outcomes based on the strategic or policy strategic objectives of the national government, its international partners and the organisation or organisations that are the focus of the programme.
Guidance for this activity is currently under development.
Guidance for this activity is currently under development.
Evaluation is an important activity to be carried out in SSR projects, programmes and policies. As defined by the OECD, an evaluation is "the systematic and objective assessment of an on-going or completed project, programme or policy, its design, implementation and results. The aim is to determine the relevance and fulfillment of objectives, development efficiency, effectiveness, impact and sustainability. An evaluation should provide information that is credible and useful, enabling the incorporation of lessons learned into the decision-making process of both recipients and donors." OECD (2002),Glossary of Key Terms in Evaluation and Results Based Management.
In this section, you will find a useful and operational methodology to undertake both mid-term/on-going/formative evaluations and ex-post/summative evaluations of individual SSR projects, complex SSR programmes and/or SSR policies.
Throughout this methodology, specific evaluation terms are used and are underlined. By mousing over these terms, their definition will appear. They all come from the OECD (2002),Glossary of Key Terms in Evaluation and Results Based Management.
Preliminary remarks on the specificities of evaluating SSR
A Saferworld study on evaluating SSR and the OECD SSR handbook, section 10 on evaluation and monitoring, highlight a number of specificities/challenges for evaluating SSR. Some are common to all development aid evaluation exercises, and some are more specific to SSR. Here below are the most relevant for this OGN. Further details can be found in the above-mentioned documents.Preliminary remarks on the specificities of evaluating SSR
- SSR is often undertaken in fragile and conflict-affected environments:
- Lack of data
- Rapidly changing and insecure situation
- Limited design and bad planning due to time pressure
- Important to ensure conflict-sensitivity of the evaluation process and results
- SSR has a multi-sectoral and integrated/holistic dimension:
- The performance of one sub-sector is influenced by others. It is thus a challenge for attributing the performance when evaluating only one sub-sector.
- Evaluating a whole-of-government approach and/or an integrated approach requires specific methodology
- SSR is highly politically sensitive and has a culture of secrecy:
- SSR actors (e.g. soldiers, policemen, lawyers, judges and intelligence officers) often have limited understanding of evaluation processes:
- Hostility towards evaluation processes
- Difficult to reach agreement on what to measure and how
- Difficult to integrate the lessons learned
- Limited numbers of specialist SSR evaluators
- Local ownership of SSR and of the evaluation process is crucial:
- Difficult to evaluate local ownership
- Results of the evaluation process have to be locally owned. Evaluation findings should be shared with local partners and recommendations adapted to their needs.
The evaluation methodology laid out above is a useful tool to tackle most of these issues. It offers a structured approach to deal with these difficulties. It allows informing SSR actors on the evaluation process in a simple manner and get them involved in each step of the methodology. And it provides evidenced-based findings and recommendations which are key for the success of your evaluation. Of course, the situation on the ground is different from the theory. The activities, sub-activities and tools provided here should thus be adapted to each evaluation mandate.
Please click on "Start Prezi" in the middle of the box below to view a presentation of the evaluation methodology. Use the arrows that will appear at the bottom of the box to move forward and backward through the presentation. If you want to see the presentation in full screen, click on the full screen icon at the bottom right hand corner.