What is it?
Collecting data for an evaluation is a process of gathering the needed information to provide evidence-based findings and recommendations. Data collection takes place:
- At the planning phase and during the building of the evaluation framework. You gather data from different sources, on broad and specific issues, in an open way for capturing enough information in order to:
- Comment the ToRs;
- Elaborate your task plan;
- Describe the context of the project/programme/policy and the environment in which it takes place;
- Understand the logic of the project/programme/policy;
- Draft your Evaluation Questions, sub-questions and indicators.
- But also and mainly during the field phase. You gather data from different sources, in a structured manner, focusing on the Evaluation Questions, sub questions and indicators.
Why is it important?
To provide answers to your Evaluation Questions, i.e. your findings based on evidence.
How do we do it?
You can use different tools to collect data. It is highly recommended for SSR evaluations to use multiple tools to collect quantitative data (in numerical form) and qualitative data (in non-numerical form). For example, you can analyse statistics on the number of crimes (quantitative data) and complement it with interviews to obtain explanations (qualitative data) on the evolution of number of crimes over the years.
The most useful tools that can be use to collect data for an SSR evaluation are the following, but are not limit to:
- Interviews (Open, semi-structured and structured);
- Documents review;
- Focus groups;
- Direct observations;
- Case studies;
- Statistical analysis of security and justice-related indicators;
- Meta-analysis of previous evaluation reports.
The choice of the tools will greatly depend on your Evaluation Questions and the defined indicators. It will also depend on the demand from the mandator, the time and the resources available.
Type of evaluation
Data collection tools
Evaluation of a simple SSR project
Evaluation of a complex SSR programme
Ex-post evaluation of a SSR policy
Tips for collecting qualitative data (adapted from: The World Bank “The Road to Results”, 2009)
Use tools to keep track
Example from ISSAT experience
ISSAT is using a simple grid to gather and share the data collected during the field phase. The grid contains the Evaluation Questions (EQ), the sub-questions and the indicators. Moreover, a blank line is introduced under each indicator (or sub-question if there is no indicator). After each day, the team members fill in the grid with the data collected that is relevant to the indicators. Each team member has a color code to write down the information so that during the analysis phase, clarifications can be asked to the relevant team member if needed. This grid is a great tool to capture and store the information during the field phase but it also helps for sharing the information and moreover analysing the information under each indicator. Indeed, the team member responsible for drafting the answers to the EQ will have all the data related to each EQ, even though he did not collect all the data. This grid can then be included in an annex of the final report in order to provide the facts on which the answers to the EQ are based. Or it can be kept internally, if the mandator does not ask explicitly for it. It will always be useful to keep this grid to go back to the facts if you receive comments on the answers to the EQ or request for more evidence.
Interviews (Open, semi-structured and structured)
Interviews are one of the main data collection tool used during an evaluation. Three types of interview can be used: open, semi-structured (most commonly used in evaluation) and structured:
Open interviews: the interviewee expresses himself/herself freely and can discuss unplanned topics, because there is no predetermined set of questions. The evaluator intervenes only to generate and develop questions relating to the interviewee's comments.This type of interview is particularly interesting at the start of an evaluation, in order to get a global view of the subject, and identify the major topics and issues.
Semi-structured interviews: the evaluator uses an interview guide set prior to the interview in order to develop useful areas of inquiry. S/he also gives the opportunity to the interviewee to express himself/herself freely but brings him/her back on track if needed.This type of interview is the most frequently used, particularly when the evaluator knows sufficient about the aims and the main questions to pose during the evaluation.
Structured interviews: the evaluator follows strictly the interview guide's instructions. S/he asks different interviewees the same set of questions, in the same order, and using the same words. The evaluator avoids generating and developing additional questions, and the interviewee is not given the opportunity to express himself/herself freely. Answers to each question tend to be short. Structured interviews are seldom used in evaluation, where the evaluator needs to adapt to the situation.
General tips for interviews
Prepare your interview
- Draft a general interview guide with all the issues at stake. Then adapt it according to your interlocutor (donor’s HQ staff in charge of strategy, donor’s staff in the field in charge of managing the project/programme, direct beneficiaries, indirect beneficiaries).
- Structure your interview guide according to your Evaluation Questions (EQ). Indeed, you want to ask questions on relevance, effectiveness, efficiency, sustainability, impact, coordination, ownership etc. But you cannot ask the question: “what is the effectiveness of the project/programme”. However, your sub-questions and your indicators formulated under each EQ can easily be transformed into interview questions. Group some indicators or sub-questions in order to avoid having too many questions.
- Use observations from previous interviews to adapt your questions and fill the gaps.
- Ensure those you will interview are clear about the purpose of the interview and that they know who will interview them.
- Think through in advance who from the evaluation team needs to be involved in which interviews.
- Find out how interviews are normally conducted in the country/culture where you are. There will probably be protocols and expectations that are different to yours. For example, it may be considered rude if you begin the meeting before being welcomed by the interlocutor.
During the interview
- Conduct interviews as privately as possible, but preferably in the usual working place of the interlocutor, and during working hours. Holding the interview in the interlocutor’s working place can give you a sense of her/his reality as well as a further insight into her/his perspective, beyond the words exchanged in the interview.
- When you begin, start by explaining who you are, your mandate and background. Mention that the interview is confidential (when applicable) and that the meeting notes you are taking are only for internal purposes.
- Clarify the amount of time the interlocutor has for the interview – even if a set time was allocated last minute changes often arise. Doing this at the start of an interview reassures the person/people you are interviewing and also allows you to know if/how you need to adjust the interview to prioritise certain questions.
- Respect the speaker and keep in mind that you have to build a relationship based on mutual confidence that will last beyond the field mission.
- Consider if you can benefit by repeating some of your questions in different ways. This is sometimes necessary if the interviewee seems not to understand the purpose behind the question; or if there are language or accent difficulties.
- Listen and avoid interrupting the interviewee; even if s/he drifts off topic, s/he may share relevant information that you did not expect from her/him. Always ask if s/he has something to add at the end of the interview. Don’t commit to support if you don’t have the mandate to do so and try to manage expectations of the interviewee.
- Depending on the general context, stay critical on the answers given; interviewees may say what they think you want to hear.
- As you get toward the end of the interview, ask yourself if and why something is missing from the picture.
- As you close the interview and thank the interviewee, clarify what will happen next and if (should the need arise) you can contact her/him again to follow up on any points raised.
- If appropriate, ask your interviewee if you can select a few individual citizens (service recipients) and privately interview them. Or, if you are planning to interview citizens and you do not need to seek permission for this from the interviewee, simply inform her/him if appropriate.
Further considerations for interviews
- For the wider interviews that you may conduct with individual citizens, ask what problems they face, what services they expect, and in which conditions services are delivered (timing, cost, assistance, etc.). Ask how they solve problems related to policing, justice and corrections, and what difficulties they face in dealing with police, justice and corrections institutions. Corroborate/triangulate information by consulting a wide range of sources.
- Interviewees should include: representatives of all police, justice and corrections actors (ideally of every professional category, both from the management and the working levels); central and local governments; lawyers and bar associations; bailiffs; unions of police, justice and corrections professionals and other unions; academic institutions; members of the public (samples of the service beneficiaries); representatives of victim and other marginalised groups; detainees and prior offenders; journalists; local, national and international NGO’s; other international partners (United Nations agencies, regional organisations, donors, etc.).
2. Document Review
Reviewing documents may provide useful information throughout the evaluation. It is one of the main tools used in evaluation that are limited in time and resources. Indeed, if you do not have time to gather yourself the primary data, you may use existing reports, data sets gathered by others, newspaper articles, web pages, etc. These are secondary data that will help you answering your evaluation questions.
The type of documents to be reviewed will depend on the objective and scope of the evaluation. For an evaluation of SSR projects/programmes/policies documents to be reviewed include but are not limited to:
- Mandator’s policies/strategic documents.
- Partner country’s policy documents, poverty reduction strategies, security strategies etc.
- Other donors’ policies/strategies in SSR, or in the country/regions where the project/programmes take place.
- Civil society reports on security, justice, human rights, governance, corruption, vulnerable groups etc.
- International organisations/think tanks data sets on security and governance issues.
- Project/programmes documents, logframes, progress reports, monitoring and evaluation reports.
All the documents used should be included in a bibliography.
Reviewing documents may take place at different phases of the evaluation and may have different objectives. Before starting the review, it is important to determine which objectives you will address through the review, what type of information you are seeking and how you will track the information you find (i.e. how you will code or organise the data you collect through the review):
- When planning the evaluation: the document review aims at gathering general information on the project/programme/policy under evaluation. It should aim at supporting the drafting of the ToR, elaborating the work plan and defining the responsibilities within the evaluation team.
When building the evaluation framework: the document review aims at understanding the context in which the project/programme/policy is taking place, the logic behind the design of the project/programme/policy and the main key issues, preliminary difficulties and successes. Examples of documents to be reviewed in this phase include, but are not limited to:
- the general security and justice characteristics, challenges of the country in which the project/programme/policy is taking place;
- the overall national and donor strategy (country strategy and/or sector strategy in SSR, police reform, defence, justice, etc.);
- the logical framework of the project/programme/policy under evaluation, the planning documents (e.g. identification mission, assessments, etc.);
- baseline surveys/studies.
- evaluation reports from previous project/programme/policy in the same field.
- When collecting data in the field phase: the document review aims at gathering data for answering your evaluation questions, and more specifically for substantiating your indicators defined under each evaluation question. Documents like progress reports, monitoring reports and data, survey results, meeting notes, previous evaluations, etc. can provide useful information for the indicators. Before reviewing the documents it is important to have all the relevant indicators in mind. Then, when finding relevant data, indicate to which indicator it relates. This will ease the process of analysing all the data collected for each indicator for the drafting of the final report.
Tips for document reviews
- Specify what data to collect and how to code it. Are you looking for general data or data for specific indicators? Identify a simple way of coding the data collected so that you do not lose it and can come back to it during the analysis and the drafting of your findings. It is important that the entire team uses the same coding so that data collected can be used by others.
- Understand the limitation of the documents or the data records. For examples data sets are usually made for a specific purpose which might not fit exactly yours. Be aware of that and ask questions to people have recorded the data.
- Address issues of confidentiality of information. Always refer the sources of documents when quoting data.
- Ensuring that the team can always refer to the source of the document when using the data collected.
3. Focus Group
A focus group can be a useful tool to collect information from a small group of participants in a systematic and structured format.
It allows identifying issues that might not have been registered through other evaluation tools. Indeed, the advantage of using focus groups over individual interviews is that it allows participants to benefit from comments of others to spur their own thinking about the subject at hand, generating views and insights that may otherwise not have been arrived at. As a rule, a focus group consists of 4-12 participants, and is led by a neutral moderator. The role of the moderator is to guide the discussions. These should normally be framed around a single and clear question (e.g. the impact of the project). He/she is to ensure that everyone’s views are being heard and should try to identify areas of consensus among the participants.
Different methodologies are used in organising focus groups. The most frequently used methodologies include the following:
- Brainstorming: this can be done by the group as a whole, or by asking each participant individually to note down their initial thoughts, which are then later shared with and discussed by the group (this is the so-called “nominal group technique”).
- SWOT: an analysis of strengths, weaknesses, opportunities and threats as a guiding methodology for a group discussion.
- Straw polls: this entails conducting an informal vote in order to quickly gauge opinions among participants.
These methodologies do not have to be strictly followed however. More importantly, insights derived from focus groups can be of even greater use if they can be “tested” or verified across different groups simultaneously, as well as over time at different intervals. This can help to control for undesired effects such as “groupthink”.
- Questions – To stimulate discussion among participants, questions asked should be open-ended.
- Moderator(s) should not only be knowledgeable on the subject matter but also be seen by the participants as neutral.
- Logistics – The venue of the focus group session should be non-contentious, easy to reach, and equipped with all the necessary facilities. The seats in the meeting room(s) should be arranged in such a way as to stimulate conversation (usually in a circle). Persons with identical interests or who may intimidate each other should not be seated next to each other.
- Participants should be selected so as to arrive at a representative sample of the target group in relation to the issue being evaluated. To stimulate a frank discussion, it would be best if the participants do not know each other or the moderator(s), even if in practice this may not be easy to achieve. It is also advisable to put together participants of the same social class, gender or interest group to avoid people from not speaking up because they may feel intimidated by the presence of people with different interests.
- Costs and budget – Generally, focus groups are seen as an inexpensive means for gathering information. The bulk of the cost will relate to travel costs and accommodation for participants, renting the facilities (if needed) and the recruitment of moderators. To entice people to sign up, small honorariums could be offered to participants. But by no means should that be a reason for participation.
Conducting a Focus Group
- Capture the findings – The best way to record the comments of participants is to capture the comments on video or audiotape. Obviously, permission to use such equipment should be obtained from participants in advance, while the camera or tape recorder and accessories should be located in an unobtrusive spot in the room where the discussion takes place. To avoid views from being missed due to a defect of the technology, note-takers should also be present to record the statements.
- Ground rules – Before the discussion starts, the moderator(s) should present the ground rules, explain the purpose of the exercise, and what will be done with the results. To get all of the participants to speak, a warm-up exercise may be carried out.
- Equal participation – An important role for the moderator(s) is to prevent one or a few participants from dominating the conversation, as well as to avoid the focus group from degenerating into a bitter exchange. If necessary, the moderator should draw more taciturn participants into the discussion with easy and open-ended questions and, in extreme cases, be prepared to ask dominant participants to be silent or even to leave the group.
- Conclusions – Finally, the moderator(s) should encourage the group to come to one or a few broad consensus-based conclusions, and should summarise these so as to get approval from the group. This will help all participants to feel that their contributions are valued and have been taken into account, and will also stimulate them to take part in a subsequent focus group exercise.
- Duration – A single session should last between anything from 45 minutes up to three hours, and could include one or several breaks.
Post-focus group analysis
- Views captured – Immediately following the focus group session, the moderator(s) and organisers should get together to evaluate the information that was expressed during the session.
- Preparing a report – Based on the evaluation and the transcripts of the session, a report should be put together wherein the views of all participants should be reflected, as well as the conclusions that were reached in the focus group(s). Quotes from individuals can be included, but should not be attributed.
4. Direct Observation
Observations or sites visits
Observations or sites visitsare a way to collect descriptive information about what happens (or has happened) in a project/programme mainly on its activities, processes and outputs. It also helps to crosscheck findings from interviews or other data collection tool. Generally, observations or site visits should be carried out with key informants.
Evaluators would carry out observations or sites visits in order to:
- determine whether the programme’s processes are being implemented as intended;
- observe activities’ physical outputs and satisfaction/use of these outputs by the intended beneficiaries;
- understand actual behaviours of individuals or groups rather than self-reports or perceptions;
- learn about unanticipated effects of the project/programme;
- provide an alternative data collection tool when others are inappropriate or infeasible (e.g. in some culture, some questions are forbidden to ask through interviews. You may however want to see it by yourself);
- Repeated observations or sites visits can provide information about changes over time.
You may want to take notes and photos but always ask permission. Taking photos can be very sensitive in prisons, military camps or police stations.
Be aware of the following:
- Observations may be expensive and time consuming. Organisation, travel and security.
- Observations may be intrusive and may influence the event being observed.
- Key informants may guide you through the most successful activities and outputs. They may give you only a partial view of the reality.
- Confidentiality of participants/beneficiaries of the projects/programmes must be considered.
Transect walks are a specific form of observation. During the walk with key informants along a defined route across the community, the evaluation team discusses anything noticed, facilitates exchanges by asking questions and making observations, and informally interviews any people met during the walk to get their views on the security threats, conflicts, abuses and service delivery deficits in the community. Transect walks introduce the evaluation team to the community and its inhabitants.
A survey is a tool to collect data about people’s perceptions, opinions and ideas. It can be used in assessment (to establish a baseline) and/or evaluation (to answer your evaluation questions) missions. It is used to collect data in a standardized manner, asking the same questions in the same way and sequence to a large number of individuals. It can be administered orally (face-to-face or telephone) or in writing (email or on-line survey). It allows you to obtain quantitative data and to a lesser extent qualitative data (though open questions in a survey). It is recommended to use survey in addition to other data collection tools (interview, observation, etc.) in order to cross-check the information and interpret the data.
Carrying out a survey is a lengthy process which requires sufficient financial and human resources. Questions should be well chosen (open or closed) , the wording should be clear and appropriate, the sequencing should be logic and the layout should be attractive. The response choices are also very important. The administration of the survey and the analysis of the data collected are lengthy processes. So be aware of this when planning the undertaking of a survey.
For concrete information on how to carry out a survey, please refer to this useful document drafted by Search For Common Ground: http://dmeforpeace.org/sites/default/files/2.6%20Survey.pdf
Free online survey: