13 - Monitoring and Evaluating Participatory Research and Development: Some Key Elements: International Development Research Centre

International Development Research Centre (IDRC)

Web Archives > Publications > IDRC Books > All our books > PARTICIPATORY RESEARCH AND DEVELOPMENT FOR SUSTAINABLE AGRICULTURE AND NATURAL RESOURCE MANAGEMENT: A SOURCEBOOK >

Topic Explorer

People

ID: 85056

Added: 2005-07-18 15:16

Modified: 2007-03-25 21:40

Refreshed: 2012-02-10 18:30

Click here to get the URL for the RSS format file

13 - Monitoring and Evaluating Participatory Research and Development: Some Key Elements

Document(s) 16 of 34

Ronnie Vernooy

The deliberate and careful integration into the project cycle of monitoring and evaluation activities can strengthen the learning, accountability, and effectiveness of research efforts. Using a participatory approach to do so facilitates the realization that what matters is not only what is assessed, but who does the measuring and assessing. In addition, such an approach can contribute to a better understanding of how different concerns and interests are represented and negotiated in a research process. In other words, it helps to understand and assess how and why participation takes places or does not take place.

For more information, refer to: McAllister, K. and R. Vernooy. 1999. Action and Reflection: A Guide for Monitoring and Evaluating Participatory Research. Ottawa: IDRC. http://web.idrc.ca/cbnrm

Monitoring is the systematic, regular collection and occasional analysis of information to identify and possibly measure changes over a period of time. Evaluation is the analysis of the effectiveness and direction of an activity or research project and involves making a judgement about progress and impact. The main differences between monitoring and evaluation are the timing and frequency of observations and the types of questions asked. However, when monitoring and evaluation are integrated into a research strategy as a project management tool, the line between the two becomes rather blurred. Participatory monitoring and evaluation (PM&E) is the joint effort or partnership of two or more stakeholders (such as researchers, farmers, government officials, extension workers) to monitor and evaluate, systematically, one or more research or development activities (Vernooy et al., 2003).

In designing monitoring and evaluation activities it is useful to consider six interrelated questions:

Why monitor and evaluate?

For whom?

Who will monitor and evaluate?

What will be monitored and evaluated?

How?

When?

We discuss here the Why? and What? questions and also the very important issue of appropriateness. All six questions are discussed in more detail in McAllister and Vernooy (1999).

WHY to Monitor and Evaluate?

In general, goals can be:

To assess project results: to find out if and how objectives are being met and are resulting in desired changes.

To improve project management and process planning: to better adapt to contextual and risk factors such as social and power dynamics that affect the research process.

To promote learning: to identify lessons of general applicability, to learn how different approaches to participation affect outcomes, impact, and reach, to learn what works and what does not, and to identify what contextual factors enable or constrain the participatory research.

To understand different stakeholders' perspectives: to allow, through direct participation in the monitoring and evaluation process, the various people involved in a research project to better understand each others views and values and to design ways to resolve competing or conflicting views and interests.

To ensure accountability: to assess whether the project is effectively, appropriately, and efficiently executed to be accountable to they key agencies supporting the work (including, but not exclusively, the donors) (Estrella and Gaventa, 1998).

Usually, a monitoring and evaluation plan includes a combination of these goals, but it may be necessary to put more emphasis on one of them, depending on available resources, skills, and time and on the point in the project life-cycle during which the monitoring and evaluation will be done (see Table 1 for an example).

Table 1. Monitoring and Evaluation Plan Proposed by the Guizhou Academy of Agricultural Sciences CBNRM Project Team
Project research topic: Water resource management
Why conduct PM&E?
1. To identify problems, to analyze reasons, to find out solutions, and to improve project plan and implementation.
2. To find out to what extent the project research meets the needs of the local people, local government and researchers.
3. To find out to what extent the project facilitates the sustainable utilization and management of natural resources.
	Outputs	Outcomes and impact?	Reach
What?	Effective management group Effective management regulations	Improved management capacity of community organization Value of and attitude toward resource use	Households within or outside the project area, local governments, local hydraulic department, project researchers and visitors
Indicators	Improved services of management staff Fairness of water distribution Increased irrigated areas (village level and household level) Effectiveness of facility operations	Timeliness of water distribution Ease of water fee collection Decreased conflicts in water distribution Labor mobilization for building and maintenance of facility	Number of visitors Times of experience sharing with other researchers Number of villages that have adopted the measures of water resource management of the project
Who?	Management group Beneficiary households Researchers	Local households Researchers Local government Village committee	Local households Researchers Local government Village committee
For Whom?	Local government Target community Hydraulic department Researchers	Local government Target community Researchers	Local government Target community Researchers
When?	Following the workshop, during last year of project implementation
Tools	Matrix scoring Semi-structured interviewing at household and group levels Sampling survey Community resource mapping Self-monitoring book
Source: Adapted from Vernooy et al., 2003

Efficiency, Effectiveness and Relevance

Underlying reasons for monitoring and evaluating are frequently framed in terms of:

Efficiency refers to the amount of time and resources put into the project relative to the outputs and outcomes. A project evaluation may be designed to find out if there was a less expensive, more appropriate, less time-consuming approach for reaching the same objectives.

Effectiveness describes whether or not the research process was useful in reaching project goals and objectives, or resulted in positive outcomes.

Relevance or appropriateness describes the usefulness, ethics, and flexibility of participatory research within the particular context and for the particular research question.

Combined, these criteria enable judgment about whether the outputs and outcomes of the project are worth the costs of the inputs. Effectiveness, efficiency and appropriateness can be considered for the different methods, tools and approaches rather than questioning the value of the research approach as a whole.

In this context, the efficiency of a particular method or approach can consider factors such as the time involved for local people balanced against the value of the information gained and whether this information was available through other means, or whether or not the accuracy or the detail of the information gained from the research method warrants the extra time taken.

Effectiveness of particular participatory methods can consider whether or not the approaches or methods allowed representation of different local interests, whether they were able to generate desired results, whether or not they encourage strengthening of local individual and organizational capacity, and whether or not they encourage farmer experimentation.

Relevance or appropriateness relates to the flexibility of the process to adapt to the local context and emerging needs, whether or not the tools are suitable to the capacities of the researchers and community, and whether or not the approach is reaching stakeholders at the scales relevant to be effective for addressing the research problem.

It is important to define from the outset what weight will be given to each of these dimensions.

WHAT to Monitor and Evaluate?

Understanding the condition of the community before the project was initiated is useful in order to provide a point of comparison for monitor and evaluating changes that occur during the project and to understand how the research process contributed to these changes. Participatory baseline analysis conducted at the beginning of the project can provide a point of reference for comparison and for understanding change in the community. It is useful to distinguish between the different kinds of results generated from the research: outputs, processes, outcomes, impact and reach. These can be briefly defined as follows:

Outputs describe the concrete and tangible products of the research as well as the occurrence of the research activities themselves (see the Guiding Questions 1 below).

Processes describe the methods and approaches used for the research.

Outcomes describe the changes that occur within the community or with the researchers that can be attributed, at least in part, to the research process and outputs.

Impact describes overall changes that occur in the community to which the research project is one of many contributing factors. One such impact often expected from participatory research is social transformation (see the Guiding Questions 2 below).

Reach describes who is influenced by the research and who acts because of this influence.

GUIDING QUESTIONS 1: Assessing the Quality of Common Outputs of Participatory Research

New technologies or production systems developed in partnership with local people and researchers (agro-forestry, soil-conservation, farming systems, etc.)

Are these based on priorities identified by local people? Were local people involved in the development or experimentation process? Were gender and social factors taken into account?

Are technologies still being used or adapted by local people? By whom?

Have local people adapted the experimental approach to other areas of their livelihood?

Has the innovation been taken up by other people who did not participate in the study?

Have people been teaching each other about the use of the technologies?

Community-level organizations created or strengthened

Who is actively involved? How did these people participate in the research?

Is there an active leadership?

Whose interests are represented by the organization or by the leaders? Do gender and social factors influence the functioning of the organization?

Are the interests of less powerful groups represented?

Are the organizations and leaders accountable to the community? Are they representative of important stakeholders? Are they legitimate in the eyes of the community? What is the motivation for people's involvement?

Community-based management systems

Are local people able to systematically monitor the results of their activities and adapt activities that are not sustainable?

Are they able to enforce sustainable practices? How do they ensure compliance?

Is there equity in representation and participation? Do gender and social factors influence representation and participation?

Is there an effective or improved forum or mechanism for conflict resolution concerning the use of common resources?

Are methods for decision-making improved or more representative of various interests?

Are less-powerful voices included in decisions?

Is there strength in the leadership?

Is there a system of accountability, and to whom is the system accountable?

GUIDING QUESTIONS 2: Assessing the Potential of Participatory Research to Result in Social Transformation

Strengthening local awareness of issues and options

Is the research process increasing local awareness of issues and facilitating them to develop local options for improving their situation?

Participation of local people in decision-making, planning and action to address problems

Is the participatory process facilitating local involvement in decision-making and action to address problems?

Who in the community is involved and whose perspectives are being represented?

Perception of ownership of the process

How do local people perceive the research in terms of whose research is it?

Who controls the research questions and agenda, and to what extent are the issues and questions defined by the researchers?

Are local people involved in identifying and defining research priorities and plans? In data collection and analysis? In defining solutions and actions? In monitoring results of their activities or experiments and in defining their own indicators and criteria for success?

Strengthening existing individual and organizational capacities

Has the research identified and made explicit existing individual and community capacities (existing resource management norms, decision-making processes, conflict management skills, etc.)?

Is the research process strengthening these individual or group capacities and organizational skills?

Is the process contributing to individual and community awareness of local problems and strengthening their ability to deal with them effectively?

Is the process strengthening community capacity and motivation to continue activities such as resource management, or is community motivation dependent on researcher facilitation?

Creating linkages between stakeholders

Have the researchers identified existing linkages, and areas where linkages need to be made in order to effectively address the research problem?

If appropriate to the research question, have the researchers been able to encourage participation of stakeholders at different levels of governance and create linkages between these stakeholders?

Have they been able to create forums or networks for negotiation or information sharing between these different groups, or between groups of similar interests (e.g., farmers)?

Source: Adapted from McAllister and Vernooy, 1999

Appropriateness

Appropriateness and risks concerning monitoring and evaluating participatory research can be considered from various perspectives. In terms of risks, research may not be able to meet its goals and/or could unintentionally cause harm to the community or to specific groups within the community despite achieving it goals. For example, research aimed at sustainable community management of common resources may be manipulated by more powerful stakeholders and may unintentionally neglect representation of marginal groups or women. In consequence, these groups may lose access to important resources.

These kinds of social risks research need to be carefully anticipated during proposal development and monitored throughout the project to ensure that specific groups are not significantly disadvantaged by the research. Careful anticipation of social risks involved in the research can help establish the need for care in identifying the different groups who might be affected by the research. Potential risks from participatory research and from not recognising and involving stakeholder groups can be anticipated before the project begins. This brings us to ethics.

Ethics

Creating unrealistic expectations for concrete development interventions at the community level is a common problem for participatory research projects and causes disappointment and suspicion in the community. This can also have negative consequences for future work of the research institutions by affecting their acceptance in communities. This issue should be addressed in the proposal. There are several possible ways to deal with local expectations.

The participatory research project could be linked to a development initiative that has the mandate to provide concrete services to the community. However, this is not always an option.

Researchers should be clear in the proposal about how they will be transparent to the community about the goals of the research and what the community can realistically expect to gain.

The proposal could demonstrate a mechanism for generating some small concrete livelihood benefits to the community early on in the research process, such as small rotating credit schemes or helping establish seed banks, so that local people see benefits from the time they have given to participate in the research.

Confidentiality of information and security about how information gathered from community participation will be used is important because often researchers discover activities that would be illegal according to the state (for example, capturing of endangered species or logging in protected areas). Furthermore, information concerning resource ownership could be used by the government, for example, to extract taxes. It is important that the researchers address this issue in the proposal, and that they ensure that the identities of informants are concealed in their research notes and reports. One way of doing so is by using numerical codes for interviewees, and keeping their identities separate from the research documents. In addition, aliases should be used in reports which will be made public and which include anecdotal information from specific individuals or groups.

Informed consent from local people and groups for participation in participatory research is not as simple as it seems, and in many cases, gaining genuine informed consent for community involvement in the research process is difficult. Obstacles include:

The concept of informed consent is not always clear among researchers, let alone among community members. Researchers may not respect or understand peoples' wish NOT to be involved.

The risks of involvement in the research process may not be apparent to either the researchers or community - therefore it may be difficult to estimate the costs and benefits of participation.

Power relations between researchers and community, and within the community itself may result in coerced consent. Individuals may feel they cannot refuse involvement because of pressure from village leaders or government officials. In addition, cultural/social relations of respect for researchers may make it impolite or socially unacceptable for local people not to agree to participate.

Anticipation and expectation of benefits by community members from their participation in the research process may lead people to participate, even if the limitations of potential benefits has been articulated by the researchers.

Informed consent is related to transparency of the whole research process; hence, this includes the monitoring and evaluation activities. Researchers should address the informed consent issue upfront in the proposal (see the Guiding Questions 3 below).

GUIDING QUESTIONS 3: Assessing Appropriateness of the Participatory Approach

Type of participation

What is the level of community involvement in and control over the research and is this appropriate for the goals of the research? Have these questions been discussed up-front at the start of the research?

Is the community benefiting from the research? Who in the community benefits?

Transparency of the research process

Are the researchers transparent about the limitations and scope of the participatory research activities?

Are local people aware of these limitations or do they have unrealistic expectations?

Are local people aware of the overall goals of the research and do they understand these goals? Have they agreed to them?

Motivation for participation

Are local people participating and how?

Why are people motivated to participate? Is participation voluntary or compliant?

Do local people perceive that they are benefiting from their participation in the research?

How is the research process benefiting from community participation?

Relevance of the methods and approaches to the local context

Is there a process for local feedback into the research design?

Is there a systematic mechanism for occasional reflection and interaction between researchers and local people?

Are the results from community participation informing the research design?

Are the research goals and methods being redefined and adapted as the research proceeds?

Are the methods and tools effective for encouraging participation and representation? For strengthening local capacity? For enabling community-ownership of the process? For reaching objectives and goals of research?

Source: Adapted from McAllister and Vernooy, 1999

References

Estrella, M. and J. Gaventa. 1998. Who Counts Reality? Participatory Monitoring and Evaluation: A Literature Review. Prepared for the International Workshop on Participatory Monitoring and Evaluation: Experiences and Lessons, held at the International Institute of Rural Reconstruction (IIRR), Cavite, Philippines, November 24-27, 1997. IDS Workshop Paper 70. Sussex, UK: Institute of Development Studies.

Margoluis, R. and N. Salafsky. 1998. Measures of Success. Designing, Managing and Monitoring Conservation and Development Projects. Washington DC/Covelo, California: Island Press.

McAllister, K. and R. Vernooy. 1999. Action and Reflection: A Guide for Monitoring and Evaluating Participatory Research. Ottawa: IDRC.

Vernooy, R., Sun Qiu and Xu Jianchu (eds). 2003. Voices for Change: Participatory Monitoring and Evaluation in China. Ottawa: International Development Research Centre and Kunming: Yunnan Science and Technoloy Press.

Contributed by:
Ronnie Vernooy
Email: rvernooy@idrc.ca

Document(s) 16 of 34

guest (Read)(Ottawa) Login