Document(s) 12 of 18

Responsive, clear, and action-oriented evaluation usually addresses and balances three key questions and can be encapsulated as follows: Who wants to know what for what purpose?* As the many different whos, whats, and whys pull the evaluation process in different directions, it is often the voice of the client, beneficiary, or target participant that gets lost. In the rush to gather information and assess program impact, the needs and voices of donors and project implementers generally overpower those of the actual participants, and important information is lost.

This chapter provides the opportunity to discuss how this voice can be brought back into the process. In doing so, it addresses the definition, purpose, impact, and potential of participatory evaluation. To add implementation issues into the mix, the chapter also supplies an outline of a training program designed to provide field-workers with the skills and experience to facilitate participatory evaluation with women's groups.

What Is Participatory Evaluation?

Let's start this exploration with a definition provided by one of the Kenyan participants at the end of the second participatory evaluation workshop. In response to the question What is participatory evaluation? she wrote: "It is a democratic approach [for] examining the values, progress, constraints, and solutions of individuals, groups, or group activities by involving all people. It recognizes and values the subtle contributions of grassroots people, and grass-roots workers plus the communities. And believes that all human beings are capable of receiving, and coming up with ideas which may be used to make

This paper was first presented at the Sixth Annual Conference on Ethnographic and Qualitative Research in Education, University of Massachusetts, Amherst, 1994. The paper was a product of input by the staff of the YWCA of Kenya through many workshops facilitated by Bonnie Mullinix.

* Dr. David Kinsey, Center for International Education, University of Massachusetts, Amherst, developed this structure in the early 1980s to help move beyond "why" (the classic third question) and have participants in evaluation address how they hope to utilize findings.

better their socioeconomic status—but as long as they are empowered to know and believe that they can be and are in control of their destiny."

While some may wish to polish or modify this definition, it is offered here because it captures several key aspects of participatory evaluation that make it different from traditional evaluation practices. It also reflects both the impact of the training program and the existing philosophical orientation of the guiding organization (the YWCA of Kenya).

What Makes Participatory Evaluation Different?

As captured in the above definition, participatory evaluation has certain characteristics that set it apart from evaluation that assigns a role to participants. These include:

• Origin of purpose/questions: Evaluation questions emerge from the interests and priorities of the participants.*

• Extended usefulness/application: Participants develop an understanding of the purpose and importance of evaluation and the ability to conduct meaningful evaluations.

• Skills development: Through participation in the evaluation activity, participants develop the ability to collect, analyze, and act on information.

• Locus of control: Involvement in participatory evaluation activity empowers participants to take responsibility for assessing and articulating the impact a project has had on them according to their priorities.

Basically, when done properly, participatory evaluation promotes empowerment, confidence, self-esteem, and independence.

A Context for Exploration and Application

Utilizing participatory evaluation requires a commitment to and understanding of the purpose and benefits it can bring to both a project and an organization. As a membership organization that balances the needs of its members against the requirements of donor agencies, the YWCA of Kenya is such an organization. It has struggled for many years with the failure of traditional evaluation mechanisms to adequately identify and address the impact of its programs. Established guidelines often dictated that the impact of a project be measured by the degree to which it affected the financial status of its participants. The collective experience of the staff of the YWCA of Kenya at village,

* The term participants refers to the women and men involved in and most directly affected by a project. As participants in the participatory evaluation process, field-workers are able to contribute concerns and questions from the NGO/donor community (provided they do not dictate and drive the procedure).

branch, and national levels told them that there was much more going on, and that donor focus and time frames were often too narrow to accurately capture the true impact of projects on women and their groups. Given the need to define its actions based on member input and the general support for institutional development provided by the United States Agency for International Development, the YWCA chose to explore participatory evaluation as a strategy for capturing some of this lost information. In articulating its rationale for this selection, the YWCA cites the need to identify what its members (not donors) believe to be important, as well as the desire to share the skills and capacity to conduct evaluation activities with the women.

A Structure for Implementing Participatory Evaluation

In order to adopt participatory evaluation as a key evaluation strategy, several things needed to happen:

1. Field-workers and YWCA staff needed to have the opportunity to explore what evaluation was in general, and what participatory evaluation was in particular.

2. They needed to reflect on and structure mechanisms for sharing what they had learned with members of women's groups.

3. They needed to spend time in the field conducting participatory evaluation.

4. If impact was to be measured, this activity would need to span at least two years.

5. If women were to appreciate the value of evaluation, activities had to be useful from the start.

In response to these requirements, a two-year scheme was structured. This began with a training workshop for field-workers and staff with a focus on evaluation as a needs assessment and planning tool. This workshop incorporated a three-day village-based training and evaluation exercise with three separate women's groups. The training introduction was designed by participants as a participatory activity that could help explain the purpose and importance of evaluation and have participants reflect on their group needs and possible action plans. Role plays, skits, case studies, and small-group activities provided effective mechanisms for generating understanding and ideas from within the group. From these initial ideas, the women generated questions and designed and carried out a research and evaluation activity to gather information and determine which course of action might be most beneficial to the group. This phase of the activity provided participants with an introduction to process and skills and a practical understanding of what participatory evaluation is and how it works.

The second phase of the activity took place a year later and focused on consolidation of skills and conduct of an impact evaluation. Participatory evalua­tors (trainees) designed participatory reflective processes utilizing memory-activation mechanisms (materials, reports, stories, and photographs from the year before) that allowed participants to fix a reference time in their minds. This collective experience and memory made it possible for them to look back on activities that had happened over the past year and consider the changes that they saw in their group, families, communities, and themselves. The resulting discussion of impact was both broad and powerful.

Training Support for Implementation

The following is an overview of the information and activities included in the Participatory Evaluation Training Workshop Series and field activities.

First Participatory Evaluation Training Workshop

The first workshop on participatory evaluation was an eight-day workshop for branch and national-level staff of the YWCA of Kenya and three women's groups in the Mombasa region. As it was considered to be the first in a two-part series of workshops, it focused on establishing a participatory evaluation practice that could be monitored and completed in nine to twelve months' time.

This first workshop then, maintained and met the following objective: to orient participants to evaluation in general and participatory evaluation in particular through training, discussion, and practical application of techniques. By the end of the workshop, participants had

1. Discussed definitions, purposes, and types of evaluation;

2. Listed the steps in evaluation and dimensions of evaluation;

3. Identified guidelines for conducting evaluation;

4. Discussed issues surrounding participatory evaluation;

5. Described a variety of evaluation tools and discussed their relative advantages and disadvantages;

6. Developed and implemented a participatory evaluation activity with a women's group;

7. Identified key issues and components of an evaluation plan for the YWCA of Kenya; and

8. Experienced a range of evaluation techniques and shared and evaluated their own experiences during the workshop.

Second Participatory Evaluation Workshop—Impact Evaluation

The second workshop on participatory evaluation was a five-day follow-up workshop for branch and national-level staff of the YWCA of Kenya and three women's groups in the Mombasa region. As the second part of the two-part series of workshops, it focused on helping the members of the women's groups to be able to assess and articulate changes in their group, individual, family, and community lives and identify the impact of group activities after twelve months.

The second participatory evaluation workshop had the following goals and objectives: to allow participants to conclude their exploration of participatory evaluation, identify field-generated indicators for success, determine reporting structures and mechanisms, and generally consider the importance of participatory evaluation as a tool for planning and impact assessment. By the end of the workshop, participants had

1. Conducted participatory evaluation follow-up with women's groups that had participated in the 1992 activities;

2. Utilized women's opinions as a basis for identifying indicators for evaluating impact;

3. Explored and utilized mechanisms for assisting in memory activation and access;

4. Developed a process for helping women's groups to collect, record, and analyze information that will allow them to evaluate group activities;

5. Determined guidelines for keeping staff journal entries and utilizing these for evaluating subtle indicators; and

6. Developed a basic format for and a written sample of evaluation reports.

Reporting Participatory Evaluation Findings

The primary products of the participatory evaluation activities were reports on the participatory evaluation of women's groups. The following format for participatory evaluation reports was produced by participants based on their field experience and data collected.

1. Introduction

• Rationale/reason for evaluation

• Description of methodology

• Review of previous evaluations & findings

2. Background information on group to include

• Location: division/district/province

• Year when started

• Membership

• Year when joined YWCA

• Other points of interest (as related by the group)

3. Summary/recommendations

4. Members' backgrounds to include

• Age

• Social status

• Family size

• Economic status

• Marital status

• Religious affiliation

5. Leadership structure

6. Group activities

• Programs/projects/social, economic, educational activities

7. Resources/assistance provided

• Technical assistance

• Financial assistance

• Educational assistance

a) Trainings/seminars/workshops organized by

• YWCA

• Other governmental organizations/nongovernmental organizations (NGOs)

b) Exchange visits/programs

8. Impact of activities (social/economic/educational/political)

Benefits to group

• Dividends

• Soft loans

• Merry-go-round

• Training

• Individual savings

• Sharing ideas

• Fellowship/moral support

Benefits to individuals

• Income (for business/self)

• Promotion of business

• Improved standard of living

• Improved self-confidence

• Status in/recognition by community

Benefits to family

• Increased income to family

• Support from group

• Improved standard of living

Benefits to community

• Service to community (committee membership)

• Voting

• Community support/role models

9. Group levels

• Where they are now (present)

• Where they were (past)

10. Future plans

• For existing programs/projects

• For new programs

• For technical skills—trainings/seminars

11. Comments/observations

Evaluating Participatory Evaluation

Obvious questions arise about the comparative benefits and disadvantages of participatory evaluation versus traditional evaluation. Some of the more critical questions revolve around time to carry out an evaluation, the quality of the information collected, the training support required for evaluators (or evaluation assistants), and the readiness of participants. Since the YWCA had recent experience being trained in and conducting a traditional evaluation exercise, there was a direct basis for comparison that is used to highlight the generalized categories of concern.

Time. While it would seem that participatory evaluation may take more time, when compared with the time needed to administer individual questionnaires or interviews to a significant number of participants, initial appearances shift. For example, teams of two to three YWCA field staff spent an average of twenty to twenty-four hours with between nine and sixteen women collecting information about the group, its members, and its activities. A more traditional impact evaluation was carried out by two field staff and required contact with roughly five to six members of the women's groups on an individual basis, representing roughly fifteen to eighteen hours (one to one and a half hours initial interview plus one to one and a half hours impact interview). While many other issues exist, actual time and information coverage per participant were more effective in the case of participatory evaluation.

Quality of Information. Questions of methodology and validity come into play when considering which type of evaluation may be more effective. Basically, participatory evaluation offers the opportunity for participants to generate, collect, and analyze data as a group. By handing control for questioning and data collection to the participants and group, some of the information on individual members may be lost, but other rich and equally important information is invariably found. The quantitative data so highly prized in traditional evaluation methodologies may diminish slightly in the beginning, but the qualitative stories that emerge offer striking images that cannot be found in the numeric summaries resulting from structured questionnaires and interviews. It is the blending of the qualitative images with the quantitative data through participatory evaluation strategies that lends credibility to the data collected.

Given serious time constraints, the YWCA participatory evaluation teams were never quite able to collect and record a large amount of information on individual member status. They were, however, able to gather a large amount of group information as well as individual vignettes that were valuable and unanticipated. Following the field activity, YWCA evaluation team members were given a copy of the impact assessment questionnaire that had recently been developed for use with women's groups. They found that they could easily complete 80 to 85 percent of the questionnaire, missing only certain financial details that are best gleaned from records. (Visitors who heard the women's one- to two-hour presentations on their evaluations of their groups were able to complete 30 to 50 percent of this questionnaire.) Venturing well beyond the confines of this questionnaire, the participatory evaluation process generated information on sociocultural benefits that were prized by the women and in keeping with the YWCA mission and goals as well as those of empowerment and development.

Training Required. At a minimum, traditional evaluation mechanisms require training in interview techniques and occasionally questionnaire design and/or administration. Training in facilitation of participatory evaluation is undoubtedly far more involved. Participatory evaluators must develop and maintain a deep understanding of what evaluation is in general and what participatory evaluation is in particular. With this understanding, they must be immersed in the process and allowed adequate time to reflect upon their experience.

The YWCA found that in the region where all field staff and branch staff had participated, participatory evaluation activities had been started with other women's groups. Where only the branch secretary had been trained, no such activities had been carried out. This was not surprising; it merely highlighted the fact that participatory evaluation cannot be explained but must be experienced if it is to be understood. During the training workshops, it took a role play/simulation, two pre-field simulations of the implementation experience, and actually carrying out a participatory evaluation activity in the field before participants truly had an understanding of and feel for the process of facilitating participatory evaluation. Once in place, these skills and abilities translated nicely to other locales and remained in tact over the course of the intervening year.

Readiness of Participants. The ability of a group to take responsibility for self-evaluation requires a certain level of collective maturity. Conducting an evaluation and implementing a needs assessment and project planning evaluation activity can be done by a group as soon as it can be identified as such. To conduct an effective impact evaluation, the group must have advanced beyond initial stages of group development, as outlined in the model below.

Levels of Development for Women's Groups*

* This framework was developed by Tototo Home Industries, Mombasa, Kenya, and was utilized and adapted by the YWCA of Kenya over years of fieldwork with women's groups throughout Kenya.

This framework provided a mechanism for observing group progress and identifying necessary preconditions for participation. The results of the participatory evaluation exercises in relation to group development are shown in Table 10.1.

Although groups that are just barely formed (as low as level two—possibly one) can master the skills necessary to lay the groundwork for a project, conducting a participatory impact evaluation requires that the group have cohesion and some level of collective experience and history (minimum level four). Further, it was found that the participatory evaluation process directly assisted the Case 1 group in progressing quickly from level two to level five in the group development framework.*

* Case 3 regressed from a level three/four to a level two/three due to a series of external interventions, including the substantial and uncoordinated provision of external funds from several donor agencies.

Building in Evaluation Capacity—Transfer of Responsibility and Control

As one participant put it when defining and explaining what she understood about participatory evaluation: "With some training and in the long run, the process [of participatory evaluation] is supposed to become in-built so that the members are able to carry out the evaluation on their own even without help from an outsider."

This is certainly a goal of participatory evaluation: the sharing of skills and the establishment of a capacity for self-evaluation. By identifying this as both goal and priority, the YWCA is looking at slowly establishing among its women's groups the ability to periodically undertake self-directed impact evaluations. In the coastal region, there are now at least three groups that can tell visitors how to carry out participatory evaluation and describe changes in their group and the impact that participation in their women's group has had on their lives, their families, and their communities. The power of the presentations by the women and their ability to evaluate their projects and their personal and group projects were inspiring. They offered a view of an alternative structural and organizational future worth striving for.

The long-term advantages, power, and potential of participatory evaluation make it an important strategy to consider. It has the ability to provide a rich data source that grows from the women's voices and leaves them with the skills to evaluate their own projects. While traditional evaluation often removes information and leaves little of use behind, participatory evaluation gives voice to those most immediately affected by a project or program. The more voices we are able to bring into this choir, the louder and clearer will be the message they are able to share, and the better the chance that we will be able to hear it.

Document(s) 12 of 18