Document(s) 11 of 18

Participatory evaluation is typically thought of as an overall attribute that any evaluation has to a greater or lesser extent. In reflecting on the concept of participation, we are inclined to believe that such a unidimensional characterization does not capture the complexity of many evaluations that have varied levels of participation by a wide array of stakeholders. Indeed, we feel that all evaluations have participation to some extent, and that the important thing is to analyze participation on each evaluation component. That is, the concept of participation is best understood in relationship to the various stakeholder groups in any evaluation, combined with a consideration of suitable levels of participation of these different stakeholders with respect to particular evaluation issues. In some evaluations there is equal participation from funders and beneficiaries; in other cases participation is delegated or unequal. When the purpose of evaluation is to extend the effects of the development project, equal participation has advantages, but when there is a focus on compliance, for example, unequal participation may better serve evaluation requirements. These differences suggest the need to define which forms and levels of participation are advantageous for particular issues and which are not appropriate. Furthermore, differential degrees of participation are consistent with aspects of contemporary evaluation thinking that emphasize the importance of stakeholder perspectives as they relate to the various evaluation questions (Guba and Lincoln 1989).

This chapter describes and analyzes our experience with a complex Canadian International Development Agency (CIDA) development project involving a dozen research and training centers in Southeast Asia and as many partner Canadian universities and colleges. It describes the evaluation and its methodology and concludes with a discussion of the implications of differential levels of involvement of different stakeholders in evaluation science.

The Southeast Asia Ministers of Education Organization

The Southeast Asia Ministers of Education Organization (SEAMEO) was founded in November 1965, for the purpose of promoting regional cooperation in education, science, and culture in its member countries. According to its charter, SEAMEO carries out this mandate in order to further respect for justice and the rule of law and for the human rights and fundamental freedoms that are birthrights of the peoples of the world. The organization has translated this vision of the benefits of peace, prosperity, and security through enlightened citizens into cooperative efforts in education, science, and culture.

Over the past thirty years, SEAMEO has focused on human resource development in the region through provision of short-term and long-term training courses, seminars, workshops, international and regional conferences, and information dissemination activities. Such programs are implemented through SEAMEO's twelve regional centers, which are located in various participating countries. There is a diverse range of specialization among the centers, from tropical biology and tropical medicine to educational innovation and technology, from science and mathematics to archaeology and the fine arts (see Table 9.1).

The SEAMEO-Canada Program of Cooperation in Human Resource Development

Since 1970, Canada has been involved in SEAMEO as one of five associate members, contributing financial and operational support that has averaged $1.5 million per year since 1985, when Phase I of the SEAMEO-Canada program of cooperation began. That year, the SEAMEO Pilot Project for Integrated Community-based Human Resource Development was launched under the financial assistance of CIDA. The five-year project (1985–1989) provided $9.5 million for regular program support and institutional cooperation.

Phase II of the program (1990–1995) provided a Canadian financial contribution of $7.1 million to SEAMEO in support of its regular training programs and to strengthen the institutional capacity of the SEAMEO centers and the organization's Secretariat (SEAMES). Project funds provided three main types of inputs:

• Financing for the regular training programs and activities of SEAMEO centers, thereby providing approximately 1,000 short- and long-term training opportunities;

• Institutional linkages that supplied technical assistance, training (in Canada and at the centers), equipment, and materials support in order to strengthen the institutional capacities of SEAMEO centers; and

• Canadian technical assistance to SEAMES to help develop its strategic management capacities.

The funding to support SEAMEO regular programs was managed directly by SEAMES; however, SEAMES subcontracted certain lead Canadian universities and colleges to manage and administer institutional strengthening projects at SEAMEO centers.

The Evaluation

The evaluation discussed in this case (Universalia 1995) was conducted in the fifth, and final, year of the project and was intended to provide information to SEAMES, CIDA, its Project Advisory Committee (PAC), and the involved SEAMEO centers, the main general project stakeholders. In conjunction with its ongoing strategic planning efforts, SEAMES continually undertakes analyses of its work, so the evaluation was designed to provide information that could guide future SEAMEO capacity development. The design also had to account for the fact that the evaluation was being undertaken, in part, to fulfill the conditions of the agreement between CIDA and SEAMES and to provide accountability for CIDA's investment in the project. Hence, the challenge for the evaluators was to successfully mix the agendas of project understanding and compliance using a participatory methodology. The major evaluation questions and issues can be summarized as follows:

Methodology

The approach involved various stakeholder groups in different levels of participation under the overall guidance and direction of the external evaluators from Universalia. Participatory evaluation methodologies were viewed as a learning process in which evaluation stakeholders gained a better perspective of the project and its relationship to the organization. We defined the participatory component of this evaluation as a process in which the various stakeholders worked with us, the independent evaluators, and contributed to defining the evaluation mandate, as well as participating in the data collection and analysis. The responsibility for producing the actual report, its conclusions,

and recommendations rested with the independent evaluators, with input from all stakeholders. The major stakeholders and a summary of their roles are shown in Table 9.2. All of the stakeholders took part in the process by providing data, and each of the stakeholders participated in one or more of the other evaluation activities: developing of the terms of reference, making

revisions to the work plan, developing data-collection instruments, collecting data, analyzing data, and discussing and suggesting revisions to the evaluation report. Table 9.3 presents the major sources of data for the evaluation, with the instruments used and the response rates obtained.

The Process

Planning

Although Universalia had a primary role in the preparation of the evaluation work plan, there was participation from many of the stakeholders in both implicit and explicit ways.

CIDA, through its standards, procedures, and manuals, provided the policy guidelines for the terms of reference and work plan. CIDA also reviewed both documents to ensure that the donor's needs and interests would be met.

In planning the participation of the various other stakeholders in the process, their relationship to the major evaluation questions had to be given careful consideration. Proactive participation of the project stakeholders was primarily related to project effects and impacts, designed to help the various stakeholders increase their mutual understanding of the project and its implications for the future, as well as to refine and develop center strategies. However, questions relating to project efficiency and effectiveness were designed to generate data to be used primarily to assess project compliance and to enable CIDA to make decisions on further funding. For these reasons, participation of the other stakeholders for these issues was limited to providing data and assisting with interpretation rather than with making judgments.

Development of Data-Collection Instruments

A number of instruments were developed for this evaluation, namely, questionnaires and interview protocols. They are listed in Table 9.3. However, two of the instruments deserve particular attention, as they required significant participation from some of the stakeholders at one or more level: the Center Self-Assessment Guide and the Regular Program Graduate Questionnaire.

The Center Self-Assessment Guide was developed to assist SEAMEO centers in understanding their center, its context, future directions, and needs. Universalia developed this tool using a framework we developed with the International Development Research Centre (IDRC) for institutional assessment (Lusthaus, Anderson, and Murphy 1995). This framework focuses on four dimensions: the environmental context of the institution, institutional motivation, capacity, and performance. The self-assessment tool developed by Universalia for this evaluation was a fifty-page guide that provided a procedure and content to assist centers in conducting a thorough self-assessment of their capacities and performance, and a conceptual framework for centers to help refine and develop. A draft of the guide was discussed with selected center directors and then modified to be more usable in the SEAMEO context.

Over the last few years, centers have been conducting follow-up surveys of their regular training programs. However, there was no coordination between the centers on these efforts, making aggregate data analysis impossible. As well, response rates to these instruments were very low (seldom over 20 percent), so the data collected had minimal usefulness. The centers all expressed an interest in developing a more effective, standard way of collecting data. The evaluation presented an opportune time to do so, as Universalia's expertise in instrument-building and data-collection techniques complemented the centers' knowledge of their information needs and understanding of the population. Universalia reviewed all of the existing instruments the centers had been using in the past and produced a draft questionnaire, using the latest research on evaluating training programs (see Kirkpatrick 1994). Universalia met with the center directors in small groups and worked together to revise the draft so that it was appropriate in terms of content, language, and culture. This cooper­ation resulted in better questions and allowed the centers to gain ownership of the instrument, which will help ensure its future use.

The process of participation in building the instruments was a lengthy but essential element of participation that built understanding of the participants' diverse perspectives. The use of words was troubling, as many phrases and ways of asking for information that were suitable in Canada were confusing to those from different backgrounds. Canadians are often concerned with counts and quantification, which may be inappropriate in other parts of the world. Even within Southeast Asia there are wide differences: Singapore and the Philippines use English in daily life, while other countries do not; people from Thailand are reluctant to express critical views openly; most Southeast Asians are concerned with people being able to save face. The challenge is to build data-collection instruments that respect diversity while not making them so general that the data generated loses its meaning.

Data Collection

As mentioned earlier, all of the stakeholders provided data for the evaluation. SEAMES/SEAMEO, the centers, the Canadian institutions, and the graduates from the regular training programs provided data that pertained to project efficiency (compliance) issues; the centers, Canadian institutions, and regular training program graduates also provided data that related to project effects and impacts; and data for project effectiveness issues came from CIDA, SEAMES/SEAMEO, the centers, the Canadian institutions, and the regular training program graduates.

In addition, the centers participated in proactive data collection. It was the centers' responsibility to send out and collect the Regular Program Graduate Questionnaires. One thousand questionnaires were distributed to individuals who had participated in one or more training programs between 1991 and 1993. Fifty-three percent of the questionnaires were completed and returned to the centers, which forwarded them to Universalia for preliminary analysis. This response rate surpassed, by far, previous data-collection attempts by the centers. Their involvement in the design and distribution of the instrument was a contributing factor to this success, because it ensured that the questionnaire was relevant and culturally and linguistically appropriate for the trainees; the participants responded well to something from an institution that was already familiar to them. Once again, we feel that this success, experienced firsthand by the centers, gave them ownership of the process and has enhanced the chances of continuation of this data collection by individual centers.

Completion of the center self-assessments involved major data-collection and analysis activities within each center. Each center mobilized an assessment team, ranging in size from three to thirteen staff members (three-quarters of the teams were composed of more than six members). Team members reviewed key documents, talked to people involved at different levels in the center, observed facilities and activities, and observed interactions between people in different contexts (classroom, meetings, and so forth). They discussed their observations and had to develop a consensus on the different issues presented in the self-assessment guide. The centers then sent the completed guide to Universalia. It was a complex process in which centers participated with varying levels of effort, and this variance was reflected in the depth and quality of data collected. Those centers that put significant effort into the self-assessment indicated to us that they had learned a lot about their organization in terms of future directions and needs. At least one of the centers plans to make this a regular activity.

Data Analysis

Data analysis occurred at a number of levels, with varying degrees of participation with the stakeholders. The completed Regular Program Graduate Questionnaires were forwarded from the centers to Universalia for preliminary analysis because Universalia has in-house expertise in qualitative and quantitative analysis. Universalia conducted statistical and content analyses, and partway through the evaluation presented the data to the center directors at their annual meeting. The evaluation was planned to coincide with this meeting because it was the major midpoint at which stakeholders could contribute to the analysis. At the same time, Universalia presented the data from the Center Staff Training and Canadian Technical Assistant Questionnaires. The group analyzed and interpreted the data together to build a collective understanding of what it meant. It was an interesting experience for all involved, including Universalia, as it clearly demonstrated the role culture plays in judgments. For example, how much "agreement" with a Likert-scale item is enough? Does the required level vary from country to country? How are differences interpreted? A puzzling finding to us was that, when asked whether they would like to return for additional training, the vast majority of the questionnaire respondents said "yes," but a further question revealed that they would prefer to do so in a different center. We found it impossible to understand why a person who valued training in a particular center preferred to return to a center that specialized in another field. We were inclined to consider the result negative, whereas the Southeast Asians did not. In participatory evaluation, the emphasis is on building shared understanding, and discussing such culturally based findings is a good means of doing so. Comparative levels of performance of different training programs also uncovered cultural differences, because Southeast Asians do not easily relate to data that make colleagues appear less good. Although comparisons between centers were made, they ended up being buried in an appendix of the final report because of such cultural concerns.

Universalia collected the completed center self-assessments, reviewed them, and prepared summaries of each one, with a short analysis of concerns and priorities. We reviewed these summaries with the center directors at their annual meeting, however, it was not done as a group activity. The summaries were treated as confidential to the evaluation team and to the respective centers. Center directors met, individually, with a Universalia team member to discuss the self-assessments. After the annual meeting, the center directors took the summaries back to their assessment teams for review. There was subsequent ongoing communication with those centers that had comments to make, to ensure that the resulting analysis was correct and acceptable to the center. It was decided at this time that the self-assessments would not be included in the evaluation synthesis report, although aggregate findings were embedded in it, thus leaving it a significant part of the evaluation owned exclusively by the participants themselves. Neither CIDA nor SEAMES received copies of the self-assessments.

Center participation in the analysis of another evaluation component was sought—the CIDA Project Analyses. Universalia prepared summaries and analyses of the effectiveness and effects of the project in building capacity within the centers through the linkages, Canadian technical assistants, and center staff training. We used data from the SEAMEO-CIDA Project Questionnaire, completed by the centers, as well as from the Center Staff Trainee and Canadian Technical Assistant Questionnaires. The analyses were shared with the center directors, confidentially, at their annual meeting. Ideally, they would also have been shared with the Canadian partners at this stage, but it did not prove feasible to do so during the summer months, so that step came much later. As with the self-assessment summaries, each center director reviewed the CIDA Project Analysis with his or her staff and sent Universalia comments for incorporation.

In most cases in which centers sent back revisions on both the self-assessments and the CIDA Project Analyses, their comments were informative and valid. The participatory nature of these two components was important because the centers' self-assessments and responses to the SEAMEO-CIDA Project Questionnaires allowed them to contribute their in-depth knowledge of both their individual centers and how the project had affected them. However, Universalia's critique and questioning of their data from a perspective that looked at the project as a whole ensured that the concerns of other stakeholders were reflected in the analyses, making them more relevant to all stakeholders of the evaluation.

Final Report

Two types of documents were produced as a result of the evaluation. Each center had produced its own self-assessment, which was supplemented by our observations and critique. These documents were confidential to the individual centers and were not presented to all of the stakeholders.

The second major written output from the evaluation was the final evaluation report. It consisted of a synthesis document and three volumes of appendices. It was presented, in its draft form, to the PAC for discussion and review. At that time we learned an important lesson about the need to circumscribe roles of the involved stakeholders. As it turned out, the PAC had invited center directors and also project heads from Canadian institutions. It was the first time the latter group had been involved in more than providing data for the evaluation. Although some of the data and findings were familiar to the PAC and center directors because of their participation throughout the evaluation, none of the separate stakeholder groups had seen the overall analysis of the external evaluator, and most felt that their perspectives had been overshadowed by those of other stakeholders. In particular, most of those who had benefited from the project considered the results overly negative and much too centered on developing capacities that participants had agreed to initially but had difficulty achieving, rather than activities that may have been successful in isolation but did not contribute to capacity development in a sustainable way.

Evaluation Findings

The final evaluation report presented a number of findings related to the questions the evaluation set out to answer.

The project had been designed essentially as a continuation of previous practice (Phase I) supported by the CIDA context at the time of its inception. However, no in-depth needs assessment was done at the beginning of Phase II, and thus the design supported some components in inappropriate ways and missed opportunities to be more strategic. In addition, the context in which this project was conceived had changed dramatically over the five years of its implementation—changes in Canadian foreign policy and development assistance, social and economic developments in Southeast Asia, and SEAMEO's transition into a truly regional organization and the related changing roles of the centers. All of these changes significantly altered the continuing soundness of the rationale of such a project.

Although the project appears to have been effective in producing its intended outputs, achieving these outputs did not ensure realization of the purpose of the project: institutional strengthening. Effectiveness of project management was also limited for similar reasons—weak project design, inefficient management processes, and a lack of initiative by major players to take leadership and adapt the project in order to make it more effective in achieving its purpose.

The evaluation found that the successes of the linkage partnerships in building capacity in the centers varied widely. The choice of partners and their degree of compatibility played a large role in determining the success of the linkage. Another factor was the large turnover in project staff, including center directors and heads at the partnering Canadian institutions (eleven of the former and six of the latter have been replaced since the beginning of the project). Additionally, weaknesses in the project design limited the effectiveness of the linkages. However, institutional needs assessments in the individual centers may have compensated for the weak design and would have more clearly identified the individual needs of each center.

Overall, the responses of the regular training program graduates were quite positive. However, there were significant differences across centers. It also became evident that, with a few exceptions, the regular training at the centers was supply-side programming and was not highly relevant to the participants' jobs (the main exception being TROPMED/Philippines, which was very demand oriented and, with help from the linkage project, brought telemedicine technology to the islands). Although the project's investments into regular program training added some sustainable quality in a few of the centers, Canadian technical assistance and funds in this area served primarily as input substitution.

Results of the Evaluation Process

We feel that this evaluation, with its participatory components, built capacity within SEAMEO in two important ways. First, the process appears to have given the stakeholders a much deeper understanding of their organization (SEAMEO as a whole) and how it could link to Canada's development interests. This was reflected largely in the center directors' discussions about a possible Phase III of the CIDA project. In these discussions, which followed the evaluation process, the concepts are more demand driven than supply oriented. The thoughts are more strategic and present a thematic cut across centers, as opposed to support for centers as individual units. A more competitive or selective process for funding is also being considered by the center directors, reflecting an understanding that funding may not be available for all centers and will not be provided to be used for input substitution.

Second, it helped to build evaluation capacity. Some of the centers plan on using the self-assessment tool (or an equivalent) on a regular basis. A few of the centers are using the information gained from this process in their strategic planning. The centers' participation in the development of the questionnaire, data collection, and analysis for the regular training component gave them ownership of the process, and the success (higher rates of return and useful data) has encouraged them to continue using the instruments for future training reaction assessments.

Implications and Conclusions

When Is Participation Legitimate?

The major premise of this chapter is that the extent and nature of participation depend on the purpose of that participation. The case study illustrates one way of conducting a participatory evaluation when the concern is both to continue the development process and to evaluate project compliance. In the approach used, the evaluation partners had considerable freedom in some demarcated areas and limited input in others. This evaluation attempted to structure participation of the stakeholders with their varying interests in and degrees of concern about the evaluation issues in mind.

Essentially, the SEAMEO evaluation had three sets of issues that intersected with the legitimate concerns of the stakeholders—compliance, self-analysis, and understanding the effects on others. Compliance, in our view, is primarily the concern of the donor and requires an external and objective review in order to ensure accountability as demanded by donor policies and realities. This concern focuses on how well the project has been implemented and its effectiveness and effects as a way of enabling the donor to judge whether it has been a good investment from the donor's perspective. Thus, in donor-assisted projects, the benefits of involving beneficiaries in the generation of knowledge through participatory evaluation need to be balanced with the donor's separate need for accountability. This may be a Western paradigm, but it is a real one. In a context of severe fiscal constraint, donor governments and their publics are often skeptical of investments in distant projects, and self-reports may be an insufficient basis on which to continue project funding. The public demands accountability, and only an independent evaluation suffices. In contrast, if the goal is to engage beneficiaries in sustainable development, then full participation may be a better route.

The case incorporated what was actually an embedded evaluation: the center self-analyses. These complemented the ongoing strategic planning activities of the centers and were clearly an area where the centers owned the process and the data. They learned from the exercise, and they were able to decide whether or not it was in the center's interest to share it. The approach we used was similar to that incorporated in an impact evaluation of CIDA's project to construct a Natural Resources College in Malawi (Anderson 1989). In that evaluation project, a dozen senior faculty in the college worked with us to improve their knowledge of and skills in research methods and then applied this to the collection and analysis of data from college alumni employed throughout the country. The college development project had been completed several years before, so there was no need to revisit issues of compliance. The focus in that evaluation was on finding out the impact of the college graduates in rural development in the period since graduation, and it was useful for researchers from the North and from the South to help one another in understanding the answer to this complex question. Although all involved evaluators were keen to find out what had happened, few of the college personnel had been involved in the construction phase, so their concern with impact did not conflict with their role in execution of the project. Not so with the SEAMEO project, which stressed compliance as well as understanding. The answer to the evaluation questions in the SEAMEO case reflected directly on the performance of those who had been involved in compliance, so there were some aspects of the SEAMEO evaluation that were not considered part of the evaluation mandate of those from the involved centers and Canadian institutions. In both the Malawi and SEAMEO cases, we worked with project ben­eficiaries to develop a shared and mutually understood methodology that the local participants then implemented. They, not we, were the custodians of knowledge, and we benefited only to the extent that they were willing to share.

The third aspect, understanding the effects of the general training programs, was a legitimate concern of both SEAMEO and the donor, but from differing perspectives. The survey study of training graduates was an aspect in which the centers stood to benefit from knowing how their training was being received. The donor also had a legitimate concern, because the project had invested in this component. However, the center's concern was essentially formative, while the donor's was summative—the centers wanted to know how to improve the training; the donor wanted to know whether the training that had been delivered had made any development difference. The findings were positive from SEAMEO's perspective but negative from CIDA's. SEAMEO needed cash to continue conducting general training activities. This is an understandable need, and the CIDA project had succeeded in providing financial resources to this end in an efficient way. Efficiency, however, is but one concern. In this case, CIDA's investment did not further CIDA's interest in building sustainable local capacity for relevant training. Indeed, as the external evaluators concluded, support of the regular training component may have had overall negative effects because it postponed a strategic decision about the nature of this training that SEAMEO would have to make once the flow of donor funding was reduced. The perspective of the involved Southeast Asians was that this component served their needs and should be continued; however, the donor was concerned with developing sustainable capacities, not with providing money as a substitution for local inputs. In other words, the beneficiaries wanted resources with few conditions, while the donor wanted accountability for its agenda of sustainable development. Under such circumstances, the investor's perspective dominates, so it is the investor that needs to make the judgment on the value of continuing such a project component.

It is fundamental to participatory evaluation that those involved in the project or organization being evaluated are recognized as the "key custodians of knowledge" (Freedman 1994, 3) about the endeavor being evaluated. There is little doubt that the perspectives of such participants are valid for assessing effects and impact on them and their organizations, but there is considerable doubt that these stakeholders are able to pass judgment on many aspects of compliance. Any group, with the help of suitable data, is capable of assessing whether deliverables were in fact delivered and of judging whether they were provided in a timely way, but it is the investor, not the beneficiaries, that can say whether the investment met the investor's needs. Thus, the answer to questions about the development investment are fundamentally grounded in the purpose of the investment and how different stakeholders value different outcomes.

How Should Participation Be Shared?

Equal leadership is inherent in equal participation, but when there are differing areas of legitimate concern, participation is inherently unequal. This approach is not shared by everyone, as it reflects Rigg's (1991) contention that attitudes toward participation mirror the concerns participants have about top-down versus bottom-up management. Our main conclusion is that evaluators need to ensure that the various roles and responsibilities are well understood. Freedman (1994) notes the advantage of using existing groups in participatory evaluation. In this case, there were several different groups involved. However, the level of collective learning experience within each group varied widely, and those who had more experience gained more from, and contributed more significantly to, the evaluation. The PAC meets only once a year as a group and therefore had a limited role. The Canadian institutions have never met collectively, and it is for this reason that their participation was limited to providing data. Within each center, systems were already in place for group working and learning: The internal Strategic Planning Groups built upon their existing procedures by using the Universalia/IDRC framework on organizational capacity and performance to conduct a self-analysis of their centers. At the next level, the center directors worked collectively to link the results of individual self-assessments to an analysis of the whole organization and the relationship of the CIDA project to it. The center directors often meet as a group and are becoming a strong unit. Their collective involvement in instrument development and data analysis was very useful to the evaluation and increased their learning in the process.

One of the lessons we learned was the advisability in future work of this type of clarifying the roles of and demarcation between the groups. If each group of participatory evaluators knows what it is responsible for and how its work relates to that of other groups, then the results may come together more easily. In this case, we encountered difficulty when some groups tried to expand their roles to exert their influence at other levels. There were several noteworthy instances. First, we initially thought of including the center self-assessments as part of the overall evaluation report. However, the center directors made us realize that the self-assessments would be more candid and have more impact on the concerned centers if they were confidential to each center. We changed the initial plan and kept our feedback and the final self-assessment reports confidential between us and the center directors, and, in at least a couple of instances, these self-assessments had considerable impact on individual centers. Another example involved the PAC, which is a small group of individuals who have an arm's-length relationship to the centers and the Canadian universities and colleges. When the draft report was presented, however, all the center directors and several Canadian university project heads were present. While these guests were in fact observers, they raised questions about why they had not been consulted in advance on the overall conclusions. The overall conclusions were, of course, the legitimate concern of only the smaller PAC.

The notion of creating a common body of knowledge (Freedman 1994, 57) is a good one, but it is not necessary that every group of participants be involved in collecting all the information or even that people endorse the perspectives of the other groups. Once the information is shared, at least people can challenge it and, with supportive group processes, can use it to build their understanding of how others act on the common knowledge.

Goulet (1989) refers to participation being seen either as a goal or as a means. If it is intended to do nothing more than help in the analysis, it may represent yet another example of the dominant group exploiting the intended beneficiaries. If it is viewed as a legitimate extension of the development project, then it may be an excellent way to empower beneficiaries in a sustainable way.

Do the Costs Justify the Benefits of Participatory Evaluation?

The costs of participatory evaluation are structured differently from those in expert models. One difference is the requirement for a substantial investment in planning, training, and coaching. It allocates resources to the means of evaluation rather than applying resources directly to the tangible outcomes. While this may have greater long-term impact, it is clearly an investment in the future and needs to be understood as such, including the necessity for a longer period of time for the evaluation process to take place. Another difference is the cost of the time demanded of participants. It takes more of their time because they are involved, and also because they are typically involved in group processes that are inherently time-consuming. The dollar value of an investment in participatory evaluation may not appear any greater than that in expert evaluation, but the overall costs to society when people's time is included are undoubtedly much more.

The benefits are in building participants' capacities to understand the development efforts in which they are involved. Because capacity development is a lengthy and incremental process, the growth in human capacity is difficult to evaluate, but a growth in capacity is essential if beneficiaries are to assume control over their own destinies. Perhaps the solution is in viewing participatory evaluation as a legitimate part of the development project rather than as a separate component not generally charged to programming. If viewed in this way, then it could begin earlier, could legitimately demand participant time, and could itself be judged for its development impact rather than for its contribution to donor knowledge.

References

Anderson, G., ed. 1989. "Natural Resources College of Malawi: Impact Evaluation." Report on the Participatory Component of the NRC Evaluation. NRC, Lilongwe, and CIDA, Hull.

Freedman, J. 1994. "Participatory Evaluations: Making Projects Work." Dialogue on Development, Technical Paper No. TP94/2. International Centre, University of Calgary, Calgary.

Goulet, D. 1989. Participation in Development: New Avenues. Oxford: Pergamon Press.

Guba, E. G., and Y. S. Lincoln. 1989. Fourth Generation Evaluation. Newbury Park, Calif.: Sage Publications.

Kirkpatrick, D. L. 1994. Evaluating Training Programs: The Four Levels. San Francisco: Berrett-Koehler Publishers.

Lusthaus, C., G. Anderson, and E. Murphy. 1995. Institutional Evaluation: A Framework for Building Organizational Capacity for IDRC's Research Partners. Ottawa: IDRC.

Rigg, J. 1991. "Grass-roots Development in Thailand: A Lost Cause?" World Development 19 (2/3): 199–211.

Universalia. 1995. "Evaluation of SEAMEO-Canada Programme of Cooperation in Human Resource Development, Phase II." Universalia Management Group, Montreal.

Document(s) 11 of 18