Document(s) 10 of 18

This case study discusses an alternative evaluation methodology known as process evaluation and its application to the Nepal Health Development Project (HDP). It describes the HDP, provide details of the evaluation methodology and underlying concepts, and recount the various stages of implementation. The latter part of the chapter summarizes major findings and discusses lessons learned and benefits and costs of the methodology.

Brief Project History

The HDP is a participatory health development project of the University of Calgary's Division of International Development, the Institute of Medicine (IOM) at Tribhuvan University in Nepal, and the Ministry of Health. The first seven-year phase of the project ended in March 1995, and a second phase is under way. Project funding provided by the Canadian International Development Agency (CIDA) for the first phase was Cdn. $4.6 million.

Three evaluations of the HDP were conducted, all of them at the initiative of the HDP partnership. The 1989 and 1991 evaluations used a combination of conventional and participatory methodologies. The HDP subsequently developed the process evaluation methodology for tracking human resource development (capacity-building) initiatives and their outcomes. This methodology was implemented in the final year of the first phase of the project.

The Nepal Health Development Project

Project Setting

Nepal is a country of 20 million people, bordered on the north by Tibet and on the west, south, and east by India. It comprises three distinct topographical zones: the Himalayas in the north, the foothills in the center, and the Gangetic plains, or Terai, along the southern margin.

To a large extent, Nepal was isolated from the rest of the world until 1954, when the traditionalist monarchy was forced by internal and external pressures to initiate a multiparty democracy. This pluralist system was quickly replaced by a one-party, palace-controlled government that lasted three decades. However, the process of opening the country to outside influences continued. In 1991, after a brief popular uprising, there was a return to a multiparty parliamentary system. There has been a continuing commitment since then to pursue broad-based development goals through democratization and decentralization.

Administratively, the country is divided into seventy-five districts. Each district has been divided into municipalities and clusters of villages called Village Development Committees (VDCs). Altogether, there are thirty-six municipalities and 3,995 VDCs.

The Ministry of Health is responsible for providing curative and preventative services through a network of hospitals and remote rural health posts. Primary objectives of the government's national health policy are to upgrade the health status of the majority in rural areas by extending basic primary health services to the village level, and to provide accessible and effective referral services.

Project Description

The HDP developed in response to what the project partners perceived as a gap between the Ministry of Health's stated intentions to raise the level of health in rural areas and its actual performance. Specifically, the gaps were perceived to exist on three levels:

• Between the ministry's stated programs and the ability of regional and district-level managers to implement these programs;

• Between the expectations of the regional and district managers and the performance of extension workers in the field; and

• Between actions of the extension workers throughout the district and the services required by community members.

Thus, the stated purpose of the HDP was:

To strengthen the capacity of the government's health-related institutions and rural communities in Surkhet District to meet health needs through community-based participatory development, management strategies, and the training of generalist physicians.

At the peak of the project, there were twenty-seven full-time and eight part-time staff in VDCs, Surkhet District, and in Kathmandu. Four equivalent staff positions were held by Canadians acting as counterparts to Nepalis in the following roles: project coordination, community health and development, district management, physician training, and documentation and research. The Canadian coordinator and documentation and research officer worked part-time from the project's Calgary office.

Three Streams of Project Activity

The evaluators used the process evaluation methodology to focus on the capacity-building experience resulting from the three streams of project activity: community development, district health strengthening, and generalist physician training.

Community Development Stream

HDP was active in five VDCs in Surkhet District. These VDCs, with a combined population of 35,000, are remote agricultural communities located in the foothills. Project community development staff and local facilitators were trained in participatory research and participatory appraisal techniques, which they utilized in their work with village groups.* These processes were designed and utilized to empower villagers, independently and as a group, to address community issues.

In all the communities, villagers have organized themselves at the ward level according to interest, such as women's health or forestry groups. These neighborhood groups meet monthly to address local issues. From time to time, representatives meet to share, exchange, and plan at the village (VDC) level. Initiatives arising from this community development process include irrigation and clean water schemes, forest conservation, vented stove construction, women's literacy, microenterprise development, and savings and credit schemes.

As the project "worked itself out" of a VDC, it assisted ward groups to relate to one another across VDCs so as to form a local "people's" organization. These self-help nongovernmental organizations (NGOs) are able to establish cooperatives, access external funds, and organize collaborative village development schemes. They can also better advocate for community interests with government agencies such as the Ministry of Health.

* Participatory appraisal has its roots in participatory action research (PAR) and rapid rural appraisal (RRA). It features an interdisciplinary group assessment process in a style that uses multiple techniques for data acquisition and analysis. It is people oriented and locale specific. It pursues an increasingly accurate understanding through rapid rounds of field interaction. Participatory appraisal, in particular, places the "subjects" of research in the center of the design and implementation of the research process. It taps local knowledge and combines it with modern scientific expertise. And it provides those in positions to make changes with useful information as a guide to action.

District Health Strengthening Stream

The second stream of activity addresses the delivery of health programs at the district level. The focal point of this stream of activities is the Ministry of Health's district public health office, which manages preventative, promotional, and curative health services through a network of health posts, sub-health posts, and a twenty-bed district hospital.

The project's aim is to strengthen the ministry's capacity to operate in a decentralizing bureaucracy. HDP staff assisted ministry staff to develop information-gathering systems and methods for planning and managing district health activities, and to improve the functioning of the outlying health posts, the district hospital, and the referral system that links the two. Activities within Surkhet District included needs assessment of staff, in-service training, and the development of community-managed health post drug schemes. In addition, project staff collaborated in the training of female health post auxiliaries, traditional birth attendants, and community volunteers. At the national level, the project contributed to health policy and planning through the development of policy and program alternatives and participation on national task forces related to district health services.

Training of Generalist Physicians

From the IOM's main campus in Kathmandu, the project, along with the newly developed Faculty of General Practice, coordinates a three-year postgraduate general practitioner training program. Most districts outside Kathmandu have poorly equipped hospitals and medical doctors without the requisite skills to perform emergency and obstetric surgery. The objective of the program is to place specially prepared generalist physicians with appropriate clinical and managerial skills into district hospitals. Consistent with the institutional capacity-building goal of the HDP, most of these residents come to the program from within the Ministry of Health and, upon completion, return to a government district hospital.

Focus of the Evaluation

The process evaluation examined the extent and the process by which the HDP achieved its purpose of capacity building. The evaluators were interested in seeing how the capacity-building efforts of the project had assisted in meeting needs and improving performance in the community and the health system in Surkhet.

Process Evaluation Methodology

The process evaluation methodology has four key elements:

• Use of a conceptual model around which to examine capacity building;

• Reliance on participatory strategies;

• Adoption of participatory appraisal techniques; and

• A qualitative approach to indicator development and field investigation.

No one element is new to the world of evaluation, yet combined, the HDP's recent experience suggests that these elements offer an accessible, action-oriented assessment tool for human resource development projects.

Use of a Conceptual Model

Human resource development or capacity-building projects like the HDP emphasize the process of matching beneficiary needs and competencies with financial resources, staffing, equipment, supplies, and time, and then transforming this collection of inputs into plans and activities that build human and organizational potential.

Those responsible for implementing project activities are keen to know how activities generate knowledge, attitudes, and skills, and how the learning in turn influences others who are not directly involved. Further, they want to know whether learning actually changes the way things are done in an organization or community, and whether these changes are sustainable.

Often the environment around the project has much to do with sustainability. Implementers are thus also interested in understanding social, economic, political, administrative, cultural, and other cross-currents that enable or impede capacity building.

A conceptual model addressing these issues is used to link the evaluator to the theory of development underlying the way the project was designed. The model helps keep the evaluation focused on what the project or activity is trying to achieve. In human resource development projects such as the HDP, the conceptual model provides a framework to clearly reflect the intent of capacity-building initiatives. In so doing, it enables the evaluator to assess progress in the process of building human or organizational capacity. Using the model to analyze the findings of the evaluation can, in turn, allow project managers to build on their understanding of development and make better decisions.

The "spiral" model of capacity building, developed by the HDP for this evaluation and described later in this chapter, assumes that behind every new latrine, weaving loom, or irrigation canal in a village, for example, there are less visible but equally important changes in individual and group knowledge, skills, and attitudes. Similarly, it assumes that behind every improvement in the design and delivery of health services provided by the local health post or district hospital, there are changes in the way health personnel view their roles, those of their colleagues, and the needs of their consumers. Indeed, the model assumes that even where there are no visible improvements to look behind, there may be important changes taking place in the capacity of people and organizations to improve the quality of life.

Reliance on Participatory Strategies

Participation is a cornerstone of effective process evaluation. The people who most need to know how well or poorly project activities are building capacity are those who carry them out. These are the people in charge of the construction of latrines or irrigation ditches at the village level, the people responsible for health post staff supervision and training at the district level, or the people responsible for the management of the project as a whole.

In designing the process evaluation methodology for the HDP, its originators recognized the wide range of stakeholders within the project. Among the groups and individuals directly related to the HDP, there are two types of stakeholders. There are those people whose lives and work are directly affected by the activities of the HDP. There are also those people implementing or supporting the project who stand to learn from the evaluation results. In order to be an effective guide to these stakeholders, the designers realized that process evaluation had to be relevant to each of their varying information needs.

Therefore, to be participatory and relevant, representatives from stakeholder groups had to be genuine participants in the design, data gathering, analysis, and reporting phases of the exercise. Where this was possible, the stakeholders would have a greater sense of ownership of and accountability for the evaluation. As a result, they would be more likely to respond to the research findings and recommendations.

This evaluation used a core team of four evaluators, three of whom were external to the project. This was an interdisciplinary team. Combined, it gathered expertise in evaluation, community health and medicine, cultural anthropology,

Figure 8.1: Composition of the HDP Process Evaluation Team

health economics, social policy, and community development. The core team was joined by counterpart evaluators from the Surkhet office of the Ministry of Health, the district hospital, and the three VDCs participating in the exercise (six villagers per VDC). This joint team was supported by HDP staff and representatives of the major stakeholder organizations: the IOM, the University of Calgary, and the Central Office of the Ministry of Health. The composition of the process evaluation team is illustrated in Figure 8.1.

Staff supplied insights into the operations of the project. Villagers and Ministry of Health officials provided an understanding about the context of the project. The core evaluators contributed their own disciplinary perspectives. Given their relative distance from the day-to-day project routine, they asked probing questions and brought a broader perspective to the research. They also contributed a technical understanding of evaluation.

Adoption of Participatory Appraisal Techniques

Participatory appraisal provides a toolbox of techniques to help interdisciplinary teams function effectively and efficiently. These techniques—semi-structured interviews, focus groups, social/community mapping, accidental interviews, group treks, and many others—help evaluators talk with and listen to local people and other team members, observe local conditions, and study preexisting information.

While guided on a daily basis by the conceptual model and the parameters of the project/activity's design, the team was free to choose the information-gathering instrument and angle of inquiry that made sense at the time. Sometimes, these choices were made ahead of time in planning sessions; sometimes, they were not.

Team members primarily worked together so that each individual could exchange her or his interpretation of the same observation. Daily debriefing was essential to order and synthesize the information that rapidly accrued. At these sessions, the benefits of interdisciplinary team research became clear. Members contributed their various perceptions, often complementing each other's insights to build a better understanding. Sometimes, when individual perceptions clashed, the team decided whether more information was needed on the same topic, and if so, it planned the agenda and use of appraisal techniques accordingly.

Qualitative Approach

A quantitative approach is necessary but not sufficient to evaluate human and organizational capacity building. For example, quantitative information does not convey changes in attitudes and behaviors, nor does it address the question of sustainability, all of which are intrinsic to the goal of capacity-building projects. While visible outputs—the latrines, literacy students, trained physicians, and so forth—can and should be counted as indicators of progress, such information must be balanced with qualitative information within a qualitative conceptual framework.

The qualitative framework, in this case the conceptual model of capacity building, embraces the full life cycle of a project or activity—from inputs to impacts. The model guides evaluators in identifying key questions and in seeking out, testing, and verifying indicators of capacity building for each stage in the life cycle. The model, key questions, and indicators lend themselves to a qualitative approach to data gathering and indicator development.

The Spiral Model of Capacity Building

The Spiral Concept

The conceptual model for this process evaluation was based on the assumption that new knowledge, skills, and attitudes influence ever larger circles of people within an organization, institution, or community. Understanding the capacity-building process is essential, as it represents the means by which the HDP achieves its purpose—a closer fit between consumer need and health service delivery.

Figure 8.2: Spiral Model of Capacity Building

This concept is represented schematically in Figure 8.2. The figure shows a spiral in a box. The spiral is narrow at the bottom and becomes wider as it winds upward. At the bottom of the schematic is the initial exposure to problems and ideas. As the ideas are discussed, they generate enough support to be transformed into a plan of action. Contained in this plan are one or more activities. The activities of a capacity-building process may bring together groups of people who can effect the desired changes with those organizing the activity. Once in contact, existing knowledge, skills, and attitudes are sharpened and new knowledge, skills, and attitudes are acquired.

From this point on, changes in knowledge, skills, and attitudes begin to affect ever-widening circles of people, leading to corresponding changes in individual behavior. Changes in behavior, exhibited by the persons directly involved in the activity, influence changes in their own immediate workplace or community settings. This leads to concrete changes in the way things are done. Others start to notice the changes and, if they like them, support the new ways of doing things. Indeed, this level of support increases to a point where the changes become institutionalized—a part of the way things are usually done. Herein lie the seeds of sustainability.

The designers delineated five "zones" of capacity building to simplify data collection, analysis, and the presentation of the results: mobilization, planning and organization, learning, diffusion, and institutionalization. The zones overlay on the spiral and reflect aspects of the capacity-building process detailed

Figure 8.3: Spiral Model of Capacity-Building Zones

above. As illustrated in Figure 8.3, these zones of capacity building overlap. Learning, for example, takes place throughout a large portion of the capacity-building activity.

As applied to the HDP, there are—within each of the three streams—a multitude of activities. Some activities are large scale, some small; some activities are slow to come to fruition, some are much faster to take hold. It is intended that each activity in some way contributes to the achievement of the purpose of the HDP.

There are two major kinds of constraints on the capacity-building process—internal and external. Internally, the transition from one phase of the capacity-building process to the next is by no means a certainty. The upward spiral of capacity building is rarely—if ever—a regular, smooth flow. For example, a process might get off to an inappropriate start as a result of developing an idea that does not squarely address the problem. Later on in the spiral, particular people chosen for the activity may, for one reason or another, be unable to make use of the activity to bring about the desired change. Conversely, the appropriate people might be involved, but the activity may be wrongly designed or implemented.

The second kind of constraint is that imposed from outside the project activity. In Figures 8.2 and 8.3, the spiral starts well within the confines of the box, but as the idea develops into an activity and the stakes increase, the spiral begins to push against the outside forces. Sometimes, the outside forces can be so overpowering that they close in on the capacity-building activity and

slow or stop its progress. Other times, the capacity-building process can be managed in such a way that the externally imposed constraints are reduced—that is, the spiral pushes the box outward.

The same external environment that poses constraints on a capacity-building process can also contain enabling factors that, if taken advantage of, can help the activity achieve its purpose. In this conceptual model, the relationship between the spiral and the box is dynamic—one can influence the other, and the nature and strength of this influence can change over time.

From this model emerges key questions to guide the evaluation team's inquiry within all streams of project activity. Questions used in this evaluation are included in the box above. Using the questions listed in the box as a guide, the HDP evaluators examined a variety of activities within each of the three project streams. They also considered the extent to which project activities reinforced each other and moved the HDP toward its overall purpose.

Implementing the Process Evaluation

The tasks of the process evaluation were sequenced in four stages, as shown in Table 8.1. The table shows both the time span and the number of person-days required to carry out the set of activities in each stage. It indicates that the first and the last stages of the evaluation spanned the greatest amount of time, but that the information-gathering stage, while lasting only two weeks, required the greatest investment of person-days. Highlights of each stage are outlined below.

Stage I: Preparation

• A half-day "think tank" session in Calgary: This session was instigated by the Canadian coordinator and involved members of the Canadian Advisory Group and evaluation specialists. It yielded the initial concept paper with rationale, preliminary design considerations, and a rough timetable for the evaluation.

• Preparatory visit to Nepal by the evaluation coordinator: The evaluation concept and plan were further developed participatorily with all levels of HDP staff in Nepal. The spiral model of capacity building, for example, emerged from a workshop with project staff in Surkhet District. The composition of the evaluation team was finalized, including locally recommended members. Timetable, logistics, community sites, and budget were decided. A final workshop in Kathmandu involving project staff, core evaluators, and key contacts from related institutions identified key issues for the evaluation and reached consensus on what the process evaluation should achieve for the project.

• Terms of reference document developed: Based on the output of the series of meetings and workshops in Nepal, this document guided the subsequent planning activities in the field.

• HDP community development field staff took the idea of the process evaluation to the community and district health leadership: Staff encouraged villagers at all three selected communities to build up their own ideas about evaluation on the basis that outsiders were coming to "learn" about their development experience. Discussion yielded ways that maximum numbers of villagers could be involved as a learning experience.

• Creative compromises: Balancing the requirement that all researchers have an opportunity to observe the same things within the available budget and time frame forced a compromise to interdisciplinary research. The solution: two subteams of core researchers—one subteam would focus on community activities, the other on district-level activities. The teams would meet as much as possible throughout the fieldwork in order to learn from the other team's observations and insights.

Stage II: Orientation

• Orientation packages for the core evaluators: Prior to convening in Kathmandu, each team member received an information package with terms of reference and assorted project documentation.

• Two-day orientation and team-building workshop in Kathmandu: The core evaluators, project staff (Surkhet and Kathmandu), and resource persons from the Ministry of Health and IOM closeted themselves with a trained facilitator. They reviewed the spiral model and the evaluation questions. They learned the basics of interdisciplinary team research and participatory appraisal techniques, clarified role expectations of the team and the staff, and practiced evaluation techniques in role plays of community and district health situations.

• One-day orientation in Surkhet: Evaluators, staff, and resource persons participated in another orientation/familiarization session with the counterpart evaluators from the community and the district health system. Ice-breaking games and role playing enabled the counterpart "pairs" to

become comfortable prior to working together in the community or district setting. The district group planned their agenda, while the community group further practiced interviewing skills.

Stage III: Information Gathering

The core evaluators spent two weeks collecting information. They met key government officials in Kathmandu, district officials, hospital and health post staff around Surkhet District, and villagers in three of the five participating VDCs. With the help of project staff, the team singled out key questions and usually designated one or two members to lead the questioning. Any interviews with high-level officials were formal and planned ahead of time. More informal techniques were used when meeting with villagers and health practitioners. All team members present for interviews took notes.

National and District Data Gathering

The district-focused evaluators:

• Met with nearly a dozen key informants within the Ministry of Health and National Planning Commission;

• Met with the faculty and students of the Generalist Physician Training Program at the IOM;

• Observed residents of the generalist training program performing surgery and conducting rounds;

• Talked with patients and outpatients of the hospital to find out how the presence of the residents was affecting service;

• Invited the residents to breakfast and asked them to comment on the training curriculum, the training sites, and their own career intentions;

• Went to the independent prenatal health clinic, and across town to the leprosy hospital and tuberculosis clinic, to find out how the HDP's district health collaboration strategy is viewed by other health organizations;

• Went to a primary health care center and to selected health posts and

sub–health posts to gain an impression of staffing and supervision, equipment and facilities, the supply and dispensing of drugs, and collaboration with community groups; and

• Held casual conversations with the users of health posts to hear their impressions of the facilities and service.

Community Data Gathering

The community-focused evaluators operated within a much more informal environment. Instead of meeting face-to-face across a desk or room, they met in circles and clusters under trees, in courtyards, or on the street. HDP community development staff in each village, responsible for initiating village-level planning for the evaluation, had agreed upon a common strategy during the preparation stage. As a result, the two- to three-day agenda was the same at each site.

Meeting with Counterpart Evaluators

The agenda opened with a half-day session with the counterpart evaluators. In this meeting, the counterpart evaluators spoke in depth about the evolution of the community development process and how they became involved. They displayed "social maps" portraying neighborhoods within the village and plotting the visible results of the community development process.

Presentation of Social Maps

The maps portrayed such features as latrines, vented stoves, irrigation projects, neighborhood water taps, beehives, bamboo plantations, reforestation zones, and health facilities.* The authors had signed their names at the bottom of each map. Some groups had made use of symbols in their maps and shown the actual number of households, latrines, stoves, and so on in a box at the bottom of the sheet. In others, every single house, stove, latrine, irrigation canal, water tap, and so forth was accurately represented. In one village, members of the neighborhood groups had taken the additional step of analyzing their maps from a "before HDP" and "after HDP" standpoint and presenting

* The maps were created by the villagers themselves using a participatory rural appraisal (PRA) technique, a user-friendly methodology for illiterate and semiliterate populations.

the results on a separate flip-chart sheet. An abridged version of this chart is shown in Table 8.2.

The core evaluators used this information as a "springboard" for their community research. They huddled around the social maps with the counterpart evaluators and heard how the various activities unfolded—how the villagers identified the problems, arrived at a solution, found the resources, and organized themselves to carry out the work.

Community Walkabouts

Having analyzed the social maps, the evaluators and their community counterparts went on a "walkabout"—usually within the same neighborhood examined in the map. Along the way, team members stopped to talk with villagers active in the project, as well as those not involved. They sat in the middle of community-controlled forest conservation areas and learned about the measures taken by the community to curb deforestation. They stood in front of contaminated water sources about to be transformed into secure tap systems. They witnessed literacy classes in progress and asked the students about their lives, why they had joined these two-hour evening classes, and what they hoped would be different in their lives once they could read, write, and use numbers. And, upon invitation, the evaluators peered into kitchens to see the vented stoves and asked the owners why they had chosen to switch from the open fireplace to this new stove technology. In some households, they asked the opposite—why families had not chosen to adopt a vented stove or latrine.

Debriefing Evenings

Most nights, the community and district evaluators (both core evaluators and counterpart evaluators), as well as the resource persons, sat to go over the day's observations to glean insights. When possible, the community and district evaluators combined their debriefing sessions to keep each other apprised of the emerging picture of HDP capacity building. At times, the facilitator of the debriefing session (this responsibility shifted from person to person) encouraged the team members to relate observations to the spiral model. Often, particularly toward the end of the field research, this happened naturally.

The debriefing sessions set the stage for the next day's research. Participants would often become aware of gaps in knowledge and therefore plan to seek answers at the next opportunity. Sometimes, the district evaluators asked the village-level evaluators to gather health-related information from the village. In one situation, for example, the district evaluators asked the village-level evaluators to find out how villagers felt about the two-rupee registration fee charged by health post staff for every medical consultation.

Local Feedback Assembly

During the final day in each VDC, the evaluators ended their research with village assemblies. The purpose of the assembly was threefold:

• To seek verification and further analyze insights gained by the core evaluators while in the village;

• To give the counterpart evaluators an opportunity to ask questions of their peers about the progress of the locally organized groups and the value of their initiatives to date; and

• To share the impressions gained by the core and counterpart evaluators and reinforce the values underlying the community development effort.

In each of the three VDCs, more than 100 people came for the two-hour meeting, which combined small-group discussions with a plenary. The format for the small-group sessions emerged from outstanding questions or issues from the time spent in the village. Sometimes, the topics were thematic—for example, the changing roles of men and women, or agriculture and forests. Sometimes, the groups were drawn together by neighborhood affiliation for a broader discussion of the community development process as seen from that geographic vantage point. The spiral model of capacity building was not

incorporated into the discussion with villagers. Rather, the presentations built upon the social mapping and other analysis already completed by the counterpart evaluators within the community prior to the core evaluators' arrival.

Stage IV: Synthesis and Reporting

Intrinsic to the design of the process evaluation is the idea that all stakeholder groups participating in design and research should also be part of a report-back process. Thus, in the VDC, the visit ended with the village assembly described above.

Prior to leaving Surkhet, HDP field staff organized a one-day debriefing meeting for all those participants who had attended the initial Surkhet orientation

two weeks earlier—the village counterpart evaluators, Ministry of Health officials, HDP staff, and core evaluators.

Another feedback session was held in Kathmandu, following which the evaluators revised the reports and used them as a basis for writing a draft document.

A presentation and feedback session was held in Calgary for the HDP Advisory Group and the Division of International Development. The draft was then circulated among project staff and CIDA for comment and action before being finalized in its current form.

Presentation of Major Findings

The evaluators concluded that progress in the capacity-building process has been uneven across the three streams of project activity—community, district, and physician training. The HDP has been more successful in stimulating "bottom-up" development with the VDCs than in stimulating "top-down" development with the Ministry of Health.

The spiral model is used to illustrate the degree of capacity building observed. Figures 8.4 and 8.5 provide a sample representation of the findings for the community development and the district health streams of activity. The spiral is positioned on the right-hand side of each figure. Indicators of capacity building are listed on the left-hand side, corresponding to the five zones of the capacity-building process—mobilization, planning and organization, learning, diffusion, and institutionalization. Those indicators written in plain bold text represent findings observed by the evaluators. Indicators written in italics represent other expected findings or situations that were not observed by the evaluators.

Capacity Building in the Community Stream of Activities

In the community stream, the evaluators found that initiatives are on the brink of sustainability and need short-term support to consolidate independent,

Figure 8.4: Capacity Building Observed in the Community Stream of Activities

Figure 8.5: Capacity Building Observed in the District Stream of Activities

proactive local organizations. The spiral diagram (Figure 8.4) documents the presence of indicators of capacity building in each zone up to and including institutionalization.

Evidence suggests that the community development process has:

• Heightened the level of confidence among villagers;

• Built a stronger sense of community identity;

• Created a vigorous democratic decision-making structure;

• Trained villagers in community leadership (e.g., problem solving, conflict resolution, and planning), and in a variety of technical skills (e.g., stove and latrine construction, beekeeping, forest conservation, literacy, and community banking);

• Attracted attention among increasing numbers of people within the VDCs (at the time of the evaluation, between a quarter and a half of all families across the three VDCs were active participants in community development activities);

• Enthused neighboring communities about the community development process (indeed, expansion from VDC to VDC has been influenced much more by villager demand than through promotion by project staff);

• Engendered new ways of thinking about personal health and hygiene, resource conservation, gender and caste relations, community organization, and the role of external development service providers (i.e., line agencies and NGOs); and

• Matured to a point where two of the three VDCs are ready to form their own independent associations.

External Factors Influencing Community Capacity Building

Following the spiral model, the evaluators noted the major constraints and enablers influencing the capacity-building process at the community level, for example:

1. Ecological issues as an enabler. Villagers are acutely aware of the disappearance of protective vegetation and soil erosion. Forestry groups, which have been established in all five VDCs, have established relationships with the Ministry of Forestry, designated zones for reforestation, planted trees, hired wardens, and established village bylaws (with enforcement) to control access and use. Villagers are also cutting back on their consumption of firewood, both because of the increasing scarcity of the resource and because of the lower fuel needs of the new vented stove.

2. The country's political and administrative situation as both an enabler and a constraint. The restoration of a multiparty democracy in 1991 with its commitment to decentralization created new openness toward local-level planning and management. However, embedded features of the political/administrative system continue to constrain the process, notably the lack of readiness or ability of the line agencies, including the Ministry of Health, to respond to local participation.

3. Long-standing economic conditions as a constraint. The VDCs are located in food-deficit areas; those families that have not been able to grow enough food for themselves or find local work rely on work outside Surkhet, often in India. Lack of income undermines people's ability and time to participate in group discussion and inhibits risk taking. HDP field staff have approached this problem through microenterprise and savings and credit schemes.

4. Local history and attitudes toward outside development organizations as a constraint. The HDP has had to overcome a strong tendency among villagers to see the project as another "provider" of service. The rise and fall of participation levels in the community development process may be symptomatic of this tendency. As is typical of many projects, HDP staff have had to continually balance the need to engage villagers in their own problem solving with the need to achieve visible results/success, which in turn builds local support for the process.

Capacity Building in the District Stream of Activities

The evaluators found that the district management strengthening outputs, while well planned and received by the Ministry of Health and others, are not close to institutionalization. The spiral diagram (Figure 8.5) reflects the evaluators' observations that while there is some diffusion of learning as a result of project activities, there does not appear to be lasting change in the way the district health system functions.

In general, the HDP's district health initiatives have yielded:

• Successful activities designed to strengthen the District Public Health Office, such as annual report writing and the training of 140 traditional birth attendants affiliated with the health posts;

• Innovative and tested in-service training packages for all health post staff based on needs assessments—these packages have been used beyond Surkhet;

• Improved collaboration and coordination among regional and district health professionals and community-level staff;

• Efficiently functioning drug schemes in all health posts in the district; and

• A new district hospital facility.

Activities appear to have been carefully identified and implemented in collaboration with key Ministry of Health officials, as highlighted in the lower two zones of the spiral. However, the results of these activities have not, with the exception of the drug schemes, had a lasting impact on the Surkhet District health system. There has been very little diffusion of new skills, attitudes, or behaviors within the health system.

External Influences on District Capacity Building

External constraints clearly influenced the district stream of activities. The lack of diffusion in building institutional capacity can be attributed largely to the following factors:

• Frequent transfers of staff in and out of the district health system;

• Continuing scarcity of financial resources for health programming;

• Sweeping changes to the organizational structure of the Ministry of Health;

• A host of historical-cultural factors influencing the way the bureaucracy works; and

• Lack of skilled planning, given the limited resources available.

The evaluators concluded that, in view of transfer and appointment practices, it is unlikely that the gap between ministry policies and the implementation will be closed in the near future, and they recommended that the HDP reconsider the types of assistance that would be most fruitful in strengthening capacity building at the district level. In particular, the evaluators recommended that the project adopt methods for strengthening local management and staffing of health posts and the district hospital.

Reflections and Conclusions on Process Evaluation

The process evaluation methodology enabled the evaluators to look behind the visible outputs of the project—grassroots decision making, latrines, training curricula for health post staff—to find evidence of the HDP's capacity-building effect both on the communities and on the district health offices and facilities in this hilly, remote district of Nepal.

The methodology also encouraged the evaluators to appreciate the interaction effects both among individual project activities (e.g., savings and credit, literacy training) and between each of the three streams of activity (community development, district health strengthening, and generalist physician training).

Process evaluation allowed the team to identify indicators of capacity building in all three streams. It gave project implementers, from village to management level, exposure to evaluation as a relevant tool for quality control. It helped project stakeholders articulate a conceptual framework underlying the HDP—the spiral model of capacity building.

The following is a list of lessons learned from this first application of the process evaluation methodology. Lessons are arranged under the four characteristics of the methodology described earlier in the chapter.

Use of a Conceptual Model

• The model can be used to analyze a single activity, multiple activities, or the project as a whole.

• The narrower the scope of the analysis, the deeper the analysis.

• The team must be comfortable with all conceptual tools prior to fieldwork.

• Orientation is critical for building the team dynamic necessary for effective fieldwork and for understanding the conceptual model and translating it into a specific evaluation plan.

Reliance on Participatory Strategies

• The responsibility for evaluation design and management should be shared among stakeholders. If people know how they can contribute to the planning and management of the evaluation, and are keen on the exercise, they will offer their creative input. Participatory design and management, however, require good rapport and communication.

• People's participation in the process of evaluation itself builds individual capacity.

• Staff can offer a depth of understanding about subject matter.

• Staff can sometimes be put in compromising situations and might inhibit research activities and/or perceptions of nonstaff evaluators.

• Participatory evaluation is much less threatening than conventional evaluation, since it bridges cultures, staff with nonstaff, and local with external.

• Participation of local people as evaluators allows questions to be translated into village-level terminology and seems to increase the comfort level in the discourse that follows.

• Evaluators cannot assume that all stakeholders are able to analyze situations in a critical manner; some stakeholders are more analytical, others more descriptive—they should be allowed to complement one another.

• Evaluation teams should have a person designated as a process facilitator or manager to ensure that positive group dynamics are maintained.

Adoption of Participatory Appraisal Techniques

• Within a team, roles should be clearly delineated ahead of time. For example, are staff to be evaluators or resource persons? Who translates and interprets? Who leads off in the information-gathering session?

• It is important to critique one another's roles throughout evaluation.

• Evaluation team members should learn as much as possible about the others' strengths and weaknesses. In sharing responsibilities, the team should draw on member strengths.

• Everyone on a team should take notes.

• The team should keep a set of combined notes from debriefing sessions.

• Daily team debriefings and planning are essential to manage the tremendous amount of information that is collected.

• Evaluators should always refer to the conceptual model and accompanying questions when debriefing and planning for fieldwork

• It is important to make the team as inconspicuous as possible. Large numbers and "loud" presence get in the way of good information gathering.

• Accidental interviews are an important means of getting "backstage" information and a broader context for research findings.

• Social mapping is very valuable for collecting both quantitative and qualitative information from groups. Mapping is visual, participatory, and evocative.

• Using an existing body of information (e.g., a social map) can help focus inquiry.

Use of a Qualitative Approach

• Process evaluation is most effective if the methodology is designed for ongoing use from the outset of the project or activity.

• Process evaluation is a learning methodology; the more times it is practiced, the more competently it can be carried out.

• Process evaluation tends to make explicit what is known implicitly.

• Process evaluation is cost-effective if integrated into strategic planning and management of the project; otherwise it appears costly in terms of time and funds.

• Since project implementers are participants in the evaluation, the leap to planning and management is a small one.

The cycle of synthesis and reporting (which was repeated in Surkhet, in Kathmandu, in Calgary, and in the draft and final process evaluation reports) served to clarify the findings and facilitate the rapid implementation of changes in the management of the project. By the time the final document was issued, after one year, most of the recommendations had been addressed. In a sense, the earlier phases of the synthesis and reporting accomplished most of what was expected of an evaluation, while the final document serves as polished reference material.

Having used the process evaluation methodology once, the evaluators believe that it can be used repeatedly throughout a project. Each time the methodology is applied, either for ongoing monitoring (where the focus is on operational effectiveness) or for periodic evaluation (where the focus is on progress toward the project purpose), the framework evolves. This evolution occurs, over time, as conditions change and stakeholders learn about the effects of their capacity-building endeavors.

Costs of the Methodology

• Process evaluation involves a large number of individuals, from both the project setting and overseas. In the HDP, approximately twenty-seven people participated from Surkhet District (twenty from the communities, four from the district health system, and three from the HDP field office), eight from Kathmandu, and three from North America. It is estimated that the process evaluation took a total of 560 person-days to complete, an average of twenty days per person.

• The duration of the process evaluation is uncertain, as it depends on the readiness of the project staff and beneficiaries to fully participate alongside the external evaluators. In the case of the HDP, the exercise lasted twelve months from commencement of preparations in November 1993 to preparation of the draft evaluation report in October 1994. Intensive involvement lasted three months, from mid-January to mid-April 1994. As this first application of the process evaluation included the initial design work, subsequent applications would likely consume less time.

• Process evaluation is somewhat "messy," as it depends on the speed with which participation occurs—it cannot be rushed to meet the time lines of external evaluators. It may be as costly as a conventional evaluation or even more so, particularly when more individuals are involved and there is a higher total contribution of time. In the HDP evaluation, costs were comparable to those incurred by CIDA for regular end-of-project evaluations.

• In the absence of a conventional evaluation, the process evaluation methodology may have to be supplemented with surveys designed to provide information on items such as sources and uses of funds, audit procedures, allocation and costing of inputs, and authority and responsibility within the organization.

Benefits of the Methodology

• Process evaluation methodology supports the trend toward results-based management. It extends the emphasis beyond the traditional focus on outputs toward longer-term results. Process evaluation methodology helps managers gain maximum benefit from the interplay between action and reflection.

• The methodology is flexible enough to be used for short-term project monitoring and for long-term evaluation. It can inform the project of short-term operational issues as well as longer-term strategic issues. Put another way, it can be developed for use in measuring progress in relation to outputs, as well as progress in relation to the broader indicators of replicability and sustainability.

• Process evaluation is sensitive both to the concrete project outputs and to the less tangible human dynamics (individual and organizational change) that form the backdrop to the outputs.

• Because all of the primary stakeholders (those whose lives and work are directly affected by the interventions of the project) are active in the conduct of the process evaluation, it is relatively easy for recommendations that flow from the evaluation to receive full support and be quickly implemented. In the HDP, a number of the recommendations from the process evaluation were acted upon during preparation of the draft final report.

References

Bista, D. 1991. Fatalism and Development: Nepal's Struggle for Modernization. Calcutta: Orient Longman.

Davis-Case, D., ed. 1991. The Community's Toolbox: The Idea, Methods and Tools for Participatory Assessment, Monitoring and Evaluation in Community Forestry. Rome: Food and Agriculture Organization.

Freedman, J. 1994. "Participatory Evaluations: Making Projects Work." Technical Paper No. TP 94/2. Division of International Development, International Centre, University of Calgary.

Grandstaff, T. B., and D. A. Messerschmidt. 1993. "Manager's Guide to the Use of Rapid Rural Appraisal." Farm Management and Production Economics Service, Agricultural Services Division, Food and Agriculture Organization of the United Nations, draft.

———. 1995. Manager's Guide to the Use of Rapid Rural Appraisal. Bangkok: Food and Agriculture Organization of the United Nations.

Justice, J., H. Dixit, D. Harding, and P. Cox. 1995. "Process Evaluation of the Nepal Health Development Project: Final Report." University of Calgary, International Centre.

Messerschmidt, D. A. 1993. "Rapid Appraisal for Community Forestry: The RA Process and Rapid Diagnostic Tools." Institute of Forestry, Pokhara, Nepal.

———. 1995. Rapid Appraisal for Community Forestry: The RA Process and Rapid Diagnostic Tools. IIED Participatory Methodology Series. London: International Institute for Environment and Development.

Purdey, A. F., G. B. Adhikari, S. Robinson, and P. Cox. 1994. "Participatory Health Development in Rural Nepal: Clarifying the Process of Community Empowerment." Health Education Quarterly 21 (3): 329–43.

Robinson, S., and P. Cox. 1995. "Process Evaluation in Health Development: Tracking the Results of Capacity Building." Technical Paper No. TP 95/1. Division of International Development, International Centre, University of Calgary.

Robinson, S. A., P. Cox, I. G. Somlai, A. Purdey, and B. R. Prasai. 1997. "Process Evaluation: A Field Method for Tracking Those Elusive Development Results." Canadian Journal of Development Studies 18: 805–34.

Document(s) 10 of 18