Document(s) 7 of 18

In January 1983 seven of us who had been involved with rural development for several years spent three days in Secunderabad, India, discussing how to evaluate the process of participatory development. Our meetings were held in the office of the Rural Development Advisory Service, and five of us also stayed there. Getting to know one another during and outside the meeting was as rewarding an experience as the discussions themselves. At the end of the three days, all of us felt satisfied with the consensus and mutual understanding we had managed to arrive at and the friendships we had built. This chapter is an attempt to share our discussions with those who were not with us but who are as interested in these issues as we were and continue to be.

What follows is a more or less verbatim reproduction of our discussions, albeit arranged a little more systematically. We are sharing our discussions with you in the hope that this document will lead to further discussion and greater clarity about the evaluation of development activities.

Background

Different development approaches and strategies require different kinds of evaluation methods and techniques. If development projects are top-down, started by people from outside the community (governmental or nongovernmental organizations or agencies) to provide services such as health and education and to bring about certain changes in production methods and techniques, then the local people are merely recipients, targets, or objects of development. People for whom development is supposedly intended have little or no say in the content

The workshop participants were Abha Bhaiya, Aruna Roy, Datta Savale, Kamla Bhasin, M. Kurian, M. V. Shastri, and Vikas Bhai.

An earlier version of this chapter was first published by Freedom from Hunger Campaign/Action for Development Program of the Food and Agriculture Office in New Delhi, India.

and direction of such efforts. With hindsight it can be said that top-down, centralized development projects seldom help the really poor and needy because the real causes of poverty are left unquestioned and unchallenged. The evaluation of such projects is also, quite logically, top-down, geared and done by the decision makers without any participation of the local people. Those from whom information and opinions are gathered are not even informed about the evaluation outcome. In fact, often even the project holders have no say. For them, more often than not, evaluation is like an inspection being carried out by outsiders at the insistence of funding agencies, and they feel threatened by it. The main purpose of such an evaluation is clearly one of financial accountability, and emphasis is on physical targets. Because this model of development does not insist on starting a process of consciousness-raising, increasing awareness, and mobilization, little attention is paid to the assessment of intangibles such as people's participation, the decision-making process, level of awareness, and practice of democracy.

Development, however, can also be understood as a means of helping the poor to collectively analyze the socioeconomic, political, and cultural structures that keep them poor and get organized to challenge these structures. In such a development model, the oppressed people are seen as subjects, not merely objects of their own development. The program is a partnership between the local masses and outsiders. Its strength is concomitant with that of the people's organizations (POs) that emerge, their democratic functioning, and the actions they take to tilt the balance of power and resources in favor of the exploited masses. The evaluation of such efforts for development and organization has a different purpose and demands other methods, techniques, and indicators.

We who met in Secunderabad were interested only in the evaluation of the second type of development efforts. All of us felt that although a large number of action groups (AGs) are now concentrating on the mobilization and organization of the poor, there is little clarity on how these efforts should be assessed. The purpose of our talks was to achieve some common understanding on the basis of the experience and ideas we all had on evaluation.

Some Important Considerations

We agreed that the evaluation of people-centered and people-oriented efforts at consciousness-raising, mobilization, organization, and action should consider several key points.

Evaluation Is Reflection on Action

Evaluation, as we see it, is collective reflection on the actions taken by individuals within a group and by the group itself, and on the methods of functioning of a group. Its purpose is improvement both in the understanding and analysis of reality and issues and in future action. Thus seen, it is an important method of group education and learning.

Built-in and Ongoing Evaluation

For a group interested in improving not only the socioeconomic position of the poor but also the methods of functioning and the understanding of everyone involved in the work, evaluation has to be built-in and ongoing. Reflection based on concrete information has to be closely linked to action. In addition to ongoing evaluation, at the end of one or two years, there can be an overall, time-bound evaluation that is a cumulative assessment of what has taken place over a decade.

The experience of the Comprehensive Rural Operations Service Society (CROSS) shared by M. Kurian illustrated very well the method and importance of ongoing evaluation and its culmination into an annual exercise. The village sanghams (small, face-to-face groups of rural poor) initiated by CROSS assess their activities and the performance of the functionaries every month. In addition, they assess every major action undertaken by them. Evaluation sessions are also organized every three months at the cluster and area level. Apart from these evaluations by the local people, CROSS staff meets once a month to take stock of its activities and methods of functioning. Annual self-evaluation is done in January of every year.

Emphasis on Self-Evaluation

The emphasis of a people-centered and people-oriented program or organization has to be on self-evaluation in which the people and the organizers not only participate but also decide about its parameters, form, and methods. The final judges of a program's effectiveness must be the people themselves.

Evaluation of Tangibles, Intangibles, and Processes

If the objectives of development are both tangibles (such as improved economic status, improved health) and intangibles (such as increased awareness, people's participation, and democratic decision making), then obviously evaluation must also focus on both these aspects. There are techniques available for assessing tangibles, but we need to develop methods and indicators as far as intangibles and processes are concerned.

The process a group goes through to reach decisions and act is as important as the outcome of the action. We have to understand how people move toward the achievement of their objectives. It is necessary to understand how the processes within POs and AGs are related to general processes in society and how they affect each other. Their context has to be understood.

Just as there is a close relationship between action and reflection, theory and practice, there is also one between tangible objectives, such as increased access to land or higher wages, and intangible ones, such as improved level of awareness and strength of POs. Ideally, the achievement of one should lead to improvement of the other.

POs might be fighting for economic benefits, but unlike the usual development projects, POs emphasize the processes and use each struggle to educate and strengthen themselves. After achieving some small victories, POs cannot sit quietly and smugly, but need to constantly ask how much space has been created by a campaign and how that space should be used for future action. For them, the process of structural change should be an ongoing one that does not stop at any particular point. This is different from target-bound projects, considered terminated on completion of a certain number of wells, the installation of pump sets, the production of biogas, and so forth.

False Dichotomy between Consciousness-Raising and Economic Development

When the entire emphasis of development programs is on material development, quantitative analysis is primary. But when the emphasis of development efforts is on the growth of people and their organization, qualitative analysis assumes more importance. Because material development and the development of people's consciousness and their organization do (and must) go together, quantitative and qualitative analysis cannot be exclusive of each other. Some groups take an extreme position and reject all quantitative data and measurement of material development. They talk only of intangibles like consciousness-raising or increasing the level of awareness. We felt a need to have a good synthesis of evaluating tangibles and intangibles, quantitative and qualitative results. If one is working with the really poor, their material conditions have to be improved fast (mainly, of course, through their own efforts). The poor are not going to be interested in consciousness-raising for its own sake. All consciousness-raising must lead to an improvement in their material conditions, and vice versa. In fact, this dichotomy between organizational work and programs for economic development is false and misleading. Groups primarily doing organizational work also improve the economic status of the poor at least as much, if not more, as the so-called projects for income generation do. Organizations such as Bhoomi Sena, Shramik Sangathana, and CROSS have achieved tremendous economic benefits for the poor through their struggles to recover alienated lands, provide higher wages and employment opportunities, lower interest rates, fight corruption, reduce the power of middlemen, and so forth. The economic position of the poor can be improved by removing insecurity and exploitation, and if these two tasks go on simultaneously, it is ideal.

The attempts to organize the poor also improve their receiving mechanism and bargaining power and thereby enable them to make increased use of government schemes, bank loans, and the like. To recapitulate, economic development and people's organizations and action are—and should be—dialectically related. Every struggle by the oppressed should create more space for their economic development, and their improved economic status should in turn strengthen their organization.

Need to Look at Three Kinds of Processes

We need to evaluate processes in three areas or realities, and also to look at the interplay among these three:

1. The AG's reality and the processes within it.

2. The community within which the AG is working and the processes within the community.

3. The larger socioeconomic and political reality in which both AGs and oppressed communities are situated.

It is important to analyze and understand why some people form an AG, why they want to relate to a certain oppressed community, what their perception of the larger reality and structures is, what conception of change they have, and what their goals and aspirations are. Is there any homogeneity between the aspirations and understanding of the AGs and those of the community within which they work? How realistic are the objectives set by them in the context of opposition forces?

Interplay of Aspirations and Reality

It is also important to look at the objectives and aspirations of AGs and POs in the context of the forces of reality. We have to see the dynamics between both. The reality exists and operates independently of aspirations of AGs and POs that intervene to change it according to their own understanding. So we must understand the totality of the forces of society and see what the intervention has succeeded in achieving.

Not only is there need to assess the extent to which the objectives and aspirations have been achieved, but they, themselves, have to be constantly reviewed and readjusted according to changing reality and changes in AGs' and POs' understanding. We need methods and tools to assess the AGs' and POs' goals in the context of their aspirations and hypothesis, and of the larger reality.

It is only when action is taken after a systematic analysis of the overall situation and reality that it becomes meaningful and effective. For example, if one does community theater without understanding the context, and if it is not related to any action, it provides, at best, some entertainment. People's theater can inspire and lead to action only if it is done with a perception of reality, and of the needs and aspirations of the masses. When divorced from POs and from action, theater, nonformal education, or consciousness-raising efforts are uninspiring and uninnovative and lead to no change in the oppressive situation and structures.

The Role of Outsiders in Self-Evaluation

Emphasis on self-evaluation does not mean that we took the extreme position that local people and AGs can assess their work themselves. We recognized that every perception has its limitations. Just as outsiders' perception might be limited because of their lack of knowledge and acquaintance with local realities, local people's perception might be limited because of their particularity. The interaction of perceptions and views (both of insiders and of outsiders) can therefore be very beneficial.

The presence of an experienced and sensitive outsider can encourage the group to formulate and articulate its thoughts more systematically and objectively. A sensitive outsider can enrich the discussions by bringing in other experiences, perceptions, perspectives, and dimensions. There can be areas that local people either forget to look at or do not want to look at. It is the outsiders' role to bring these forgotten elements or reality into discussion, however unpleasant this might be. Local people and AGs have to be helped to realize that unpleasant facts cannot be wished away. An outsider plays an important role by asking the right kind of questions and providing useful insights for dealing with dilemmas and uncertainties.

Outsiders can play this role effectively only if they are actually insiders in more than one way. They have to be known and acceptable to the people who are assessing themselves, should identify with the group's objectives, and should be involved in the same kinds of struggles and processes, although in another area or at a different level. Insofar as they are involved and have a commitment to the same goals, they are not "objective" evaluators. Has not the myth of evaluation being objective been exploded?

For helping in assessing various aspects of work, we might need different kinds of outsiders, for example, someone acquainted with health issues when it is about a community health program.

It must be remembered that an insensitive outsider can ruin all efforts at a genuine self-evaluation; instead of leading to a common understanding she or he can further divide the people and generally harm the organization and action.

In order to be effective, an outsider has to be thoroughly prepared by gathering whatever information is available about the organizations and the local and natural realities within which they are operating and that they want to change through their interaction.

It was pointed out that AGs can also help each other in their self-evaluation. The same is possible between communities and groups. Experienced members of one group can help others in their self-evaluation. Such interaction strengthens the links between different groups and thus increases their joint strength.

Self-Evaluation Is Possible Only If the AG Is Ready for It

It was stated that all AGs do not recognize the need for an honest self-evaluation. Some of them consider it a waste of time. They want to get along with action and see reflection as separate from it. For them, reflection is unnecessary theorizing that delays action. Of course, when taken to an extreme kind of "hair-splitting," reflection can indeed delay action; in fact, at times, it becomes its substitute. But reflection is absolutely necessary (in right measures), especially to avoid the other extremes of activism.

Some AGs might recognize the need for self-evaluation but might not be ready for it, because it analyzes all aspects of work and relationships, and this can be a very painful process, especially in the beginning. It requires a certain self-confidence, the ability to look at oneself critically and to listen to criticism without getting defensive or aggressive.

It is only when at least some members of the AG recognize the need for a self-evaluation that its process can be started. As the latter goes on, other members might also recognize its usefulness and importance and join it.

Self-Evaluation: An Illustration

Aruna Roy shared with us the experience of her group, the Social Work and Resource Centre (SWRC), with a self-evaluation process. Her case study shows how, through it, changes took place in their understanding and analysis of the reality around them, and their own role vis-à-vis this reality.

SWRC started work in 1972. Initially it was primarily a group of professionals trying to provide technical and managerial solutions to the problem of poverty and injustice. At that time SWRC did not work exclusively with the poor, nor did it have their organization as its objective. In the course of the first three to four years, some questions cropped up in some of the workers' minds about the larger reality, the community within which the AG should work, the adequacy of technical solutions, the role of professionals, and so forth. This questioning by individuals within the AG led to some creative tension and changes in the work, but for another two years there was neither a collective questioning nor a clearly expressed need for evaluation. In 1978, eight to ten members started to concretize the issues, and a debate began within the group on the need for self-questioning. This small group started meeting informally to formulate the questions that were in their minds. They reflected on all issues bothering them and on the relationship of this questioning to their understanding, their work, and local reality. They also identified problems in the following areas of their work and group functioning:

• Communication within the AG itself and between its members and local people;

• Different kinds of inequalities and differences in status within the AG;

• Concentration of decision making in a few hands, and the need to create structures that would ensure broader participation and reduce the exercise of informal power; and

• Place of economic development and its relationship with politics, social change, and so forth.

The group, small at the beginning, gradually expanded to reach eighteen to twenty members. They once sat almost every day for about six weeks during which their own work was more or less suspended. This activity was not seen very favorably by some other AG members, but they did not object to it. Watching cautiously, they even joined some of the sessions, but distrustingly. The ball that had been set rolling moved on. Later a group of forty had two four-day sessions with eight outsiders well known to them and who, it was felt, would be able to help them deal with certain dilemmas faced and questions they had regarding the nature and direction of their work, the role of an institution like theirs, development programs versus organization, and the like.

There was a tremendous heterogeneity among the members in terms of their social and educational backgrounds, understanding and articulation of issues, and commitment to change. The pace of discussions was therefore slow, and everyone did not participate equally.

The kinds of questions raised and answers attempted are given here in Aruna's own words: "We demanded openness and ability to discuss even personal commitments and aspirations. We broke the barriers between our professional and personal lives. We realized that our objectives had been too general. We narrowed them down. We decided we should work mainly with the poor. We formulated a decision-making process which was participatory. We wanted a forum in which every worker could effectively take part. We decided we should evaluate ourselves (our attitudes, behavior, understanding) once a year—how honest are we, how democratic, how open, how caste-minded? What is our understanding of issues? We discussed questions like what is more important for a worker—a Ph.D. or a capacity to communicate with people and elicit people's participation? But this process of personal evaluation when related to salary structures was not very successful. Subjective factors played too important a role and did not allow for the personal evaluation to become operational in relation to judgments by peers on one another's salaries. This power was vested by the group in its director, accepting its own failure.

"Also at the village level we had talks with people who had participated in our programs. We met them at one of the five field centers once a month on the new moon day and reviewed the various programs. Meetings were sometimes held with special-interest or program groups like crafts group, health group, and so on.

"We concluded that there was a role for an institution like ours. We discussed its role in development, 'agitation,' in trying to bring about structural changes. We also discussed whether it was possible for a development group like this to shift gear and go into organizational activity. Some felt it could be done, others that it could not, and should not."

The long talks obviously led to several changes in their work, in the decision-making process, and in interpersonal relationships. These changes led to the need for more discussion and clarity. In the end, a dialectical relationship seems to have been established between action and reflection, theory and practice.

Perceived Advantages of Self-Evaluation

According to Aruna, these self-evaluation sessions were extremely useful. At the end of it all, most participants realized that this kind of communication and openness is necessary for improving a group's effectiveness and impact.

Self-evaluation can help everyone to think and learn collectively, to articulate better. If carried on sensitively, it can make every participant more honest, sensitive, analytical, and open to change. It changes everybody's awareness and consciousness, as well as people's attitudes, and helps them to cope better with conflicts.

Self-evaluation can improve a group's inner functioning by creating better relationships between the different AG members. Open discussion on certain issues removes unnecessary misunderstandings. By talking frankly, even about sensitive issues, people begin to see and appreciate others' viewpoints.

Self-evaluation helps in evolving a common perspective, a shared commitment to action, and thus transforms a loose group of individuals into a cohesive and effective AG. As the analysis of the group improves, it understands better the larger realities and the interaction of its work with them. By making members critically conscious of their actions, it improves both a group's inner functioning and the work it does with people. According to Aruna, "an attempt to resolve our own dilemmas and conflicts led to greater clarity."

Such a process alters the relationships within the group and the relationship of the AG with the people. Because the AG becomes a cohesive group and develops a certain focus, AG members do not say different things about their work, and this improves the AG's image vis-à-vis the people. The misunderstanding or confusion that people might have about the AG's role, real motivation, and so on is reduced when it develops an open dialogue with local people and also involves them in the assessment of the work initiated.

Systematic self-evaluation requires that AGs develop methods of gathering and documenting information and of conducting free interaction and discussions and keeping records of these. AGs also have to look for indicators of consciousness and articulation. Because of all these conscious efforts at evaluation, the AGs' work improves.

In addition to improvement in the above-mentioned areas, which are mainly intangible, experience shows that self-evaluation improves the achievement of tangible results. This happens because action becomes much more relevant, conscious, and focused. As part of their self-evaluation, SWRC also did qualitative analysis and found that the former had led to better tangible results. (This was also Kurian's experience in CROSS.)

Some Examples of Bad External Evaluations

We also heard examples of some bad external evaluations conducted by social scientists and rural development and management experts, using the latest cost-benefit and social cost-benefit analysis. In order to get a good analysis of their work, CROSS got an evaluation done by a well-known organization. At the end of the elaborate questioning, data collecting, and processing, what CROSS got was merely a description of its work without any analysis. The evaluation failed to provide any guidelines for future action, which was the main purpose of having it done, and ended up giving CROSS a very good chit and a substantial bill. Similarly, some management people had gone to SWRC to conduct social cost-benefit analysis, and its outcome was not helpful either, at least not to the AG and local people.

The sharing of these experiences made us realize that there are no ready-made "scientific" tools available for the evaluation of efforts to raise people's consciousness and mobilize them. Established academic institutions cannot, for obvious reasons, be expected to provide the necessary help in this matter. AGs and mass organizations, together with some sensitive academics, will have to evolve methods and tools for assessing their work.

Document(s) 7 of 18