Document(s) 20 of 27

Flowchart: Steps in the development of a health systems research proposal

NB: Development of a research process is a cyclical process. The double-headed arrows indicate that the process is never linear.

Module 14: PRE-TESTING THE METHODOLOGY

What is a pre-test or pilot study of the methodology?

WHY do we carry out a pre-test or pilot study?

A pre-test or pilot study serves as a trial run that allows us to identify potential problems in the proposed study. Although this means extra effort at the beginning of a research project, the pre-test and/or pilot study enables us, if necessary, to revise the methods and logistics of data collection before starting the actual fieldwork. As a result, a good deal of time, effort and money can be saved in the long run. Pre-testing is simpler and less time-consuming and costly than conducting an entire pilot study. Therefore we will concentrate on pre-testing as an essential step in the development of research projects.

WHAT aspects of your research methodology can be evaluated during pre-testing?

1. Reactions of the respondents to the research procedures can be observed in the pre-test to determine:
   • availability of the study population and how respondents’ daily work schedules can best be respected;
   • acceptability of the methods used to establish contact with the study population;
   • acceptability of the questions asked; and
   • willingness of the respondents to answer the questions and collaborate with the study.
2. The data-collection tools can be pre-tested to determine:
   • Whether the tools you use allow you to collect the information you need and whether those tools are reliable. You may find that some of the data collected is not relevant to the problem or is not in a form suitable for analysis. This is the time to decide not to collect this data or to consider using alternative techniques that will produce data in a more usable form.
   • How much time is needed to administer the interview guide/questionnaire, to conduct observations or group interviews, and/or to make measurements.
   • Whether there is any need to revise the format or presentation of interview guides/ questionnaires, including whether:

     — The sequence of questions is logical.

     — The wording of the questions is clear.

     — Translations are accurate.

     — Space for answers is sufficient.

     — There is a need to pre-categorise some answers or to change closed questions into open-ended questions.

     — There is a need to adjust the coding system.

     — There is a need for additional instructions for interviewers (e.g., guidelines for ‘probing’ certain open questions).

3. Sampling procedures can be checked to determine:
   • Whether the instructions concerning how to select the sample are followed in the same way by all staff involved.
   • How much time is needed to locate individuals to be included in the study.
4. Staffing and activities of the research team can be checked, while all are participating in the pre-test, to determine:
   • How successful the training of the research team has been.
   • What the work output of each member of the staff is.
   • How well the research team works together.
   • Whether logistical support is adequate.
   • The reliability of the results when instruments or tests are administered by different members of the research team.
   • Whether staff supervision is adequate.

   Note:

   The pre-test can be seen as a period of extra training for the research team in which sensitivity to the needs and wishes of the study population can be developed.

5. Procedures for data processing and analysis can be evaluated during the pre-test. Items that can be assessed include:
   • Appropriateness of data master sheets and dummy tables and the ease of use.
   • Effectiveness of the system for quality control of data collection.
   • Appropriateness of statistical procedures (if used).
   • Clarity and ease with which the collected data can be interpreted.
6. The proposed work plan and budget for research activities can be assessed during the pre-test. Issues that can be evaluated include:
   • Appropriateness of the amount of time allowed for the different activities of planning, implementation, supervision, co-ordination and administration.
   • Accuracy of the scheduling of the various activities.

When do we carry out a pre-test?

You might consider:

• Pre-testing at least your data collection tools, either during the workshop, or, if that is impossible, immediately thereafter, in the actual field situation.
• Pre-testing the data collection and data-analysis process 1-2 weeks before starting the fieldwork, with the whole research team (including research assistants, if required) so that you have time to make revisions.

Which components should be assessed during the pre-test?

1. Pre-test during the workshop

   Depending on how closely the pre-test situation resembles the area in which the actual field work will be carried out, it may be possible to pre-test:

   • The reactions of respondents to the research procedures and to questions related to sensitive issues.
   • The appropriateness of study type(s) and research tools selected for the purpose of the study (e.g., validity: Do they collect the information you need? and reliability: Do they collect the data in a precise way?).
   • The appropriateness of format and wording of questionnaires and interview schedules and the accuracy of the translations.
   • The time needed to carry out interviews, observations or measurements.
   • The feasibility of the designed sampling procedures.
   • The feasibility of the designed procedures for data processing and analysis.

   Even if you cannot assess all these components fully, the field experience will provide information that will be quite valuable to you in reviewing the methodological aspects of your proposal and in planning your work plan and budget.

2. Pre-test in the actual research area

   All the issues mentioned above will have to be thoroughly reviewed during a pre-test in the actual field situation. Other issues, such as the functioning of the research team, including newly recruited and trained research assistants, and the feasibility of the work plan, can only be tested in the research area. An important output of the pre-test should be a fully developed work plan.

   If choices have to be made as to what to include in the pre-test, the following considertions may be helpful:

   • What difficulties do you expect in the implementation of your proposal? Think of possible sources of bias in data collection techniques and sampling and ethical issues you considered during the preparation of your plan for data collection (Module 12). Can some of these potential problems be overcome by adapting the research design?
   • Inexperience with a certain data-collection technique is also a reason to include it in the pre-test.
   • Which parts of your study will be most costly and time-consuming? Questionnaires used in large surveys, for example, should always be tested. If many changes are made, the instruments should be pre-tested again. If an interview guide or questionnaire has been translated into a local language, the translated version should be pre-tested.

Who should be involved in the pre-test or pilot study?

• The research team, headed by the principal investigator.
• Any additional research assistants or data collectors that have been recruited.

How long should the pre-test or pilot study last?

The time required for a pre-test or pilot study will be determined by a number of factors:

• The size and duration of the research project. (The longer the study will take, the more time you might reserve for the test run.)
• The complexity of the methodology used in the research project.

Keep in mind that this is the last chance you will have to make adjustments which will help to ensure the quality of your fieldwork. If you have a 20 day field work period you might reserve at least 3-5 days for pre-testing your data collection tools, analysing the results of the pre-test, finalising your tools and elaborating the work plan.

Annex 14.1: Summary of points to assess during a pre-test or pilot study

Annex 14.2: Summary of possible fallacies in the design and implementation of
studies

As we have now gone through all steps of the study design, including the planning of data processing and analysis, it may be useful to summarise the critical points at which a researcher can go wrong:

• in the SELECTION of RESPONDENTS or study elements, and
• in the COLLECTION of data.

These potential errors should be reviewed while pre-testing your research methodology.

1. Errors in selection of respondents or study elements.

In the selection of respondents we may distinguish several major possibilities for error.

Too limited (or inappropriate) definition of the study population or use of incorrect sampling procedures, for example by:

• Studying registered patients only;
• Obtaining responses from male opinion leaders only (if one needs the opinion of the whole community);
• Choosing a sample because it is close to a road or in some other way easier to access (tarmac bias); or
• Conducting the study during only one season of the year (when results may be biased by not including other seasons or because access is difficult).

Errors in the assignment of research subjects to study groups in analytic and experimental studies:

• Defective matching in case-control studies;
• The inclusion of volunteers for study groups in cohort studies;
• Non-randomisation in experimental studies; or
• If randomisation is impossible: failure to develop a quasi-experimental design that corrects as much as possible for ‘rival explanations’.

Selective dropouts or non-response

Dropouts or subjects who do not respond to selected questions may represent a special category of respondents. If attrition is high or the rate of non-response excessive, results may be biased.

In cohort studies, follow-up of individuals can be difficult overtime. Bias in follow-up emerges if there is a differential drop out between those exposed to the risk and those not exposed.

2. Errors in data collection

We may obtain:

Invalid data, by applying indicators and measuring techniques or instruments which do not adequately measure what we want to measure.

Unreliable data due to:

• Variation in the characteristics of the research subject measured, as a consequence of the research;
• The use of unstandardised measuring instruments; or
• Differences between observers/interviewers in asking questions or observing.

Reliability of data collected is always required, but it is of crucial importance if we want to measure changes over time. If we find changes we must be sure that these are not caused by errors in our research methods that could have been prevented.

All the above-mentioned shortcomings may threaten the validity of your findings and conclusions.

Trainer’s Notes

Module 14: PRE-TESTING THE METHODOLOGY

It is important to note that well before this session (preferably after Module 10) the course manager should ask the groups what data collection tools they want to pre-test and on whom. Each research team, with the assistance of its facilitator, should decide how many interviews or observations will be conducted and should begin making arrangements for obtaining necessary copies of the instruments and other supplies needed. The course manager should make arrangements for all the groups (i.e., look for suitable sites to do the pre-test, inform (health) authorities and/or local leaders of the plans, ask for their consent, and arrange transport).

Introduction and discussion

• Discuss the concept and process of pre-testing or conducting a pilot study of the methodology, covering the questions posed in the module.
• Refer to Annex 14.1, ‘Summary of points to assess during a pre-test or pilot study.’ Using the annex, review briefly the important aspects of pre-testing covered in the presentation.
• Discuss the pre-test that will be undertaken during the workshop and make sure that each research team knows where it is going to do the pre-test and with whom it is going to meet. Explain how much time each team has for the preparation of the pre-test (Group Work I), the actual pre-test, its evaluation (group work II) and for reporting and discussion in plenary. Stress the importance of working in pairs during the pre-test, so that experiences can be shared.
• Emphasise that it is important to make notes on all observations during the pre-test so that they can be discussed afterwards.

Group work I

Ask the participants to meet in their working groups to design the pre-test for their project.

Pre-test field exercise

If necessary, an instruction sheet should be prepared for the field exercise including information such as:

• How the field exercise will be organised, i.e.:

  — Where each working group will go;

  — What pairs of participants will work together;

  — What formalities need to be observed with community leaders, directors of health facilities, and respondents;

  — What explanation should be given concerning the purpose of the pre-test and whether any feedback will be given to those participating as respondents;

  — How many interviews or observations should be conducted;

  — Time available for the exercise, etc.

• Points that should be assessed during the pre-test. (These could include some or all of the points listed in Annex 14.1.)
• When and where the working groups should reassemble after the field exercise for the group work II session.

Group work II

Arrange for each group to meet after its members have come back from the field to discuss and analyse their experiences and to revise their data-collection tools and possibly other aspects of the research methodology. Ask each group to prepare a short report of its main findings and conclusions.

Plenary

One member of each group should report in plenary on the main findings and conclusions of the pre-test.

Document(s) 20 of 27