Document(s) 12 of 27

Flowchart: Steps in the development of a health systems research proposal

NB: Development of a research process is a cyclical process. The double-headed arrows indicate that the process is never linear.

Module 9: STUDY TYPES

1. Introduction
2. Overview of study types
3. Deriving valid and reliable conclusions

I. INTRODUCTION

Depending on the existing state of knowledge about a problem that is being studied, different types of questions may be asked which require different study designs. Some examples are given in the following table:

Table 9.1: Research questions and study types

The type of study chosen depends on:

• the type of problem;
• the knowledge already available about the problem; and
• the resources available for the study.

When investigating health management problems, such as overcrowding in a hospital out-patient department or shortage of drugs at PHC level, a good description of the problem and identification of major contributing factors often provides enough information to take action.

When exploring more complicated management or health problems, we usually want to go further and determine the extent to which one or several independent variables contribute to the problem (for example, the contribution of low-fibre diet to cancer of the large intestine). For these types of problems more rigorous analytical or experimental studies will have to be conducted before we decide on appropriate interventions.

II. OVERVIEW OF STUDY TYPES

Several classifications of study types are possible, depending on what research strategies are used. The table below categorises studies, based on the combination of research strategies they use, including:

1. Non-intervention studies in which the researcher just observes and analyses researchable objects or situations but does not intervene; and
2. Intervention studies in which the researcher manipulates objects or situations and measures the outcome of his manipulations (e.g., by implementing intensive health education and measuring the improvement in immunisation rates.)

NON-INTERVENTION STUDIES

We will first concentrate on non-intervention studies and their use in health systems research. We will discuss:

• Exploratory studies
• Descriptive studies
• Comparative (analytical) studies

1. Exploratory studies

For example:

A national Acquired Immunodeficiency Syndrome (AIDS) Control Programme wishes to establish counselling services for Human Immunodeficiency Virus (HIV) positive and AIDS patients, but lacks information on specific needs patients have for support. To explore these needs, a number of in-depth interviews are held with various categories of patients (males, females, married, single) and with some counsellors working on a programme that is already under way.

When doing exploratory studies we describe the needs of various categories of patients and the possibilities for action. We may want to go further and try to explain the differences we observe (e.g., in the needs of male and female AIDS patients) or to identify causes of problems. Then we will need to compare groups.

In HSR, small-scale studies that compare extreme groups are very useful for detecting management problems. We could, for example, compare:

• Two district health teams (DHT) that are functioning well and two that do not function satisfactorily, in order to detect the possible reasons for bottlenecks in the functioning of the district health teams; *
• One community with high and another with low participation in health activities, in order to identify factors that contribute to community participation;
• 20 mothers who delivered in a maternity and 20 who delivered at home, in order to identify possible reasons for the low percentage of supervised deliveries.

Exploratory studies gain in explanatory value if we approach the problem from different angles at the same time. This is called triangulation. In a study that is looking for causes of the low percentage of supervised deliveries, it may be very useful to include observations and interviews with health staff in the maternity centres that should serve the mothers in question and interviews with their supervisors, as well as with the mothers themselves. In this manner, information from different independent sources can be cross-checked.

For some management problems such a ‘rapid appraisal’ may provide sufficient information to take action. Otherwise, a larger, more rigorous comparative study will have to be developed to test differences between groups with respect to various independent variables.

* Such small-scale studies may be called exploratory case studies if they lead to plausible assumptions about the causes of the problem and explanatory case studies if they provide sufficient explanations to take action (Yin, 1984).

2. Descriptive studies

Descriptive studies can be carried out on a small or larger scale.

(1) Small scale, descriptive case studies

Descriptive case studies describe in-depth the characteristics of one or a limited number of ‘cases’. A case may be, for example, a patient, a health centre, or a village. Such a study can provide quite useful insight into a problem. Case studies are common in social sciences, management sciences, and clinical medicine. For example, in clinical medicine the characteristics of a hitherto unrecognised illness may be documented as a case study. This is often the first step toward building up a clinical picture of that illness.

However, if one wishes to test whether the findings pertain to a larger population, a more extensive, cross-sectional survey has to be designed.

(2) Large scale, cross-sectional surveys

Cross-sectional surveys aim at describing and quantifying the distribution of certain variables in a study population at one point of time. They may cover, for example:

• Physical characteristics of people, materials or the environment, as in

  — prevalence surveys (of bilharzia, leprosy, HIV), or

  — evaluation of coverage (of immunisation, latrines, etc.),

• Socio-economic characteristics of people such as their age, education, marital status, number of children and income,
• The behaviour or practices of people and the knowledge, attitudes, beliefs, opinions which may help to explain that behaviour (KAP studies), or
• Events that occurred in the population.

Cross-sectional surveys cover a selected sample of the population. If a cross-sectional study covers the total population it is called a census.

A cross-sectional survey may be repeated in order to measure changes over time in the characteristics that were studied. The surveys may be very large, with hundreds or even thousands of study units. In these cases only a limited number of variables will usually be included, in order to avoid problems with analysis and report writing. If cross-sectional surveys are smaller they can be more complex. They may include all the elements just mentioned. Small surveys can reveal interesting associations between certain variables, such as between having tuberculosis and socio-economic status, sex, and ways of coping.

Researchers often go further and will combine a description of the study population with a comparison of a number of groups within that population (see below). Such combinations are very common, and thus the distinctions between descriptive and comparative studies are sometimes quite fuzzy.

3. Comparative or analytical studies

Three commonly used types of analytical studies will be discussed here:

Figure 9.1: Types of analytical studies

(1) Cross-sectional comparative studies

Many cross-sectional surveys focus on describing as well as comparing groups.

For example, a survey on malnutrition may wish to establish:

• The percentage of malnourished children in a certain population;
• Socio-economic, physical, political variables that influence the availability of food;
• Feeding practices; and
• The knowledge, beliefs, opinions that influence these practices.

The researcher will not only describe these variables but, by comparing malnourished and well-nourished children, he will try to determine which socio-economic, behavioural and other independent variables may have contributed to malnutrition.

In any comparative study, one has to watch out for CONFOUNDING or INTERVENING variables. (Please look at Module 8 for examples and discussion, as well as to the next page, and Module 26).

(2) Case-control studies

Figure 9.2: Diagram of a case-control study*

For example, in a study of the causes of neonatal death, the investigator will first select the ‘cases’ (children who died within the first month of life) and ‘controls’ (children who survived their first month of life). (S)he then interviews their mothers to compare the history of these two groups of children, to determine whether certain risk factors are more prevalent among the children who died than among those who survived.

As with a cross-sectional comparative study, the researcher has to control for CONFOUNDING VARIABLES. In case-control studies, this may be done to some extent beforehand, by MATCHING the groups for expected confounding variables. Matching means taking care that the cases and controls are similar with respect to the distribution of one or more potentially confounding variables. However, we cannot then look at the effect of the matched variable as a risk factor because we have made the cases and controls exactly the same with respect to that variable.

For example, in a study on causes of malnutrition in children-3 years you may match the welland the malnourished on age, because this factor may influence many other variables influencing your problem (e.g. time of weaning, time of teething, which are both related to diarrhoea and consequently to malnutrition). But you will not match them on economic status of parents, as you do want to know whether poverty influences malnutrition.

In general you will only match for strong confounders (such as age) that you cannot properly control by stratification during data analysis, unless you double or treble the number of informants. You also match for potentially confounding variables such as location/source of origin, as these can influence many other potential confounders (e.g. ethnic group, religion, economic status), some of which you may not even expect beforehand.

* Adapted from WW Holland et al., eds. (1985) Oxford Textbook of Public Health, Volume 3: Investigative Methods in Public Health. Oxford: Oxford University Press.

(3) Cohort studies

A well-known example of a cohort study is the study by Doll and Hill (1950) of smokers and non-smokers that was conducted among doctors to determine the importance of smoking as a risk factor for developing lung-cancer.

A study may start with one large cohort. After the study starts, the researchers determine who is exposed to the risk factor (e.g., smoking) and who is not, and follow the two groups over time to determine whether the study group (of smokers) develops a higher prevalence of lung cancer than the control group. If it is not advisable to select one cohort (for example, because only few people are affected by the risk factor, which necessitates a very large sample), two cohorts may be chosen, one in which the risk factor is present (study group) and one in which it is absent (control group). In all other respects the two groups should be as alike as possible.

The control group should be selected at the same time as the study group, and both should be followed with the same intensity.

Figure 9.3: Diagram of a cohort study

Uses and limitations of different types of analytical studies

You may use any of the three types of analytical studies (cross-sectional comparison, case-control or cohort) to investigate possible causes of a problem.

For example, if you assume there is a causal relationship between the use of a certain water source and the incidence of diarrhea among children under five in a village with different water sources:

• You can select a group of children under five years and check at regular intervals (e.g., every two weeks) whether the children have had diarrhoea and how serious it was. Children using the suspected water source and those using other sources of water supply will be compared with regard to the incidence of diarrhea (cohort study).
• You can also conduct a case-control study. For example, you may compare children who present themselves at a health centre with diarrhea (cases) during a particular period of time with children presenting themselves with other complaints of roughly the same severity, for example acute respiratory infections (controls) during the same time, and determine which source of drinking water they had used.
• In a cross-sectional comparative study, you could interview mothers to determine how often their children have had diarrhea during, for example, the past month, obtain information on their source of drinking water, and compare the source of drinking water of children who did and did not have diarrhoea.

Cross-sectional comparative studies and case-control studies are usually preferred to cohort studies for financial and practical reasons. However, cohort studies are stronger in establishing causal relationships because confounding variables are to a large extent eliminated. If the study is well designed, the ‘confounders’ are equally distributed among the cases and controls. Experimental studies have the same advantage as cohort studies.

INTERVENTION STUDIES

In intervention studies, the researcher manipulates a situation and measures the effects of this manipulation. Usually (but not always) two groups are compared, one group in which the intervention takes place (e.g. treatment with a certain drug) and another group that remains ‘untouched’ (e.g. treatment with a placebo).

The two categories of intervention studies are:

• experimental studies and
• quasi-experimental studies.

1. Experimental studies

An experimental design is a study design that gives the most reliable proof for causation.

Figure 9.4: Diagram of an experimental study

A number of experimental study designs have been developed. These are widely used in laboratory settings and in clinical settings. For ethical reasons, the opportunities for experiments involving human subjects are restricted. However, randomised control trials of new drugs are common.

For example, a researcher plans to study the effect of a new drug. (The drug has already been tested extensively on animals and has been approved for trial use.) He plans to include 300 patients in the study who are currently receiving the standard treatment for the same condition for which the new drug has been designed. He explains the study to the patients asking their consent to be divided into two groups on a random basis. One group will receive the experimental drug while the other group will continue to receive the standard treatment. He makes sure that the medications are disguised and labelled in such a manner that neither the research assistant administering them nor the patient know which drug is used. (This is called a ‘double blind’ experiment.)

At community level, where HSR is frequently undertaken, we experience not only ethical but also practical problems in carrying out experimental studies. In real life settings, it is often impossible to assign persons at random to two groups, or to maintain a control group. Therefore, experimental research designs may have to be replaced by quasi-experimental designs.

2. Quasi-experimental studies*

One of the most common quasi-experimental designs uses two (or more) groups, one of which serves as a control group in which no intervention takes place. Both groups are observed before as well as after the intervention, to test if the intervention has made any difference. (This quasi-experimental design is called the ‘non-equivalent control group design’ because the subjects in the two groups (study and control groups) have not been randomly assigned.)

Figure 9.5: Diagram of a quasi-experimental design with two groups

* For a more detailed explanation of experimental and quasi-experimental designs and their advantages and disadvantages, one excellent reference is Campbell and Stanley’s Experimental and Quasi-Experimental Designs for Research (1963).

Example of a quasi-experimental study:

A researcher plans to study the effects of health education on the level of participation of a village population in an immunisation campaign. She decides to select one village in which health education sessions on immunisation will be given and another village which will not receive health education and serves as a control. The immunisation campaign will be carried out in the same manner in both villages. A survey will then be undertaken to determine if the immunisation coverage in the village where health education was introduced before the campaign is significantly different from the coverage in the ‘control’ village which did not receive health education. (Note: The study is quasi-experimental because the subjects were not assigned to the control or experimental groups on a random basis).

Another type of design that is often chosen because it is quite easy to set up uses only one group in which an intervention is carried out. The situation is analysed before and after the intervention to test if there is any difference in the observed problem. This is called a ‘BEFORE-AFTER’ study. This design is considered a ‘pre-experimental’ design rather than a ‘quasi-experimental’ design because it involves neither randomisation nor the use of a control group.

Figure 9.6: Diagram of a before-after study

Example of a ‘before-after’, pre-experimental study:

The out-patient clinic of hospital X is extremely crowded. Waiting times of over 5 hours for patients before they are attended to are not uncommon. The hospital management has a study carried out to analyse the bottlenecks and implements most of the recommendations made. Three months later, another study is done to check to what extent the bottlenecks have been solved and where further action is necessary.

This design is often used for management problems that pertain to one single unit (hospital, school, village). However, if the problems occur at a larger scale or if they might be influenced by other factors apart from the intervention during the trial, it is highly recommended that the design include both a study and a control group.

In the trial with health education on immunisation, for example, it would have been quite risky to work without a control group. Outside events (such as a health education campaign on immunisation by radio or other mass media) might have led to improved knowledge on immunisation in both the study group and the control group. (NB: The immunisation campaign by radio provides a so-called ‘rival explanation’ for your results.) If you had had just a study group and no control you might have concluded erroneously that all of the increase was due to your own intervention.

III. DERIVING VALID AND RELIABLE CONCLUSIONS

What are validity and reliability in research findings?

Validity means that your scientific observations actually measure what they intend to measure (your conclusions are true).

Reliability means that someone else using the same method in the same circumstances should be able to obtain the same findings (your findings are repeatable).

Reliability (repeatability) refers to the possibility to replicate (repeat) the observations and is related to the precision of the instrument used for scientific observations. Validity refers to the soundness of the observations and to the accurateness of the data collected by the research method/instrument.

Figure 9.7: Validity and reliability; graphic presentation of possible combinations

For example:

Four different teams of researchers set out to determine the body weights of three children whose true body weights were 10 kg, 15 kg and 20 kg respectively and obtained following four sets of results.

Team 1

The first set of results is not valid because the results do not represent the true body weights. They are not reliable because they are sometimes too high and sometimes too low, and the relative difference from the true body weight varies from child to child.

Team 2

The second set of results is not valid because the results again do not represent the true body weights. However, they are reliable because the results are too high by the same proportion (10%) for every child.

Team 3

The third set of results is fairly valid because the results are almost representing the true body weight. They are not reliable because two weights are too high and one is too low and the proportion by which they differ from the true body weight is different for each child.

Team 4

The fourth set of results is both valid and reliable because the results are the same as the true body weights, and these results have been obtained for every child.

How to deal with threats to validity and reliability

At various stages of the research validity and reliability could be threatened:

• At the moment of the selection of the study type and design. You should not start with a big survey when your knowledge of the situation and problem(s) is superficial, but always first do an exploratory study. Otherwise validity and reliability will be limited. Distortion may also occur during sampling or due to selectivity in assigning different subjects into various groups. (See Modules 9 and 11 part III.)
• At the level of the data collection (related to the instrument): the instrument itself may be unreliable; bias (distortion) may occur at various stages of data collection. (See Modules 10A part III, 10B part V.)
• At the level of the analysis of the data collected: confounding variables or events that disturbed your study design, and unnoticed weaknesses in study type and data collection may lead to misleading conclusions. (See Modules 9, 26.)

Examples of threats to validity:

1. Confounding factors

Example:

You might find that children who have had pre-school education subsequently perform better in primary school. Can you conclude that pre-school education leads to better school performance?

Rival or alternate explanations include:

• Educational and income level of parents may be contributing to both pre-school education and school performance; and
• Educational and income level of parents may, through availability of education toys in the home, television etc., influence learning performance in pre-school as well as primary school.

Education and income are therefore confounding factors.

2. History

Unexpected factors beyond your control might have produced the same effect as the intervention you were studying, thereby making it impossible for you to know whether it was your intervention that produced the impact.

Example:

A well-known example is when a certain agency that had designed a health education program for early detection of breast cancer designed a study to test the effectiveness of the program by studying the increase in the proportion of women who reported doing self-examination of breasts. However, while the study was in progress the President’s wife developed breast cancer and she appeared widely on mass media to advise women on early detection of breast cancer.

3. Differential subject loss in various groups

The type of subjects who drop out of your study or control groups may be related to some of the characteristics you are studying.

Example:

You are studying the effectiveness of a ‘weight watchers’ program by comparing the average weight loss in the ‘weight watchers’ group with that of a control group. However, a number of women in the ‘weight watchers’ group found the program too demanding and have dropped out.

4. Selectivity (or bias) in assigning subjects to various groups

Example:

You intend to study whether a programme on ‘how to stop smoking’ will be effective in helping the smokers in your hypertension clinic. Therefore you invite those who would like to attend to register themselves. You plan to compare the percentage who stop smoking among those who attend the programme with those who do not. However, it is likely that those who register themselves are those who are strongly motivated to stop smoking while those who are not motivated do not join the program (Also see Module 11).

Strategies to deal with threats to validity

1. Triangulation. Approaching a research problem from different angles (e.g., by selecting complementary study populations or using different research techniques at the same time) (see Modules 9 and 1 0).
2. Control group. Observing a control group who is not exposed to the risk factor or intervention reduces threats due to unexpected and confounding factors.
3. Appropriate sampling procedures and assignment of subjects to research groups. This reduces threats due to selectivity (see Module 11).
4. Before and after measurements. This allows us to assess whether there has been selectivity as well as differential loss of subjects. If there has been an inevitable loss of subjects, it may enable assessment of the dropouts to determine whether they had peculiar characteristics that distinguished them from those who did not drop out (see this module).
5. Unobtrusive methods of data collection and allowing adaptation time for subjects to get used to being observed or interviewed (see Module 10B).
6. Careful design and pre-testing of instruments, stressing the participation of health managers, staff and community members, reduce bias due to instrumentation (see Modules 10 and 14). Training of interviewers and standardisation of interview techniques and tools such as questionnaires are also important in reducing this bias.
7. Knowledge of the environment events enables the researcher to be sensitive to external events that could affect validity (i.e., history). In case of an expatriate researcher, local key informants can contribute a lot to the validity of the study.
8. Stratification and matching for confounding variables during the analysis of the results (see Module 26 on confounding).

Selection of study design

In selecting the design of the study, you have to consider the type of information you want to obtain and devise strategies to enable you to obtain that information.

REFERENCES

Abramson JH (1990, 4^th ed.) Survey Methods in Community Medicine. London: Churchill-Livingstone.

Beaglehole R, Bonita R, Kjellström T (1993) Basic Epidemiology. Geneva: World Health Organization.

Campbell DT, Stanley JC (1963) Experimental and Quasi-experimental Designs for Research. Chicago: Rand McNally.

Doll R and Hill AB (1950) Smoking and carcinoma of the lung: Preliminary report. Br. Med. J. 2:739-748.

Gordis L (1996) Epidemiology. Philadelphia USA: W.B. Saunders Company.

Holland W, Detels R, and Knox G (ed.) (1984-85) Oxford Textbook of Public Health, Volumes 1-4. Oxford: Oxford University Press.

Katzenellenbogen JM, Joubert G, Abdool Karim SS (1997) Epidemiology; A manual for South Africa. Capetown: Oxford University Press.

Kidder LH, Judd CM (1987) Research Methods in Social Relations. Hong Kong: CBS Publishing Japan Ltd.

Moser CA, Kalton G (1989, 2^nd ed.) Survey Methods in Social Investigation. Hants, UK: Gower Publishing Company.

Patton MQ (1990, 2^nd ed.) Qualitative Evaluation and Research Methods. Newbury Park, USA: Sage Publications.

Rose J, Barber DJP (1989) Epidemiologie for the uninitiated. Plymouth GB: Latimer Trend & Co/ British Medical Association.

Yin RK (1984) Case Study Research; Design and Methods. London, UK: Sage Publications.

Vaughan JP, Morrow RH (1989) Manual of Epidemiology for District Health Management. Geneva: World Health Organization.

Trainer’s Notes

Module 9: STUDY TYPE

Introduction and discussion

It is helpful if participants read this module the evening before the presentation, so they are more familiar with the topic.

The goal of the module is to give participants an understanding of the major issues involved in choosing different research strategies rather than having them learn the various possible study types by heart.

Present Table 9.1 at the beginning of the module to illustrate basic questions that lead to the choice of different study types without getting into the details concerning each type. Then proceed with the detailed discussion of each study type. Repeat the presentation of Table 9.1 at the end of the module, summarising the different study types that are possible.

It should be stressed that unless all variables to be investigated are clearly defined, small-scale studies are preferable to large-scale studies. A combination of study types (triangulation) can be considered if some variables still have to be explored (e.g., by open-ended questions) whereas other well-defined variables need to be measured on a larger scale (e.g., degree of utilisation of services).

It is advisable to proceed with the presentation of Module 10A (Overview of data-collection techniques) before participants do their group work to choose the type(s) of study they will use for their research projects. After Module 10A, the participants can be asked to do an exercise which involves selecting a study type as well as data collection techniques for certain problems.

Document(s) 12 of 27