From the Editor |
How to critically read observational studies
Note: This is a part of a series discussing how to critically read the published scientific literature.
In observational studies, researchers
measure what is naturally happening on farms. Observational studies
are often used to describe the relationship between management
factors and either disease and production parameters.
The analysis of observational studies begins by describing the occurrence of morbidity, mortality, or productivity. Then the relationship between the outcome parameter and the putative causative factors is determined using appropriate statistical tests. Although we can determine association, it doesn't prove a causal relationship between the disease and the factor of interest; e.g., a management strategy. Consider a situation, for example, in which multisource pigs are being commingled in the nursery (the factor of interest) and there is a problem with E. coli diarrhea in the nursery (outcome). An observational study could prove whether the outcome (E. coli diarrhea) is associated with factor of interest (commingling) but can't prove that commingling caused the diarrhea. For this reason, factors found associated with outcome measurements during observational studies require further study in controlled field trials or laboratory studies. However, observational studies provide an excellent tool to identify important management factors and enable the researcher to determine the association between the outcome and one management factor while controlling for other management factors. Factors that are included in the analysis but that are not of primary interest are called "covariates." Also, because the study is conducted in the field, on commercial swine units, there is not a concern that the results of the study apply to the "real world."
There are three types of observational studies, determined by the selection process for including the farm or the animal in the study.
- Case-control studies are studies in which the researcher identifies herds that have the disease of interest and then matches these with herds without the disease. Case-control studies, usually used for rare diseases, are moderately easy to conduct, have a moderate ability to prove causation, and are moderately close to a real-life situation, but provide little control to the researcher.
- Cohort studies involve finding herds with the factor of interest and other herds without the factor of interest and then following these herds over time to determine whether or not they become affected by the disease. Cohort studies provide an excellent opportunity to prove causation because the factor is identified before the disease occurs. Although the studies are expensive and time consuming to conduct, the investigator has a moderate degree of control over the investigation and the results apply very well to real-life situations. Unfortunately, cohort studies are rare in the veterinary literature.
- Cross-sectional studies are the most common and least expensive of the observational study types. In these, a group of farms is randomly selected to participate in the study. On these farms, the factor of interest and putative factors are measured. Then the researcher determines which factors are most commonly found on positive versus negative farms. Cross-sectional studies provide little opportunity to prove causation because the research cannot determine which occurred first: the factor of interest or the disease. For example, a herd with a respiratory problem may institute a treatment program whereas a herd with no respiratory problem will not. The researcher may find that the treatment program is associated with having the respiratory disease. However, the results of the study are likely to be very relevant to the real world and so provide useful information to practitioners.
Interpreting observational studies
To critically interpret observational studies, one needs to evaluate a specific set of problems that are not as important in other study types. The first issue is the difference(s) between the study population and the reference population. The study population is the animals that were actually included in the study. The reference population is the population of animals to which the findings of a study can be validly generalized. The study population (the pigs you are working with) must be as similar as possible to the reference population for the results to be validly generalizable. In a cross-sectional study, the farms may be selected randomly. This increases the likelihood that the farms apply to a wide reference population.
Often the study farms are a convenience sample made of farms close to the researcher or volunteers willing to participate in the study. At times, the convenience sample is determined by the data required. For example, the study population may include only those producers who use computerized production records. In a case-control study, the study population may include only herds that have submitted animals to the diagnostic laboratory. If your clients use computerized records and use the services of a diagnostic laboratory, then the results of the study probably apply to their herds. However, producers who do not use records may differ systematically from those that do use records. Therefore the associations found on one type of farm may not be generalizable to other farms.
The case definition and the factors of interest in observational studies must be clearly stated to enable the reader to evaluate the study. The results will apply to your practice if you observe the same type of case and conduct a similar diagnostic work-up. For example, a study might mention postweaning mortality due to E. coli. This may involve edema disease in some parts of North America or sudden death and diarrhea due to K88 E. coli in other parts of the industry. Control farms (pigs) must be exposed to the same, thorough work-up as the case farms. One producer with sudden death in the nursery barn may ignore the problem whereas another producer may continue to submit pigs to the diagnostic laboratory until a diagnosis is made. If the researcher does not insist on equal investigation of case and control herds, there could be some disease-positive herds in the control group.
The outcome of interest needs to be measured in the same manner on all farms. If there is a subjective element to the outcome, then it must be measured in a "blind" manner. Blinding refers to withholding the status of individual animals so that the person who determines the outcome will evaluate all animals in the same manner. For example, in a diarrhea study, the postmortem evaluation may be more thorough for case pigs than for those from control herds.
Bias is of real concern in observational studies. Selection bias is the inclusion of farms that differ from the reference population and therefore the results apply only to that specific type of farms. For example, studies conducted on 4000-sow, multisite units may not apply to the 150-sow farrow-to-finish unit.
Confounding is the name given to the bias when an association is found between a factor and the outcome but the association is due to a third factor. In one farm analysis, I found an association between low farrowing rate and the use of artificial insemination. However, upon further investigation, I noticed that 90% of the gilts were bred by artificial insemination and all of the sows were bred naturally. We know that gilts typically have a lower farrowing rate than sows. Therefore, in this analysis we were not able to separate the effect of parity from the effect of artificial insemination on the farrowing rate. This relationship is called "confounding." Multivariate analysis is a very important analytic tool in observational studies as it enables the research to include possible confounding variables in the model.
Observational studies in swine production medicine often include a very large number of animals in the study. This may be necessary to determine the association between many putative factors and the outcome. However, as the sample size increases, the power to find a statistically significant association also increases. Therefore, it is important to separate statistical significance and biological significance. Observational studies report the strength of an association with either a relative risk or an odds ratio; the latter is used for case-control studies. As the odds ratio increases, the probability of biological significance increases.
Reference
Dohoo IR, Waltner-Toews D. Interpreting clinical research part III. Observational studies and interpretation of results. Compendium of Veterinary Continuing Education. 1985; 7(10):S605-S612.
--Cate Dewey