Quasi-Experimental Designs
Quasi-experimental Designs and Experimental Look-a-likes
Sometimes a true experiment is not appropriate or is not possible; you can surely think of plenty of “independent” variables that can’t be manipulated, or at least shouldn’t be. Time, funding, and ethics may limit a researcher’s ability to conduct a true experiment. Researchers that wish to conduct a true experiment in medical science or social services may be required to deny necessary treatment to clients, which is violates professional ethics. Even research projects that do not involve administering necessary medications or treatments may limit the researcher’s ability to conduct a classic experiment. Just because we can’t assign people to experience different levels of those variables doesn’t mean we can’t study what happens to folks who naturally experience differences. When true experiments are not possible, researchers often use quasi-experimental designs.
Quasi-experimental designs are similar to true experiments, but they lack random assignment to experimental and control groups. The most basic of these quasi-experimental designs is the nonequivalent comparison groups design (Rubin & Babbie, 2017). The nonequivalent comparison group design resembles the classic experimental design, but it does not use random assignment. In many cases, the groups may already exist. For example, a researcher might conduct research at two different agency sites, one of which receives the intervention and the other does not. The researcher does not need to assigned participants to treatment or comparison groups because the groupings already existed prior to the study. While this method is more convenient for real-world research, researchers cannot be sure that the groups are comparable. Perhaps the treatment group has a characteristic that is unique, such as higher income or different diagnoses, that make the treatment more effective.
Quasi-experiments are particularly useful in social welfare policy research. Social welfare policy researchers like me often look for natural experiments, or situations in which comparable groups are created by differences that already occur in the real world. For example, Stratmann and Wille (2016) were interested in seeing how the state healthcare policy called Certificate of Need effected the quality of hospitals. The researchers clearly could not assign states to adopt one set of policies or another, so they used hospital referral regions (the areas from which hospitals draw their patients) that spanned across state lines. Since the hospitals were in the same referral region, the researchers could reasonably assume that the client characteristics were similar. In this way, they could classify patients in experimental and comparison groups without affecting policy or telling people where to live.
The issue with pre-determined groups is always self-selection (also called selection effects), which is the idea that those who are in the separate groups are different from each other in other ways as well. The question of marriage and health is a great example of this: if we simply compare married people to not-married people and look at their health, are we really capturing just the differences due to marriage? No way! We know that marriage is predicted by lots of things that also likely influence health: education, age, parental stability, etc.
There are many important examples of policy experiments that use random assignment, namely the Oregon Medicaid experiment. Oregon’s Medicaid waitlist was so long that state officials decided to conduct a lottery to see which individuals from the wait list would receive the service (Baicker et al., 2013). Researchers used the lottery as a natural experiment that included random assignment: people selected to be a part of Medicaid were the experimental group and those on the wait list were in the control group. There are some practical complications with using people on a wait list as a control group, including the possibility of people from the wait list being accepted into the program while data is still being collected. Natural experiments aren’t a specific kind of experiment like quasi- or pre-experimental designs. Instead, they are like a feature of the social world that allows researchers to use the logic of experimental design to investigate the connection between variables.
Another approach to assign participants to experimental and comparison groups in a quasi-experimental design is through matching. Researchers should think about what variables are important in their study, particularly demographic variables or attributes that might impact their dependent variable. Individual matching involves pairing participants with similar attributes. When this is done at the beginning of a true experiment, the matched pair is split—with one participant going to the experimental group and the other to the control group. We talked about that method in the section before. In contrast, an ex post facto control group is when a researcher matches individuals after the intervention is administered to some participants. In this case, each member of the pair was not randomly assigned to the group they’re in. Finally, researchers may engage in aggregate matching, in which the comparison group is determined to be similar on important variables but again, is not assigned to be in the comparison group.
Though it is beyond the scope of this textbook to describe the plethora of quasi-experimental designs, one more design is worth mentioning. The time series design uses multiple observations before and after an intervention. In some cases, experimental and comparison groups are used. In other cases where that is not feasible, a single experimental group is used. By using multiple observations before and after the intervention, the researcher can better understand the true value of the dependent variable in each participant before the intervention starts. Additionally, conducting multiple observations after the intervention allows the researcher to see whether the intervention had lasting effects on participants. Time series designs are similar to single-subjects designs, which we will discuss later.
Pre-Experimental Designs
When true experiments and quasi-experiments are not possible, researchers may turn to a pre-experimental design (Campbell & Stanley, 1963). Pre-experimental designs are called such because they often happen before a true experiment is conducted. Often, researchers want to see if their interventions will have an effect on a small group of people before they seek funding and dedicate time to conduct a true experiment. Pre-experimental designs, thus, are usually conducted as a first step towards establishing the evidence for or against an intervention. However, this type of design comes with some unique disadvantages, which we’ll describe as we review the pre-experimental designs available.
If we wished to measure the impact of a natural disaster like a major hurricane, then we might conduct a pre-experiment by identifying an experimental group from a community that experienced the hurricane and a control group from a similar community that had not been hit by the hurricane. This study design, called a static group comparison, has the advantage of including a comparison group that did not experience the stimulus (in this case, the hurricane). Unfortunately, it is difficult to be sure that the groups are truly comparable because the experimental and control groups were determined by factors other than random assignment. Additionally, the design would only allow for posttests, unless one were lucky enough to be gathering the data already before the hurricane in question hit. As you might have guessed from our example, static group comparisons are useful in cases where a researcher cannot control or predict whether, when, or how the stimulus is administered, as in the case of natural disasters.
In cases where the administration of the stimulus is quite costly or otherwise not possible, a one-shot case study design might be used. In this instance, no pretest is administered, nor is a control group present. In our example of the study of the impact of a hurricane, let’s say we wanted to examine the effects of a specific storm, like 2017’s Hurricane Maria, a researcher using this design would test the impact of Maria only among a community that was hit by the hurricane and would not seek a comparison group from a community that did not experience the hurricane. Researchers using a one-shot case study design must be extremely cautious when making claims about the effect of the stimulus, though the design could be useful for exploratory studies that aim to testing one’s measures or the feasibility of further study.
Finally, if a researcher cannot identify a sample that is large enough to split into control and experimental groups, or if they simply do not have access to a control group, they may use a one-group pre-/posttest design. In this instance, pre- and posttests are both taken, but there is no control group to compare the experimental group to. We might be able to study of the impact of Hurricane Maria using this design if we’d been collecting data on the impacted communities prior to the hurricane. We could then collect similar data after the hurricane. Applying this design involves a bit of serendipity and chance. Without having collected data from impacted communities prior to the hurricane, we would be unable to employ a one- group pre-/posttest design to study Hurricane Maria’s impact.
The preceding examples where we considered studying the impact of hurricanes highlight that experiments do not necessarily need to take place in a controlled lab setting. In fact, many applied researchers rely on experiments to assess the impact and effectiveness of various programs and policies. You can probably think of ways this could be done, such as by randomizing the clients in a social services office to receive a new method of assistance or business-as-usual, or by examining how a county that institutes a new rule differs from a neighboring county that does not.
References
Baicker, K., Taubman, S. L., Allen, H. L., Bernstein, M., Gruber, J. H., Newhouse, J. P., … & Finkelstein, A. N. (2013). The Oregon experiment—effects of Medicaid on clinical outcomes. New England Journal of Medicine, 368(18), 1713-1722.
Campbell, D., & Stanley, J. (1963). Experimental and quasi-experimental designs for research. Chicago, IL: Rand McNally.
Rubin, C. & Babbie, S. (2017). Research methods for social work (9th edition). Boston, MA: Cengage.
Stratmann, T. & Wille, D. (2016). Certificate-of-need laws and hospital quality. Mercatus Center at George Mason University, Arlington, VA. Retrieved from: https://www.mercatus.org/system/files/mercatus-stratmann-wille-con-hospital-quality-v1.pdf
