14 Experiments

Experiments are an excellent data collection strategy for those wishing to observe the consequences of specific actions or stimuli. Most commonly a quantitative research method, researchers in criminal justice, psychology, and other social science disciplines use experiments to examine various research questions. Even if you never plan to experiment, understanding how they are administered will help you evaluate the experiments you might read about. Students in research methods classes use the term experiment to describe all kinds of empirical research projects. In social scientific research, the term has a unique meaning and should not be used to describe all research methodologies.

What is an Experiment?

An experiment is a method of data collection designed to test hypotheses under controlled conditions. Experimental research can be conducted in laboratory or field settings. Laboratory experiments are conducted in artificial settings created by the research team. Field experiments are conducted in the real world, such as in a real agency or organization. Regardless of where a researcher conducts their experiment, basic terminology applies to all types of experiments. We’ll discuss that terminology here, focusing on different types of experimental designs.

In experimental research, some participants receive an experimental stimulus, and others receive no such stimulus. Social science researchers use all sorts of experimental stimuli, such as short written passages, images, videos, and even sounds or smells. The group of participants who receive the stimulus is called the experimental group, and the group of participants who do not receive the stimulus is called the control group. Researchers measure the effects of the stimulus by administering surveys or conducting interviews before and after introducing the stimulus to the experimental group. The measurements they take before the stimulus are called pre-tests. The measurements they take after the stimulus are called post-tests.

Researchers using experimental designs must consider the roles of random selection and random assignment in their experiments. Random selection refers to choosing participants using a random sampling technique, which we discussed in Chapter 8. After sampling, experimental researchers should aim for random assignment if possible. Random assignment is the process of randomly assigning participants to experimental or control groups. This practice increases the chances that experimental and control groups are similar to each other before the researcher administers the stimulus.

Types of Experiments

Researchers use a few different types of experimental designs to test their hypotheses. These designs can be grouped into “true experiments” and “quasi-experiments.” Both types contain some combination of three key features: independent and dependent variables, pre-testing and post-testing, a stimulus, and experimental and control groups. The key difference, as we’ll discuss in more detail below, is that only true experiments use random selection and random assignment to form their experimental and control groups.

True Experiments

In general, true experiments contain independent and dependent variables, pre-testing and post-testing, and experimental and control groups, chosen and assigned using random selection and assignment techniques. Three common types of true experiments include the classic experiment, the Solomon four-group design, and the post-test-only control group design.

In a classic experiment, a researcher tests the effect of a stimulus by comparing two groups: one exposed to the stimulus (the experimental group) and another that does not receive the stimulus (the control group). In other words, the classic experiment tests the effects of an independent variable on a dependent variable. Because the researcher’s interest lies in the effects of an independent variable, they must measure participants on the dependent variable before and after the independent variable (or stimulus) is administered. Thus, pre-testing and post-testing are both important steps in a classic experiment.

Table 14.1 illustrates a classic experimental design. The “R” in front of each group denotes the researcher assigns participants to groups using random assignment techniques. Group 1 is the experimental group because everyone receives a pre-test, stimulus, and post-test. Group 2 is the control group because everyone receives only the pre-test and the post-test.

Table 14. 1 Classic Experimental Design

One example of experimental research can be found in Shannon K. McCoy and Brenda Major’s (2003) study of people’s perceptions of prejudice. In one portion of this study, all participants took a pre-test to assess their levels of depression. During the pre-test, the researchers found no significant differences in depression between the experimental and control groups. Participants in the experimental group then read an article suggesting that prejudice against their racial group is severe and pervasive; participants in the control group read an article suggesting that prejudice against a racial group other than their own is severe and pervasive. Upon measuring depression scores during the post-test period, the researchers discovered that people who had received the experimental stimulus (the article citing prejudice against their racial group) reported greater depression than those in the control group.

Thus, this research contained all key features of a true experiment: an independent variable (the reading), a dependent variable (depression), pre-tests and post-tests, and experimental and control groups. It’s a classic experiment because it tests the effects of a stimulus (the reading) on an outcome (depression), using pre-tests and post-tests of one experimental group and one control group.

The Solomon four-group design is a second type of true experiment. As in a classic experiment, the Solomon four-group design involves a control and an experimental group. However, the four-group design includes two additional groups: one that receives the stimulus and then takes the post-test, and another that does not receive the stimulus but does take the post-test. Table 14.2 demonstrates the Solomon four-group design. Once again, groups are randomly assigned. Groups 1 and 2 are the same as in the classic experiment. Group 3 receives the stimulus and post-test, and Group 4 receives only the post-test.

Table 14. 2 Solomon Four-Group Experimental Design

The post-test-only control group is considered a true experimental design, though it lacks any pre-tests. In this design, the researcher randomly assigns participants to experimental and control groups, administers the stimulus, and measures respondents on the dependent variable. This type of design skips the pre-test phase of the experiment with the assumption that if the researcher has randomly assigned people to experimental and control groups, then no pre-test is necessary. Table 14.3 illustrates the post-test-only control group design.

Table 14. 3 Post-Test-Only Control Group Experimental Design

Notice that neither group receives the pre-test. Group 1 (the experimental group) receives the stimulus and the post-test, and Group 2 (the control group) receives only the post-test.

Quasi-Experiments

Quasi-experimental designs are almost identical to true experimental designs but lack the key ingredient of random assignment. Lack of funding, time constraints, or limitations of the research topic or question, may all constrain researchers’ ability to randomly select and assign participants into groups. For instance, when a colleague and I wanted to test the effects of a curriculum change in a local police training program, organizational constraints meant we could not randomly assign students to experimental and control groups. Instead, we chose entire training cohorts to receive the new curriculum and compared their post-test results to those of other cohorts that had received the traditional curriculum.

The lack of random assignment increases the chances that the experimental and control groups will be non-equivalent groups, groups with important differences that might impact the study’s findings. For example, in our study of the effectiveness of a curriculum change, the cohorts we chose might have had more prior knowledge in the area of the curriculum we were testing than the control groups. While non-equivalence introduces various potential issues with experimental studies, sometimes researchers have no choice but to use a quasi-experimental design.

Many true experimental designs can be converted to quasi-experimental designs by omitting random assignments. For instance, the quasi-equivalent version of a classic experiment is called a non-equivalent groups design. Table 14.4 illustrates a non-equivalent groups design. If you compare this design to that depicted in Table 14.1, you’ll see only one difference: the “R” before the groups have been changed to an “N.” This change signifies the non-random assignment of groups. Apart from that difference, everything else remains the same as both groups receive the pre-test and post-test, and Group 1 (the experimental group) receives the stimulus between the two testing periods.

Table 14. 4 Non-Equivalent Groups Design

Another type of quasi-experiment is a version of the post-test-only control group design discussed earlier. In the post-test-only non-equivalent groups design, the researcher administers a stimulus to the experimental group and then uses post-tests of the experimental and control groups to measure the effects of the stimulus.

Table 14.5 illustrates this type of experimental design. If you compare this table to Table 14.3, you’ll see that once again, the only difference is the “N” before the groups, indicating non-random assignment.

Table 14. 5 Post-Test-Only Non-Equivalent Groups Design

This is the type of quasi-experimental design we used in our research on the effects of a curriculum change in police training: We exposed a few cohorts to the new curriculum, obtained post-test results, and then compared those results to other cohorts that had not received the new curriculum. Because we did not randomly assign participants to groups, it was a quasi-experimental design.

Strengths and Weaknesses of Experimental Research

As with other research methods, experiments have strengths and weaknesses that researchers must consider while designing experimental research. One strength of experiments, particularly laboratory experiments, is that the researcher has substantial control over the conditions to which participants are subjected. Experiments are also generally easier to replicate than studies that use other data collection methods. Such replication is essential for determining whether the findings of an experiment hold across multiple people and groups.

For social scientists, experiments also have the drawback of being artificial. While this is especially true for laboratory experiments, even field experiments do not fully reflect the real world. A drawback specific to field experiments is that the researcher has less control over the stimulus and other conditions that might impact participants’ behavior. When the conditions of an experiment don’t match those of the world outside of the boundaries of the experiment, researchers run into problems with the generalizability of their findings. For example, in the case of the research study mentioned earlier about prejudice, can we say for certain that the stimulus applied to the experimental group resembles the stimuli that people are likely to encounter in their lives outside of the lab? Will reading an article on prejudice against one’s race in a lab have the same impact it would outside of the lab? Asking these kinds of questions doesn’t mean that experimental research cannot be valid, but experimental researchers must always recognize and address issues of generalizability that can occur with experiments.

Another potential concern with experiments deals with how confident a researcher can be that the stimulus, rather than some other factor, produced the observed effect. Other factors that might create an observed effect could be conditions of the experiment that the researcher had not considered or changes in participants over time.

In sum, the strengths and weaknesses of experimental research designs include a researcher’s control over conditions, and ease of replication by other researchers. Some of the weaknesses of this method include the artificiality of the setting and issues with generalizability and confidence that the stimulus produced the outcome. Table 14.6 summarizes the strengths and weaknesses of the experimental research design.

Table 14. 6 Strengths and Weaknesses of Experimental Research

Summary

• Experiments are quantitative data collection methods designed to test hypotheses under controlled conditions. All experiments involve some combination of independent and dependent variables, pre-tests, a stimulus, post-tests, and experimental and control groups.
• In true experiments, researchers randomly select people for their sample and randomly assign participants to experimental and control groups. In quasi-experiments, researchers do not use random selection or assignment to choose participants and assign them to groups.
• An experimental group is a group that receives some stimulus or treatment. A control group is a group that does not receive a stimulus or treatment.
• Some types of true experiments include the classic experiment, the Solomon four-group design, and the post-test-only control group design. Quasi-experimental designs include non-equivalent groups design and the post-test-only non-equivalent groups design.
• The benefits of experimental research methods include the ability to control conditions to test hypotheses and the ease with which other researchers can replicate the research. The drawbacks include the artificiality of the setting or stimulus, a potential lack of generalizability, and uncertainty about whether the stimulus, rather than some other factor, caused the outcome.

Discussion Questions

1. Why might a researcher need to use a quasi-experimental design rather than a true experiment? What are some of the downsides to quasi-experimental designs that would not be present in a true experimental design?
2. Compare and contrast the main features of the three types of true experimental designs covered in this chapter. How are the three types similar and different? Why do you think there are so many types of true experimental designs?
3. What are some ways that researchers might overcome some of the weaknesses of survey research?

Works Cited in Chapter 14

McCoy, S. K., & Major, B. (2003). Group identification moderates emotional response to perceived prejudice. Personality and Social Psychology Bulletin, 29, 1005–1017.