Studies of Studies and Samples of 1
In this section, let’s consider two research methods that might feel as different from each other as possible: meta-analysis, which often brings together thousands of data points (or more!), and single-subjects designs, also called “n of 1″ studies because the sample size is, you guessed it, one. Both have their place in research, and bring some of the data collection techniques we’ve already discussed into interesting territory.
Meta-analysis
Existing data can be used in a number of ways; as we’ve discussed before, you can research in an archive, analyze the content of published (and unpublished) work, and examine the evidence left behind by human behavior. As a further example of this, let’s talk about a unique method for analyzing data across multiple studies. Sometimes called a study-of-studies, a meta-analysis uses prior research studies as the data in their own, new, analysis. A researcher using meta-analysis takes all those other findings, enters the data from those studies into a dataset, and statistically analyzes them to come up with an overall result that has much higher power than the other results did alone due to the increased sample size. These studies are different than papers like “systematic literature reviews” or even a content analysis because they literally combine the results of the prior studies to generate new statistical results, rather than just studying what’s been written before and synthesizing or analyzing the information that way. As we’ve discussed before, one research study alone can only tell us so much about a question or problem. Meta-analyses take many small drops and apply their techniques to get a better sense of the big picture on a topic.
The kinds of issues that occur when collecting data from individuals can also occur when collecting data from existing documents, especially in the case of meta analysis. The most obvious problem that might cross both techniques is selection bias. When looking for information from existing data, be careful to understand what’s being contributed as well as what’s not. This is know as the file-drawer problem (also called publication bias) because when meta-analytic researchers are gathering the studies that they want to use as data, it’s likely they’ll only get part of all the research done on a topic if they focus only on studies that have been published. Research that didn’t “work” for some reason – the results weren’t exciting, or didn’t align with the hypothesis, or didn’t go all the way through publication for one reason or another – is likely locked up in someone’s file drawer somewhere (or more likely languishing on a cloud server nowadays). This is an issue for any existing data design, but especially when the goal is to collect all the data on a given question to re-analyze them all together, we can’t rely only on what a researcher can find in published sources. Often, researchers conducting meta-analyses will reach out to other researchers in the field to help find the data in file drawers, by having researchers who study the topic share even their unpublished data, and they can also conduct sophisticated analysis to estimate what data are missing from the picture. This can go a long way in helping make meta-analyses as strong as possible.
Let’s take a moment to examine how one meta-analysis dealt with the file drawer problem and what it meant for their results. When Dr. Elizabeth Fawcett and colleagues (2010) wanted to study how well premarital education programs worked for improving couple satisfaction and communication, they used a meta-analysis to combine the results from all the studies they could find on the topic. Their search (described in detail in a companion article, Hawkins et al., 2008), included not only general research databases, but also the reference lists of other meta-analysis and literature reviews. To find those studies that might not have been published, the authors searched a dissertation database (as, if you’ll recall from our discussion of types of literature, dissertation studies are not required to be peer-reviewed to be published in their university’s database) and also “made extensive efforts over the course of 2 years at national conferences and through e-mail to contact researchers and practitioners to find unpublished (and in-press) reports” (p. 725). By searching thoroughly and not assuming that all the data they needed would be published, they were able to collect a number of sources they might not otherwise have found. Because of this, they were able to show that including the unpublished studies mattered a great deal for the conclusions to be taken!
Basically, using existing data for our research means that we must acknowledge that the data we find exist for a reason, and that there are likely data that don’t exist but should and would change our interpretations if they did. You might have heard the phrase “history is written by the victors,” and it can be the case here too that the documents and other records we find were often produced by only certain groups or individuals. Even if it was produced, some data don’t survive as long as others, which can also contribute to missing perspectives. Asking yourself what’s missing is a useful strategy to take when evaluating existing data designs, especially meta-analyses.
Single-subjects design