Evaluating Sources

It is likely that most of the resources you locate for your review will be from the scholarly literature of your discipline or your topic area. As we have already seen, peer-reviewed articles are written by and for experts in a field. They generally describe formal research studies or experiments with the purpose of providing insight on a topic. You may have located these articles through the databases mentioned before or through archival searching.

Generally, when we discuss the evaluation of sources, we are referring to the following aspects: accuracy, relevance, bias, reputation, currency, and credibility factors. These measures apply to all works, including books, ebooks, articles, websites, or blog postings to name a few. Before you include a source in your literature review, you should clearly understand what it is and why you are including it. Your use of sources and the quality of the evidence you start with will determine the quality of your final outcome – as the old saying goes, “garbage in, grabage out,” so be careuflul of what you put in! When evaluating a work for inclusion in, or exclusion from, your literature review, ask yourself a series of questions about each source.

1. 1. Is the information outdated? Is the source more than 5-10 years old? If so, it will not provide what we currently know about the topic–just what we used to know. Older sources are helpful for historical information, but unless historical analysis is the focus of your literature review, or the older source is important because it was one of the first studies on a topic, try to limit your sources to those that are current.
   2. How old are the sources used by the author? If you are reading an article from 10 years ago, they are likely citing material from 15-20 years ago. Again, this does not reflect what we currently know about a topic.
   3. Does the author have the credentials to write on the topic? Search the author’s name in a general web search engine. What are the researcher’s academic credentials? What else has this author written? Search by author in the databases and see how much they have published on any given subject.
   4. Who published the source? Books published under popular press imprints (such as Random House or Macmillan) will not present scholarly research in the same way as Sage, Oxford, Harvard, or the University of Washington Press. For grey literature and websites, check the About Us page to learn more about potential biases and funding of the organization who wrote the report. For journal articles, use the information described previously about assessing peer-review status, or look at listings of journals in your discipline to see if it’s a highly-ranked journal.
   5. Is the source relevant to your topic? How does the article fit into the scope of the literature on this topic? Does the information support your thesis or help you answer your question, or is it a challenge to make some kind of connection? Does the information present an opposite point of view, so you can show that you have addressed all sides of the argument in your paper? Many times, literature searches will include articles that ultimately are not that relevant to your final topic. You don’t need to read everything! Use your time wisely by being picky about the quality of articles before you spend a lot of time on each one.
   6. How important is this source in the literature? We’ve talked before about the “cited by” tool in Google Scholar, but you can also see a study’s influence (or an author’s influence) on a field by how many times you see other studies cite it as you’re reading. 
   7. Is the source accurate? Check the facts in the article. Can statistics be verified through other sources? Does this information seem to fit with what you have read in other sources?
   8. Is the source reliable and objective? Is a particular point of view or bias immediately obvious, or does it seem objective at first glance? What point of view does the author represent? Are they clear about their point of view? Is the article an editorial that is trying to argue a position? Is the article in a publication with a particular editorial position?
   9. What is the scope of the article? Is it a general work that provides an overview of the topic or is it specifically focused on only one aspect of your topic?
   10. How strong is the evidence in the article? What are the research methods used in the article? Where does the method fall in the hierarchy of evidence? You are not expected to be able to evaluate every detail of the study methods yet (that’s what this class is all about, after all), but you can make some general judgements based on the type of study that is described:
       • Meta-analysis and meta-synthesis: a systematic and scientific review that uses quantitative or qualitative methods (respectively) to summarize the results of many studies on a topic.
       • Experiments and quasi-experiments: include a group of participants in an experimental group as well as a control group. These groups are monitored for the variables/outcomes of interest. Randomized control trials are the gold standard.
       • Longitudinal surveys: follow a group of people to identify how variables of interest change over time.
       • Cross-sectional surveys: observe individuals at one point in time and discover relationships between variables.
       • Qualitative studies: use in-depth interviews and analysis of texts to uncover the meaning of social phenomenon

The last point above comes with some pretty strong caveats, as no study is really better than another. Foremost, your research question should guide which kinds of studies you collect for your literature review. If you are conducting a qualitative study, you should include some qualitative studies in your literature review so you can understand how others have studied the topic before you. Even if you are conducting a quantitative study, qualitative research is important for understanding processes and lived experiences. Any article that demonstrates rigor in both thought and methodology is appropriate to use in your inquiry.

At the beginning of a project, you may not know what kind of research project you will ultimately propose. At this point, consulting a meta-analysis, meta-synthesis, or systematic review might be especially helpful as these articles try to summarize an entire body of literature into one article. Every type of source listed here is reputable, but some have greater explanatory power than others.

Two of the initial steps in designing a research project are identifying the overarching goals of your project and conducting a literature review. Forming a working research question, which we’ll discuss later, is another crucial step. Creating and refining your research question will help you identify the key concepts you will study. Once you have identified those concepts, you’ll need to define them and decide how you will know that you are observing them during your data collection. Defining your concepts, and knowing them when you see them, relates to conceptualization and operationalization (also things we’ll talk about in coming chapters). Of course, you also need to know what approach you will take to collect your data. Thus, identifying your research method is another important part of research design. In addition to identifying your research method, you also need to think about who your research participants will be and the larger group(s) they may represent. Last but certainly not least, you should consider any potential ethical concerns that could arise during the course of your research project. These concerns might come up during your data collection, but they might also arise when you get to the point of analyzing or sharing your research results.

This is to say that research design is an iterative process – you’ll learn something, have an idea, write something down, learn something else, modify your idea, edit your writing, read more, have more ideas…you get the picture. In the design phase, your ideas can be fluid and you can continue to build on what you’re learning as you read and discover more. Give yourself time to think as you read, and afterwards. Sometimes, with a little time, you can make connections between ideas you didn’t see before or you might have new questions come to your mind!

Decisions about the various research components also do not necessarily occur in sequential order. In fact, you may have to think about potential ethical concerns before you even zero in on a specific research question. Similarly, the goal of being able to make generalizations about your population of interest could shape the decisions you make about your method of data collection. Your professor is there to help you work through the process, so utilize the resources provided in your class to consider the many different steps you’ll need to complete (and sometimes return to!) as you work through the process.

Being a responsible consumer of research

Being a responsible consumer of research requires you to take your identity as a social scientist seriously. As you gain familiarity with how to conduct research and how to read the results of others’ research, you have some responsibility to put your knowledge and skills to use. To do so, you must be able to distinguish what you know based on research from what you do not know. It is also a matter of having some awareness about what you can and cannot reasonably know as you encounter research findings.

When assessing social scientific findings, think about what information has been provided to you. In a scholarly journal article, you will presumably be given a great deal of information about the researchers’ method of data collection, their sample, and information about how they identified and recruited research participants. All of these details provide important contextual information that can help you assess the researcher’s claims. On the other hand, a discussion of social scientific research in a popular magazine or newspaper will likely fail to provide the same level of detailed information. In this case, what you do and do not know is more limited than in the case of a scholarly journal article. If the research appears in popular media, search for the author or study title in an academic database and read the full study write-up.

Also, take into account whatever information is provided about a study’s funding source. Most times, the entities that fund a study require that they are acknowledged in the publication, but more popular press may leave out a funding source. In this Internet age, it can be relatively easy to obtain information about how a study was funded. If this information is not provided in the source from which you learned about a study, it might behoove you to do a quick search on the web to see if you can learn more about a researcher’s funding. Findings that seem to support a particular political agenda, for example, might have more or less weight once you know whether and by whom a study was funded.

There is some information that even the most responsible consumer of research cannot know. Because researchers are ethically bound to protect the identities of their subjects, for example, we will never know exactly who participated in a given study. Researchers may also choose not to reveal any personal stakes they hold in the research they conduct. While researchers may “start where they are,” we cannot know for certain whether or how researchers are personally connected to their work unless they choose to share such details. Neither of these “unknowables” are necessarily problematic, but having some awareness of what you may never know about a study does provide important contextual information from which to assess what one can “take away” from a given report of findings.