Learning how to evaluate the quality of the information you use is an important skill you need both in the academic setting and in life. Most people evaluate information daily when they make decisions, like deciding what car to buy, where to eat, what features to get on a cell phone, or which doctor to visit. The process of evaluating information for academic research is similar. There are specific things to consider when you decide whether or not to use a piece of information:

• authority of the author or source
• currency
• objectivity/bias
• references/documentation

While this course presents these as basic guidelines to consider when evaluating the quality of a source, it is important to point out that these are only guidelines. You may emphasize different guidelines depending on what you are researching. For example, the currency of a source is essential to consider when researching topics like the latest cancer treatments. Currency might not matter as much when writing a report on the Civil War.

ESTABLISHING AUTHORITY

What makes a person or an organization an “authority” on a subject? There are different types and levels of authority, depending on the situation. One type of authority is education or training. A doctor might have an M.D., and a professor might have a Ph.D., making these individuals authorities in their fields by the years of study, exams, and research required to obtain those degrees. Experience or expertise is another type of authority; that same doctor may have been doing a specific type of cardiovascular surgery for twenty years. In addition to academic training, this individual has practical knowledge through many years of experience, making them an expert in this area. Some people are an authority because of awards they have won or research they have done, and others are recognized as an authority because they work for an institution that is an authority. For example, the Massachusetts Institute of Technology (MIT) is widely known as an authority in engineering, so a faculty member researching nanotechnology for MIT would be considered an authority in this area because of their association with MIT. Some publications are considered more authoritative than others. Peer-reviewed journals are a good example. In general, a peer-reviewed journal is a more authoritative source than a newspaper, as the articles in these journals are written and reviewed by experts before publication. Some publishers, such as Oxford or Harvard University Press, are known for publishing scholarly sources, so the material included in them will be considered authoritative through association.

Here’s an example: For someone wanting to know what it is like to wear an insulin pump, several different “authorities” could answer this question. An endocrinologist has the education and training to answer this question and could tell you what the medical literature says, this individual specializes in conditions like diabetes. A nurse working at an endocrine clinic could probably also answer this question based on working with diabetics at the clinic who wear insulin pumps and describe their experiences. A person who wears an insulin pump would also be just as credible because they have experience and can explain what it actually feels like to sleep and exercise in, and how it differs from taking injections. Someone wanting to know how wearing an insulin pump compares to taking injections because they are a competitive runner will probably seek out other endurance athletes who wear one to answer this question. Therefore, it is a good idea to consider both traditional forms of authority, such as formal credentials, and informal credentials, such as experience.

Here are some questions to consider regarding authority:

• Is the author an expert on the subject? What is the source of their expertise? Is it education or experience? What is their background? Is the author affiliated with a reputable university or organization? If so, what is their position and the name of the university or organization? If the publication doesn’t answer these questions, you can try a Google search for the author, the publisher, etc.
• Has the author published anything else on this topic? If so, is it scholarly? If it’s popular, is it from a respected magazine such as Archaeology or The Economist?
• Is the information in that source peer-reviewed or edited? Edited means at least one other person looked at the information and approved it. Peer-reviewed means 2-3 specialists in the field have approved the publication. Editing is found in popular sources, while peer review (also called refereed) is in scholarly sources. Both indicate the source should be credible.

CURRENCY

When we talk about “currency,” we are not discussing money. We are talking about how new the information is. Sometimes this matters, and other times it doesn’t. If you’re interested in advances in treating depression, your sources must be pretty current — probably within the past few years. For example, older material may be acceptable if you are writing about the pyramids in Egypt. In general, you will need more current information for rapidly changing topics, like technology or medicine. If you’re writing about history, it might not matter as much. However, always ask your professor if they have specific requirements for the assignment.

Determining a publication date can be difficult when using the Web for research and using a website or webpage without a date is often not a good idea.

When you choose a topic, you should be able to explain what is considered current and why. For example, if I’m looking at the latest insulin pump treatments for Type I diabetics, I will limit my search to things published within the last few years. On the other hand, if I’m looking at how treatments for diabetes have changed over the years, I’ll need both newer and older materials.

OBJECTIVITY/BIAS

People often misunderstand the words bias and objectivity. Bias is a human trait, and most information is biased to some degree. When something is “biased,” there is some prejudice in favor of or against a thing, person, or group when compared with another. It is perfectly okay to use a biased source, as long as you are aware of the bias and work with it. Objectivity is the exact opposite and connotes neutrality. So, if one says a source is biased, it is not neutral.

Here is an example: Many medical doctors do research on various topics related to health and disease. They are highly educated experts in this area. However, there is potential for bias in the research conducted. Some may deal with personal bias; a doctor whose mother smoked all her life and died of lung cancer may very well believe that smoking caused the cancer. Another doctor who works for a company that manufactures cigarettes will probably be doing different research because the company controls the type of research conducted; the results of his research may point to lung cancer as having various causes, taking some of the blame away from cigarettes. Although both researchers are highly educated experts, their research may show different results.

Identifying bias in sources can be very difficult because it’s not always obvious. Here are some questions that will help you identify bias in a source:

• What is the intent or purpose of the source? Is it to entertain, voice an opinion, educate, inform, persuade, or sell you a product? Is there a political, commercial, personal, or social agenda? Using sources that have an agenda is acceptable, as long as you recognize it. For example, when writing a paper on logging in the Pacific Northwest, you might use information from the Sierra Club and the Pacific Lumber Company. Both will have a different view on this issue; including both will balance your paper. Bias is not always obvious. For example, many research articles will have some persuasion in the conclusion, but their general intent is to inform. Commercial websites that end with a .com are meant to persuade you to buy something but will often include informative content.
• For pro/con issues, does the information clearly support only one side of an issue? Again, the Sierra Club will have a different view on logging than the Pacific Lumber Company.
• Does the author’s background indicate that they might be biased? The doctor who does research for a non-profit company may have different findings on the link to cancer than one working for a cigarette company. An author writing about whether vaccinations are harmful or helpful for children may be a doctor whose child had an allergic reaction after receiving a vaccination. Or they may be a researcher for a pharmaceutical company that makes the vaccines. These two people may have different perspectives on the issue.
• Does the information try to show both sides? How balanced is the presentation on opposing perspectives? Do they fairly present both sides of the story, or do they support one side over the other? Information that tries to show both sides is often better but be careful. The author is still picking and choosing their information. There can be bias even when the author is trying to be objective.
• What is the tone of language used (angry, sarcastic, balanced, personal opinion, educated)? The tone can give you a hint as to whether there is bias present.
• Is the source known for bias? For example, are you getting information from a left- or right-wing blogger? Is the article from a left or right wing news source?
• Is there anything present in the source that mentions a conflict of interest? Examples might include a note that indicates a specific organization funded the research or an author conflict of interest.

BIAS IN SCHOLARLY JOURNAL ARTICLES

Sometimes you’ll find a research study in a scholarly journal. Just because it is research and was published in a scholarly publication; doesn’t mean bias might not be present. Here are a couple of things to think about that pertain specifically to scholarly research studies, in addition to the previously mentioned items:

• Funding bias: Some research is funded by specific organizations. That doesn’t mean the research is of poor quality, but there may be some bias if the study favors its financial supporters. One example might be an article on a specific medication where a pharmaceutical company funded the study. Bias could include over-emphasizing desired results or suppressing undesired data.
• Selection bias occurs when study participants do not represent the general population, either through selective inclusion or exclusion of study participants. For example, if a study focuses on the impacts of a healthy diet on children with Type 2 diabetes, but only includes girls between the ages of 9- and 12-years old living in California, the results may not be applicable to the entire population.
• Media bias: Media bias generally indicates selectivity in what stories and perspectives are covered in the media, such as when only specific stories or interpretations are covered. An example might be an article that included primarily conservative or liberal news outlets or sensational stories. Citing The Epoch Times will probably present a different perspective than citing The New York Times. In science reporting, bias emphasizes views not supported by evidence.
• Finally, there could be bias in the literature review. A study will typically have a literature review, presenting a comprehensive overview of current knowledge on a topic. If the author excludes information, intentionally or unintentionally, this could create a distorted view of an issue.

EXAMPLES OF BIAS

Here are a couple of examples of bias, one obvious and one not so obvious. First, look at a more straightforward example: the Sierra Club website. Just a cursory glance clearly shows that they are an environmentalist group. You will notice phrases like “We champion solutions to the climate crisis” and “We fight for environmental and social justice” on their homepage. You can probably guess their position on issues like oil and gas, fracking, nuclear energy, hunting, fishing, and water.

When you click on the About Us link on the site to learn more about the organization and what they do, you read that their founder was conservationist John Muir and that it is “the most enduring and influential grassroots environmental organization in the United States” (About the Sierra Club section). The Sierra Club counts their successes, “securing protection for 439 parks and monuments, to winning passage of the Clean Air and Endangered Species Acts” (How we do it section). In addition, the group “works to advance climate solutions and ensure everyone has access to clean air, clean water, and a healthy environment” (Our Roots section).

This website has a clear pro-environmental bias.

An example that might not be so obvious, is an article published in a scholarly peer-reviewed journal called MEDSURG Nursing.

The article, “Optimizing the Treatment of Type 2 Diabetes Using Current and Future Insulin Technologies,” by Mary E. Boyle, compares various insulin treatments for Type 2 diabetics. The author, Mary E. Boyle, is a certified nurse practitioner in the Department of Endocrinology at the Mayo Clinic in Scottsdale Arizona. She is also a CDE, or Certified Diabetes Educator who holds a BC-ADM certification, which means she passed the Board-Certified Advanced Diabetes Management examination. This information indicates that she’s well qualified to write on this topic and will be a pretty objective source.

However, there may be bias present in the article. The article contains the following note:

“Novo Nordisk funded writing and editing assistance ….and this financial assistance was not an educational grant. Editing and medical accuracy review were carried out on the manuscript by Novo Nordisk.” (Boyle, 2008, p. 383).

If you were to search online for Novo Nordisk, you’d see that it is a pharmaceutical company that produces insulin pens and needles. The article contains some subtle bias to pens over other types of insulin therapy. For example:

• “Insulin pens can reduce the complexity and stigma of insulin administration. They are not only convenient and easy to transport, but also unobtrusive to use” (p. 385).
• “Pens were preferred to syringe injection by 75%-90% of patients in crossover studies” (p. 385).
• “As a result of their ease of use and high adherence rates, insulin pens can reduce insulin-related hypoglycemic events and more severe long-term complications, thereby improving patient quality of life” (p. 386).
• “In addition, pen adoption probably can reduce the number of home visits for injection assistance by making injections easier for partially disabled patients to manage independently” (p. 386).

The article does contain a section on insulin pumps. It explains how pumps work and what they do. However, unlike the section about insulin pens, it does not tell the reader about the advantages of pump therapy, even though someone with experience or knowledge of this topic should be familiar with those advantages and know that according to the medical literature a pump offers even better blood glucose control than insulin pens. There is also a section that focuses on jet injectors and presents primarily negative aspects of this technique for blood glucose control.

Novo Nordisk produces insulin pens, but NOT pumps or jet injectors. The article presented positive information about insulin pens, negative information about jet injectors, and neutral information about pumps. Though written by a well-qualified author and published in a peer-reviewed journal, the article demonstrates a bias that must be considered when evaluating it as a source.

REFERENCES/DOCUMENTATION

References/documentation refers to the sources used by the author. In other words, can you tell where the information came from? References may be a list of references at the end of an article, book, or chapter, or footnotes embedded throughout the text. Sometimes, the author does not list any references but instead will include a list of sources for further reading. Sometimes, the author will mention other sources in the text: “A Harvard study found that…” Many online sources will provide links to the original study or a secondary overview. Other times, there will be a formal bibliography (reference list or works cited list) with links to each source used. Sometimes, there is no information at all. When the author provides information about where they received their information, consider the following:

• If there are links to other online sources, do the links work? Do they link to reliable sources? Are the links relevant and supportive? Are they evaluated or annotated?
• How current is the reference list? If the article was published in 1985, is the newest item in the list from 1984 (which would be fairly current at the time this was published), or is the newest item in the list from 1965? If an article was published in 2003, but the author doesn’t cite anything newer than 1970, that may be a cause for concern.
• How comprehensive is the reference list? How many things are cited? Is it a five-page scholarly article with 34 sources cited in the literature review, or is it a 365-page scholarly book that cites only 12 sources? For articles discussing cutting-edge technology, there may not be many sources because the topic itself is so new. Some topics have been written about extensively, so one would expect to see multiple sources cited. There might not be much written for obscure or new topics, so the bibliography for that article might be much smaller. Use common sense here to decide whether you might consider a reference list adequate.
• Does the author provide evidence and examples to support their information? For example, there might not be a bibliography, but if an author says that people prefer a specific thing, do they provide information from studies and research on the topic or do they assume this is common knowledge?
• Have you found similar information in other sources? Are you are seeing similar citations in other sources? If you see the same sources cited over and over, that means they are respected in the field.
• Are all the sources relevant to the topic?
• If a list of sources is provided, what is the quality of these references? What types of sources are cited? Is the author citing scholarly books? Scholarly journal articles? Popular news sources? Government publications or statistical reports? Websites? Blogs? This involves a little detective work because you’ll need to be able to tell what type of source something is just from looking at the citation.

This last category is probably the most essential way to judge whether or not a reference list is adequate. It also requires the reader to have a basic understanding of how to look at a citation and determine the kind of source being referenced. An example of a list of references is listed below.

CREATING EVALUATIVE ANNOTATIONS

In some classes you will write an evaluative annotation. An evaluative annotation describes the quality of the sources you found, written as a brief paragraph or in bullet form. When you evaluate the quality of a source, it is critical to:

• State the evaluation clearly. If you are discussing whether an author is credible, clearly state that in your paragraph: This author is credible. If you are debating the currency of the source or quality of the reference list, state your evaluation of the source explicitly: This source is not current, or the reference list is outstanding.
• Provide evidence to support your statement. If you say the author is credible, provide evidence as to why: The author is a senior researcher at the MD Anderson Cancer Center at the University of Texas. This center is nationally known and is one of the top-ranked cancer treatment centers in the United States. In discussing the currency of the source, you might say that researching cancer treatments requires the most current information available, and because of this, you limited your search to the last two years; the article in question was published in the latest issue of X Scholarly Medical Journal, which came out last month.

Here are some sample sentences used to evaluate sources. When determining the quality of a source, ALWAYS provide details and evidence for your comments.

COMMON MISCONCEPTIONS

As you evaluate the quality of the sources, it is important to remember that how the information is formatted does not necessarily denote credibility — or lack of credibility. Each source must be examined based on its merit, not how it is displayed. For example, there is a tendency to assume that because an article is published in a scholarly peer-reviewed journal, it is not biased, the author is an authority, and the source is generally credible; however, that is not always the case. Sometimes visual images are accepted as credible when the same information in print might not be, or material found in a book is more credible than that found on the Web. There is a tendency to assume that scholarly sources are better than popular sources, or that sites with a .edu or .org domain are better than sites with a .com domain. Avoid the temptation to assume credibility based on generalizations, such as the format in which something is published, the domain of a website, or other commonly accepted assumptions.

Another common mistake is to over-emphasize the checklist (what credentials the author has, what type of source it is, etc.). While these can give you some indicators of quality, it is more important to be able to evaluate claims. To do that, one must consult other sources. A study done by the Stanford History Education Group (Wineburg & McGrew, 2017) found that professional fact-checkers were better at detecting fraudulent online claims than history professors and undergraduate students. This is because they tend to do “lateral reading,” or consulting multiple sources to evaluate a claim, rather than focusing on the sources themselves. Sometimes it’s easy to be fooled by sources that seem or look authoritative.

In addition to consulting multiple sources, it is important to be aware of our personal biases and how we determine whether information is accurate. For example, confirmation bias

occurs when an individual looks for and uses the information to support [their] own ideas or beliefs. It also means that information not supporting [their] ideas or beliefs is disregarded. Confirmation bias often happens when we want certain ideas to be true. This leads individuals to stop gathering information when the retrieved evidence confirms their own viewpoints, which can lead to preconceived opinions (prejudices) that are not based on reason or factual knowledge. Individuals then pick out the bits of information that confirm their prejudices. (Spencer & Heneghan, 2018, Background section)

In conclusion, the checklist is a good starting point, particularly as you prepare research assignments in college. However, don’t stop there. Check other sources, especially those that might challenge your personal beliefs. In other words, keep an open mind as you make decisions about the value of the information you find.