Learning how to evaluate the quality of the information you use is an important skill you need both in the academic setting and in life. Most people evaluate information every day when they make decisions, such as deciding what car or home to buy, where to eat, what features to get on a cell phone, or which doctor to visit. The process of evaluating information for academic research is similar. There are specific things to consider when you decide whether or not to use a piece of information:

• authority of the author or source
• currency
• objectivity/bias
• references/documentation

While this course presents these as basic guidelines to consider when evaluating the quality of a source, it is important to point out that these are only guidelines. You may emphasize different guidelines depending on what you are researching. For example, the currency of a source is essential to consider when researching topics like the latest cancer treatments. Currency might not matter as much when writing a report on the Civil War.

ESTABLISHING AUTHORITY

What makes a person or an organization an “authority” on a subject? There are different types and levels of authority, and these differ depending on the situation. One type of authority is education or training. A doctor might have an M.D., and a professor might have a Ph.D., making these individuals authorities in their fields by virtue of the years of study, exams, and research required to obtain those degrees. Experience or expertise is another type of authority; that same doctor may have been doing a specific type of cardiovascular surgery for twenty years. In addition to academic training, this individual also has practical knowledge through many years of experience, making them an expert in this area. Some people are recognized as an authority because of awards they have won or research they have done, and others are recognized as authorities because they work for an institution that is recognized as an authority. For example, Massachusetts Institute of Technology (MIT) is widely known as an authority in the field of engineering, so a faculty member doing research on nanotechnology for MIT would be considered an authority in this area because of their association with MIT. Some publications are considered more authoritative than others. Peer-reviewed journals are a good example. In general, a peer-reviewed journal would be considered a more authoritative source than a newspaper, as the articles in these journals are written and reviewed by experts before they are published. Some publishers, such as Oxford or Harvard University Press, are known for publishing scholarly sources, so the material included in them will be considered authoritative through association.

Here’s an example: For someone wanting to know what it is like to wear an insulin pump, there are several different “authorities” that could answer this question. An endocrinologist has education and training to answer this question and could tell you what the medical literature says; this individual specializes in conditions like diabetes. A nurse working at an endocrine clinic could probably also answer this question based on working with diabetics at the clinic who wear insulin pumps and describe their experiences. A person who wears an insulin pump would also be just as credible because they have experience and can explain what it actually feels like to sleep and exercise in, and how it is different from taking injections. Someone wanting to know how wearing an insulin pump compares to taking injections because they are a competitive runner will probably seek out other endurance athletes that wear one to answer this question. Therefore, it is a good idea to consider not only traditional forms of authority, such as formal credentials, but to also consider informal credentials, such as experience.

Here are some questions to consider regarding authority:

• Is the author an expert on the subject? What is the source of their expertise? Is it education or experience? What is their background? Is the author affiliated with a reputable university or organization? If so, what is their position and what is the name of the university or organization? If the publication doesn’t answer these questions, you can try a Google search for the author, the publisher, etc. to learn more.
• Has the author published anything else on this topic? If so, is it scholarly? If it’s popular, is it from a respected magazine such as Archaeology or The Economist?
• Is the information in that source peer-reviewed or edited? Edited means at least one other person looked at the information and approved it. Peer-reviewed means 2-3 specialists in the field have approved the publication. Editing is mostly found in popular sources, while peer-review (also called refereed) is found in scholarly sources. Both indicate the source should be credible.

CURRENCY

When we talk about “currency,” we are not talking about money. We are talking about how new the information is. Sometimes this matters, and other times it doesn’t. If you’re interested in advances in treating depression, your sources need to be pretty current — probably within the past couple of years. If you are writing about the pyramids in Egypt, older material may be acceptable. In general, for topics that are changing rapidly, like technology or medicine, you will need more current information. If you’re writing about history, it might not matter as much. However, always ask your professor if they have specific requirements for the assignment.

Determining when something was published can be difficult when using the Web for research, and using a website or webpage without a date is often not a good idea.

When you choose a topic, you should be able to explain what is considered current for that topic and why. For example, if I’m looking at the very latest insulin pump treatments for Type I diabetics, I’m going to limit my search to things that have been published within the last couple of years. On the other hand, if I’m looking at how treatments of diabetes have changed over the years, then I’ll need both newer and older materials.

OBJECTIVITY/BIAS

People often misunderstand the words bias and objectivity. Bias is a human trait, and most information is biased to some degree. When something is “biased,” there is some sort of prejudice in favor of or against a thing, person, or group as compared with another. It is perfectly okay to use a biased source, as long as you are aware of the bias and work with it. Objectivity is the exact opposite, and connotes neutrality. So, if one says a source is biased, that means it is not completely neutral.

Here is an example: Many medical doctors do research on a variety of topics related to health and disease. They are highly educated experts in this area. However, there is potential for bias in the research they do. Some may deal with personal bias; a doctor whose mother smoked all her life and died of lung cancer may very well believe that smoking caused the cancer. Another doctor who works for a company that manufactures cigarettes will probably be doing different research because the company controls the type of research conducted; the results of his research may point to lung cancer as having a variety of different causes, taking some of the blame away from cigarettes. Although both researchers are highly educated experts, their research may show very different results.

Identifying bias in sources can be very difficult because it’s not always obvious. Here are some questions that will help you identify bias in a source:

• What is the intent or purpose of the source? Is it to entertain, voice an opinion, educate or inform, persuade, or sell you a product? Is there a political, commercial, personal, or social agenda? Using sources that have an agenda is okay, as long as you recognize it. For example, if you are writing a paper on logging in the Pacific Northwest, you might use information from the Sierra Club and from the Pacific Lumber Company. Both will have a different view on this issue, and including both of them will balance your paper. Bias is not always obvious. For example, many research articles will have some persuasion in the conclusion, but their general intent is to inform. Commercial websites, ending with a .com, are meant to persuade you to buy something, but will often include informative content.
• For pro/con issues, does the information clearly support only one side of an issue? Again, the Sierra Club will have a different view on logging than the Pacific Lumber Company.
• Does the author’s background indicate that they might be biased? The doctor who does research for a non-profit company may have different findings on the link to cancer than one working for a cigarette company. An author writing about whether vaccinations are harmful or helpful for children may be a doctor whose child had an allergic reaction after receiving a vaccination. Or, they may be a researcher for a pharmaceutical company that makes the vaccines. These two people may have different perspectives on the issue.
• Does the information try to show both sides? How balanced is the presentation on opposing perspectives? Do they fairly present both sides of the story, or do they support one side over the other? Information that tries to show both sides is often better, but be careful. The author is still picking and choosing their information. There can be bias even when the author is trying to be objective.
• What is the tone of language used (angry, sarcastic, balanced, personal opinion, educated)? This can give you a hint as to whether or not there is bias present.
• Is the source known for bias? For example, are you getting information from a left or right wing blogger? Is the article from a left or right wing news source?
• Is there anything present in the source that specifically mentions a conflict of interest? Examples might include a note in a research study that indicates a specific organization funded the research, or some mention of an author conflict of interest.

BIAS IN SCHOLARLY JOURNAL ARTICLES

Sometimes you’ll find a research study in a scholarly journal. Just because it’s research and it was published in a scholarly publication, doesn’t mean bias might not be present. Here are a couple of things to think about that pertain specifically to scholarly research studies, in addition to the previously mentioned items:

• Funding bias: Some research is funded by specific organizations. That doesn’t mean the research is of poor quality, but there may be some bias present if the study tends to favor its financial supporters. One example might be an article on a specific medication where a pharmaceutical company funded the study. This could include over-emphasizing desired results, or suppressing undesired data.
• Selection bias occurs when study participants are not representative of the general population. This can occur through selective inclusion or exclusion of study participants. For example, if one is studying the effects of a healthy diet on children with Type 2 diabetes but the study only included girls from ages 9-12 who lived in California, the results may not be generalizable to the population as a whole.
• Media bias: Media bias generally indicates selectivity in what stories and perspectives are covered in the media, such as when only certain stories or interpretations are covered. An example might be an article that included primarily conservative or liberal news outlets or stories that are sensational. Citing The Epoch Times will probably present a very different perspective than citing The New York Times. In science reporting, bias can be introduced by emphasizing views that are not supported by evidence.
• Finally, there could be bias in the literature review. A study will typically have a literature review which presents a comprehensive overview of current knowledge on a topic. If the author left out important information, either intentionally or unintentionally, this could create a distorted view of an issue.

EXAMPLES OF BIAS

Here are a couple of examples of bias, one quite obvious and one not so obvious… First, let’s take a look at a more straightforward example: the Sierra Club website. Just a cursory glance clearly shows that they are an environmentalist group. You will notice phrases like “We champion solutions to the climate crisis” and “We fight for environmental and social justice” on their homepage. You can probably guess what their position is on issues like oil and gas, fracking, nuclear energy, and even hunting and fishing and water issues.

When you click on the About Us link on the site to learn more about the organization and what they do, you read that their founder was conservationist John Muir, and that they are “the most enduring and influential grassroots environmental organization in the United States” (About the Sierra Club section). The Sierra Club counts as their successes, “securing protection for 439 parks and monuments, to winning passage of the Clean Air and Endangered Species Acts” (How we do it section). In addition, the group “works to advance climate solutions and ensure everyone has access to clean air, clean water, and a healthy environment” (Our Roots section).

This website has a clear pro-environmental bias.

An example that might not be so obvious, is an article published in a scholarly peer-reviewed journal called MEDSURG Nursing.

The article, titled “Optimizing the Treatment of Type 2 Diabetes Using Current and Future Insulin Technologies,” by Mary E. Boyle, compares the various types of insulin treatments for Type 2 diabetics. The author, Mary E. Boyle, is a certified nurse practitioner in the Department of Endocrinology at the Mayo Clinic in Scottsdale Arizona. She is also a CDE, or Certified Diabetes Educator who holds a BC-ADM certification, which means she passed the Board Certified Advanced Diabetes Management examination. This information indicates that she’s well qualified to write on this topic, and is probably going to be a pretty objective source.

However, there may be bias present in the article. The article contains the following note:

“Novo Nordisk funded writing and editing assistance ….and this financial assistance was not an educational grant. Editing and medical accuracy review were carried out on the manuscript by Novo Nordisk.” (Boyle, 2008, p. 383).

If you were to search online for Novo Nordisk, you’d see that it is a pharmaceutical company that produces insulin pens and needles. The article contains some subtle bias to pens over other types of insulin therapy. For example:

• “Insulin pens can reduce the complexity and stigma of insulin administration. They are not only convenient and easy to transport, but also unobtrusive to use” (p. 385).
• “Pens were preferred to syringe injection by 75%-90% of patients in crossover studies” (p. 385).
• “As a result of their ease of use and high adherence rates, insulin pens can reduce insulin-related hypoglycemic events and more severe long-term complications, thereby improving patient quality of life” (p. 386).
• “In addition, pen adoption probably can reduce the number of home visits for injection assistance by making injections easier for partially disabled patients to manage independently” (p. 386).

The article does contain a section on insulin pumps. It explains how pumps work and what they do, but unlike the section about insulin pens, it does not tell the reader about the advantages of pump therapy, even though someone with experience or knowledge of this topic should be familiar with those advantages and would also know that according to the medical literature a pump offers even better blood glucose control than insulin pens. There is also a section that focuses on jet injectors, and presents primarily negative aspects of this technique for blood glucose control.

Novo Nordisk produces insulin pens, but NOT pumps or jet injectors. The article presented a host of positive information about insulin pens, negative information about jet injectors, and neutral information about pumps. Though written by a well-qualified author and published in a peer-reviewed journal, the article clearly demonstrates a bias that must be considered when evaluating it as a source.

REFERENCES/DOCUMENTATION

References/documentation refers to the sources used by the author. In other words, can you tell where the information came from? This may take the form of a list of references at the end of an article or book or within each chapter, or footnotes embedded throughout the text of a book or article. Sometimes the author does not list any references, but instead will include a list of sources for further reading. Sometimes, the author will simply mention other sources in the text: “A Harvard study found that…” Many online sources will provide links to either the original study or a secondary overview of that study. Other times, there will be a formal bibliography (reference list or works cited list) with links to each source used. Sometimes, there is no information at all. When the author provides information about where they received their information, consider the following:

• If there are links to other online sources, do the links work? Do they link to reliable sources? Are the links relevant and supportive? Are they evaluated or annotated?
• How current is the reference list? If the article was published in 1985, is the newest item in the list 1984 (which would be fairly current at the time this was published), or is the newest item in the list from 1965? If an article was published in 2003, but the author doesn’t cite anything newer than 1970, then that may be a cause for concern.
• How comprehensive is the reference list? How many things are cited? Is it a five-page scholarly article with 34 sources cited in the literature review, or is it a 365-page scholarly book that cites only 12 sources? For articles that are discussing a cutting edge technology, there may not be many sources because the topic itself is so new. Some topics have been written about extensively, so one would expect to see a large number of sources cited. For a more obscure topic, or one that is fairly new, there might not be much written about it yet, so the bibliography for that article might be much smaller. Use common sense here to decide whether you might consider a reference list adequate.
• Does the author provide evidence and examples to support their information? For example, there might not be a bibliography, but if an author says that people prefer a certain brand of detergent, do they provide information from studies and research on the topic or do they just assume this is common knowledge?
• Have you found similar information in other sources? Are you seeing similar citations in other sources? If you are seeing the same sources cited over and over again, this is an indication that these are respected in the field.
• Are all of the sources relevant to the topic?
• If there is a list of sources provided, what is the quality of these references? What types of sources are cited? Is the author citing scholarly books? Scholarly journal articles? Popular news sources? Government publications or statistical reports? Websites? Blogs? This involves a little detective work because you’ll need to be able to tell what type of source something is just from looking at the citation.

This last category is probably the most important way to judge whether or not a reference list is adequate. It also requires the reader to have some basic understanding of how to look at a citation and tell what kind of source is being referenced. An example of a list of references is listed below.

CREATING EVALUATIVE ANNOTATIONS

In some classes you will be required to write an evaluative annotation. An evaluative annotation describes the quality of the sources you found. It can be written as a brief paragraph or in bullet form. When you evaluate the quality of a source, it is important to:

• State the evaluation clearly. If you are discussing whether an author is credible, clearly state that in your paragraph: This author is credible. If you are discussing the currency of the source or the quality of the reference list, state your evaluation of the source explicitly: This source is not current, or the reference list is outstanding.
• Provide evidence to support your statement. If you say the author is credible, provide evidence as to why: The author is a senior researcher at the MD Anderson Cancer Center at the University of Texas. This center is nationally known and is one of the top-ranked cancer treatment centers in the United States. In discussing the currency of the source, you might say that researching cancer treatments requires the most current information available, and because of this, you limited your search to the last two years; the article in question was just published in the latest issue of X Scholarly Medical Journal, which came out last month.

Here are some sample sentences used to evaluate sources. When evaluating the quality of a source, remember to ALWAYS provide details and evidence for your comments.

COMMON MISCONCEPTIONS

As you evaluate the quality of the sources, it is important to remember that how the information is presented (the format) does not necessarily denote credibility — or lack of credibility. Each source must be examined based on its merit, not how it is packaged. For example, there is a tendency to assume that because an article is published in a scholarly peer-reviewed journal, it is not biased, the author is an authority, and the source is generally credible; however, that is not always the case. Sometimes visual images are accepted as credible when the same information in print might not be, or material that is found in a book is more credible than that found on the Web. There is a tendency to assume that scholarly sources are better than popular sources, or that sites with a .edu or .org domain are better than sites with a .com domain. Avoid the temptation to assume credibility based on generalizations, such as the format in which something is published, the domain of a website, or other commonly accepted assumptions.

Another common mistake is to over-emphasize the checklist (what credentials the author has, what type of source it is, etc.). While these can give you some indicators of quality, it is more important to be able to evaluate claims. In order to do that, one must consult other sources. A study done by the Stanford History Education Group (Wineburg & McGrew, 2017) found that professional fact-checkers were better at detecting fraudulent online claims than history professors and undergraduate students because of their tendency to do “lateral reading,” or consulting a variety of sources to evaluate a claim, rather than focusing on the sources themselves. Sometimes it’s easy to be fooled by sources that seem or look authoritative.

In addition to consulting multiple sources, it is important to be aware of our own personal biases, and how we determine whether information is true. For example, confirmation bias

occurs when an individual looks for and uses the information to support [their] own ideas or beliefs. It also means that information not supporting [their] ideas or beliefs is disregarded. Confirmation bias often happens when we want certain ideas to be true. This leads individuals to stop gathering information when the retrieved evidence confirms their own viewpoints, which can lead to preconceived opinions (prejudices) that are not based on reason or factual knowledge. Individuals then pick out the bits of information that confirm their prejudices. (Spencer & Heneghan, 2018, Background section)

In conclusion, the checklist is a good starting point, particularly as you prepare research assignments in college. However, don’t stop there. Check other sources, particularly those that might challenge your personal beliefs. In other words, keep an open mind as you make decisions about the value of the information you find.