The Third Step: Appraise; Reading & Appraising Research Articles

You are well on your way! Now that you understand how to perform a rigorous search for literature, let’s learn about the sections of a research article so that you can make some sense out of what you’re reading. Remember, you are not expected to be an expert in understanding everything in a research article, but by the end of this course you are to understand some of the basic critical appraisal components so that you can be a knowledgeable consumer of research.

Content includes:

1. Sections of a research journal article
2. Critical aspects of appraisal

Objectives:

1. Identify the major sections in a published research journal article
2. Describe the major sections in a published research journal article
3. Describe the critical aspects of appraisal

A Review of the Peer-Review Process

Submitted papers or articles are often subjected to a peer review by the journal. Peer review is designed to assess the validity, quality and often the originality of articles for publication. Its ultimate purpose is to maintain the integrity of science by filtering out invalid or poor-quality articles. When the article is submitted, it will be reviewed by two or more peer reviewers who are experts in the same field. Most of the time these reviews are blinded, meaning the author of the article does not know who the reviewers are, and the reviewers do not know who the author is. As a consumer of research, you have some assurance that the journal articles are of quality and have been vetted. However, to note, is that this process is not the end-all-be-all. Meaning, just because it has been peer reviewed and published, it does not remove the need to critically appraise a publication.

Sections of Published Research Journal Articles

Journal articles follow a similar format regardless of which journal they are published in. There may be some small changes, but for the most part they follow the IMRAD format:

• Title and abstract
  • Introduction
  • Methods
  • Results
  • (and) Discussion
• References

Title and Abstract: The title presents a starting point in determining whether or not the article has potential to be included in an EBP review. Ideally, the title should be informative. It should also help the reader to understand what type of study is being reported. But don’t rely solely on a title! They can be misleading. Be careful that it is not just an informative article or opinion piece.

In qualitative articles, the title will normally include the central phenomenon and group under investigation. In quantitative articles, the title communicates the key variables and the population (PIO components). Again, these are the usual title components, but not always.

Knowledge to application link.

Quantitative article title:

 

 

Qualitative article title:

 

The abstract will be at the start of the paper. It contains a brief description of the study. Abstract components include a brief introduction, aims/goals, methods/methodology, and the results and conclusions. Some abstracts also include the research question, hypothesis, and implications to practice.

This is located after the title it is usually set apart by the use of a box, shading, or italics. The abstract is a great starting point in quickly filtering articles that may be of use. It’s a nice thumbnail sketch. But, caution! Don’t forgo reading the article! The abstract should serve as a screening device only.

Introduction: The introduction is the start of the article after the abstract. It normally does not contain a heading. The introduction contains the author’s literature review.
 
When critically appraising an article, if the abstract appears relevant, then move onto reading the introduction. The introduction contains the background as well as a problem statement that tells the reader why the study was eventually conducted. It is presented within the context of a current literature review, and it should contain the knowledge gap between what is known and what the study seeks to find (AKA: what is not known yet). It usually (but not always) contains a statement of expected results or hypothesis as well. The literature review section of an article is a summary or analysis of all the published research for prior findings in research and other information that the author read before doing his/her own research. This section is part of the introduction (or in a section called Background). This is the “Review of Literature” (ROL) that the research/author conducted before beginning his/her own study. The articles and information in the Introduction will be cited. It is important to review those citations (found in the References section of an article) to make sure they are up to date and provides a state-of-the-art synthesis of current evidence on the research problem. The ROL also needs to provide a strong rationale for the new study that the researcher wants to conduct. The Introduction section of an article is very similar to your Introduction section of the EBP Poster that you are working on.

The introduction will also usually include a description of the central concepts or variables (sometimes these are in the Methods section instead); the study purpose, research questions, or hypotheses; a review of literature; theoretical/conceptual framework, and the study significance/need for the study.

Knowledge to application link.

Quantitative introduction:

 

 

Qualitative introduction:

 

Methods: This section describes how a study is conducted. In quantitative studies, the methods section contains the research design, sampling plan, methods for measuring variables and collecting data, the study procedures, participant protection, and analytic methods and procedures. In qualitative studies, the methods section discusses many of the same issues as quantitative researchers do but with different emphases. More information is provided about the research setting in a more organically focused fashion, in the context of the study.

Knowledge to application link.

Quantitative Methods:

Qualitative Methods:

 

   
Results: The results section contains the findings of the study. This section is very objective without any researcher interpretation. These are the raw findings from data. The statistical tests will be names, the value of the statistics will be listed, statistical significance will be listed, and the level of significance is listed. The level of significance, which we will come back to later in the modules, is an index of how probably it is that the findings were not just simply due to chance and is listed as the p value. In qualitative studies, the results are a bit different. They are organized according to major themes (thematic results) and most often include raw data as quotes from the study participants.

Figures and tables are usually presented – some inferential and some descriptive. Also look to see whether the results report statistical versus clinical significance. You might have to look up unfamiliarly terminology regarding statistical tests.

Knowledge to application link.

Quantitative results:

 

 

Qualitative results:

 

 

Discussion: This section should be tied back to the introduction. The study findings should be discussed and meaning given to the raw data results. The weaknesses/limitations and bias should be reported. Any limitations and bias should be discussed in this section as well. If there is clinical significance, this will usually be discussed.

Critical Aspects of Appraisal

In module 12, we will dive deep into specific appraisal criteria, and we will delve into critical appraisal components a little at a time up to that point. However, we need to have an understanding of why we need to be able to critically appraise published research.

Critical appraising evidence is an essential step in EBP in response to a clinical question (Polit & Beck, 2021). The process of systematically evaluating research for its reliability, importance, and significance is critical appraisal (Hall & Roussel, 2014). The process is necessary in considering whether the research was conducted appropriately, because not all research is awesome.

Critical appraisal is the process of carefully and systematically assessing the outcome of scientific research (evidence) to judge its trustworthiness, value, and relevance in a particular context. Critical appraisal looks at the way a study is conducted and examines factors such as internal validity, generalizability, and relevance.

Critical appraisal allows us to:

• reduce information overload by eliminating irrelevant or weak studies
• identify the most relevant papers
• distinguish evidence from opinion, assumptions, misreporting, and belief
• assess the validity of the study
• assess the usefulness and clinical applicability of the study
• recognize any potential for bias.

The beginning guidelines for critical appraisal include asking:

1. How relevant is the research problem to the actual practice of nursing?
2. Was the study quantitative or qualitative? How did you make that determination?
3. What was the underlying purpose (or purposes) of the study? (Therapy/Intervention, Diagnosis/Assessment, Prognosis, Etiology/Harm, Description, or Meaning?)
4. What might be some clinical implications of this research? To what type of people and settings is the research most relevant? If the things were accurate, how might you use the results of this study?

An inference is a conclusion drawn. It is a challenge to design studies to support inferences that are reliable (accurate and consistent data; concerns the truthfulness in the data obtained and the degree to which any measuring tool controls random error, and if it can be reproduced under the same conditions) and valid (soundness of the evidence; validity is about what an instrument measures and how well it does so, and whether the results really do represent what they were supposed to measure) in quantitative studies, and that are trustworthy in qualitative studies. Thus, every article, even if peer-reviewed, needs to be critically appraised.

Some further aspects of critical appraisal include the following.

Bias

Ever ask someone’s opinion about the “best” brand of shoes or “best” local coffee? Their answer is based on their own experience, thus biased to their opinion. Bias is a distortion or an influence that results in an error in inference.

When reading articles, we need to be very aware of the potential for bias. Some examples of factors creating bias include:

• Lack of participants’ candor
• Faulty methods of data collection
• Researcher’s preconceptions
• Participants’ awareness of being in a special study
• Faulty study design
• Survey question design

Survey questions are very commonly biased.  With regard to bias in surveys, it is always wise to utilize participant anonymity as this would make it more likely to obtain truthful answers especially when questions are controversially-based.

Surveys are used frequently in research, so let’s dive into how surveys can thwart research results. Depending on how they are asked, questions can lead the respondent intentionally down a path to a certain answer or are phrased in a manner that can be confusing to them, leading to unclear responses. The following are some examples:

Leading questions: These are easy to spot, as it is clear there is a “correct” answer that the question is leading you toward. This will always result in false information as there was never an option for an honest response.

Example: “What problems did you have with the product?”

You can see that this question was set up in a way to assume there was something wrong in the first place.

Alternative: “On a scale of 1-10, how satisfied were you with the product?”

Vague or ambiguous questions: On the surface, these can look honest and harmless, but their vague nature can confuse the participant into a poor or inaccurate response.

Example: “How did this hospital experience compare with other hospital experiences?”

Gosh. Have they had other hospital experiences? If not, then what? Is there a benchmark to compare against? What “experience”?… price? Friendly nurses? Good medical care? So much is terrible about this question format.

Alternative: “On a scale of 1-10, with 10 being highly likely, how likely would you be to recommend this hospital to others?”

Double-Barreled Questions: These types of questions ask the participant to provide an opinion on two topics but only provides an opportunity for one response.

Example: “How satisfied are you with your hospital stay and discharge instructions?”

If they are satisfied with the hospital stay but not the discharge instructions, how do they answer this?

Alternative: Split the question into two questions.

Absolute Questions: These questions can bias the participants’ choices by forcing them into an absolutely categorical response when they might not have one. These use words such as “never”, “always”, “will”, etc.

Example: “Do you always have eggs for breakfast?”

The problem with the above example is that the answer will be no. The chances of someone having eggs for breakfast every single day is very slim. Obviously, this skews the data because there are no alterative choices.

Alternative: Give options! “Do you eat of the following for breakfast? Select as many as applicable. Add any foods.”

                                __ Eggs

                                __ Bacon

                                __ Protein Shake

                                __ Pancakes

                                __ Fruit

                                __ Cottage Cheese

                                __ Other (specify)

                                __ Other (specify)

Acquiescence Bias Questions: These questions are usually a binary yes/no or agree/disagree choice. Guess what? People are often more likely to respond positively when only two options are presented.

Example: “Is our level of customer service satisfactory?” Yes/No”

We will learn nothing of value form these types of questions.

Alternative: Either offer multiple options or offer an open form text box, such as: “Please describe your experience with customer service.”

Well? Did you find this section on biased survey questions helpful?

Lastly, motivated respondent bias occurs when people are highly motivated to complete a survey in hopes of changing public opinion or influencing the outcome of the research. Avoiding acquiescence bias questions absolutely helps to minimize this risk.

Hot Tip! It takes a keen eye to spot or to critically think about what bias may exist. It’s not always easy to notice. And, even more difficult, is spotting our own bias. Knowing this, researchers may not always be aware of bias since they are so invested into their study. It’s human nature to tent to ignore things when we want something badly, yes? So, be on the lookout!

Research Control

Research control is one way that researchers can address bias. In quantitative studies, randomness is often utilized. This allows certain aspects of the study to be left to chance rather than to researcher or participant choice. Randomization is the process of assigning participants to treatment and control groups, assuming that each participant has an equal change of being assigned to any group. The most common and basic method of simple randomization is flipping a coin. For example, with two treatment groups (control versus treatment), the side of the coin (i.e., heads – control, tails – treatment) determines the assignment of each subject.

However, subjects in various groups should not differ in any systematic way. If treatment groups are systematically different, research results will be biased. Suppose that subjects are assigned to control and treatment groups in a study examining the efficacy of a surgical intervention. If a greater proportion of older subjects are assigned to the treatment group, then the outcome of the surgical intervention may be influenced by this imbalance. The effects of the treatment would be indistinguishable from the influence of the imbalance of variables, thereby requiring the researcher to control for the variables in the analysis to obtain an unbiased result.

Sometimes, for example collecting a large aggregate dataset, the researcher will choose to randomly select a particular number of data to utilize such as every third data point. Remember our Level of Evidence Hierarchy? The levels are ranked according to study types that tend to feature more control over bias.

Blinding is used in some studies to prevent biases stemming from people’s awareness. This involves concealing information from participants, data collectors, or care providers to enhance the objectivity.

Hot Tip! When reading articles, start with the abstract to help get an idea of the overall study. Then, grab the critical appraisal tools that are posted in this course for you. Use those tools as you read articles. Make a checkmark next to each criterion as you go.

References & Attribution

“Green check mark” by rawpixel licensed CC0.

“Light bulb doodle” by rawpixel licensed CC0.

“Magnifying glass” by rawpixel licensed CC0

“Orange flame” by rawpixel licensed CC0.

Haedtke, C., Smith, M., VanBuren, J., Kein, D., Turvey, C. (2017). The relationships among pain, depression, and physical activity in patients with heart failure. Journal of Cardiovascular Nursing, 32(5), E21-E25.

Hall, H. & Roussel, L. (2014). Evidence-based practice: An integrative approach to research, administration, and practice. Jones & Bartlett.

Polit, D. & Beck, C. (2021). Lippincott CoursePoint Enhanced for Polit’s Essentials of Nursing Research (10th ed.). Wolters Kluwer Health.