"

How To Conduct a Rapid Critical Appraisal

Credit: Marcie Hopkins, U of U Health

Performing a rapid critical appraisal helps evaluate a study for its worth by ensuring validity, meaningful data, and significance to the patient.

By: Barbara Wilson, Mary-Jean (Gigi) Austria, and Tallie Casucci

Mar. 21, 2021 |  5 minutes

Learning Objectives

  1. Explain steps in conducting a rapid critical appraisal
  2. Evaluate validit6y, reliability, and applicability
  3. Determine relevance to patient or project

Case Study

Kelly’s literature search produced a considerable number of articles from the database she selected. But how can she narrow things down and determine which items are most relevant to her project? She needs to do a quick overview of them to choose which ones have the most accurate and meaningful results and can best provide information to answer her PICOT question. Is there a process she can follow that is relatively quick yet detailed?

After you’ve collected the relevant literature for you clinical question, it is time to evaluate the relevance of the literature to your PICOT (or to your evidence-based practice project) by performing a rapid critical appraisal.

What is a rapid critical appraisal?

A rapid critical appraisal (RCA) is simply evaluating a study for its worth, based on its validity, reliability and applicability to clinical practice. Generally, an RCA involves some kind of a checklist to help the reviewer quickly determine the appropriateness of the study and whether it can apply to their patients.

Although several RCA tools exist, most answer 3 very important questions to evaluate a study’s worth, namely:

  1. Are the results of the study valid?
  2. What were the results and are they important?
  3. Do the results provide an answer to my PICOT question that will help me care for my patient in this setting?

Rapid critical appraisal checklist

We can use a checklist to guide us through answering these three questions. First, let’s break down the checklist by the broader question and review the specific approach in each section.

Question 1: Are the results of the study valid?

Validity refers to the accuracy of the results. In other words, this question asks, “How accurate are the results of the study and can the results be trusted?”

To answer this question, we need to find some answers.

First, we need to consider how study participants were chosen. Were they selected at random or by some other method? Were measures taken to decrease biased results based on how participants were selected?

Next, we need to consider the accuracy and completeness of the data. Did the authors of the study use a measure or instrument (such as a survey tool) that was validated and reliable?

Finally, we need to determine what data was included and what data was excluded. Does it appear that the researchers took reasonable steps to assure accurate and complete data? Reliability refers to dependability and consistency—do we know the results are an accurate representation of what is being measured?

Question 2: What were the results and are they important?

Essentially, this question asks, “What did the study find and is it meaningful?”

For the answers to some of these questions, you’ll need to understand statistics so we’ll run through statistical terminology.

One thing you’ll often see in the literature is a reference to the “treatment effect.” We need to understand effect size and level of significance to understand treatment effect.

Effect Size 

Effect size refers to the strength of the relationship between the variables. The greater the effect size, the stronger the relationship between the two. Generally, effect size is designated as small (.2), medium (.5), and large (.8). The higher the effect size, the greater the relationship between variables.

For example, the relationship between smoking and lung cancer would have a large effect size because they have a very strong relationship. The effect size between eating dark chocolate and headaches would be relatively small, if a relationship exists at all.

Level of Significance

The level of significance deals with how likely something is to happen or not happen. In studies, it is often depicted by the p-value, or probability. The smaller the p-value, the less likely it is that the reported results happened because of a fluke or chance. Therefore, you can feel more confident in the significance of the data you are reviewing. Most studies use a p-value of .05 as “clinically significant.”

For example, a p-value of .05 means that there is less than a 5% chance, or 5 times out of 100, that the reported occurrence happened by chance or “fluke.” A p-value of .01 means that there is less than a 1% chance, or 1 times out of 100, that the reported occurrence happened by chance. However, even with extremely small p-values, such as .001 (1 times out of 1000), we still cannot say that our results prove anything.

It’s also important to consider how plausible and believable the results were. Do the results of the study seem likely? Do they appear that they could be reproduced? If claims seem far-fetched, it may warrant searching for similar studies to determine if similar results were achieved—otherwise, the intervention may not be beneficial and may have the potential to harm your patient.

Question 3: Do the results provide an answer to my PICOT question that will help me care for my patient in this setting?

This question is also known as clinical significance. Just because something has statistical significance doesn’t mean that it will be clinically significant to your patient population. In determining if it is clinically significant to your patient, you need to consider the following questions, many of which rely on your clinical expertise and understanding of your patient population.

Do the results apply to my patient? 

First, you need to assess whether your patient or patient population is similar to those in the study. There are likely to be differences, but you must determine what the important differences are and if it will impact the effectiveness on your patient population.

For example, say you reviewed a study that demonstrated weekly group education classes were highly effective in helping control blood sugar fluctuations in patients newly diagnosed with type I diabetes. When asking how the results apply to your patient population, you first determine that your clients are also newly diagnosed with type I diabetes. If they aren’t newly diagnosed, the intervention may not produce the same results as those experienced by study participants. When you look at how the populations differ, you note that your patient population is located in a rural setting with limited access to care and that the study population in the published study was located in a major metropolitan city. The results of the original study may be important and meaningful, but due to some differences, it may not help you care for the patients in your setting.

What are the risks and benefits of treatment?

In some studies, researchers are able to carefully monitor and control the risks. In clinical settings, we may not be able to provide monitoring or control for side effects in the same manner as the original study, and the risks may outweigh the benefits for your patient population.

Is the treatment feasible in my clinical setting?

Using our type 1 diabetes example, the study may have found that weekly educational sessions helped to control blood sugar levels, but given the financial costs and available resources at your facility, you may not be able to offer weekly group education sessions.

What are my patient’s values and expectations for the outcome?

Using our same example, if you have a patient who is independent and introverted, group education sessions may not be consistent with these values, even though the intervention was shown to be effective. If your patient is unlikely to attend, it won’t be a helpful intervention for them.

Conclusion

Returning to our case study: While Kelly’s literature search produced a considerable number of articles, she is able to narrow those articles down and determine the most relevant ones by asking 1) whether the results are valid, 2) whether they are important, and 3) whether they will help answer her PICOT question and help her care for her patients.

After you’ve appraised the evidence, you’ll plan how to implement your project and monitor the outcomes, which we’ll address in the next module.

 

This article originally appeared in 2021. 

Contributors

Barbara Wilson
Associate Professor, College of Nursing, University of Utah Health

Mary-Jean (Gigi) Austria

Magnet Program Director, Nursing Support Services, U of U Health
Tallie Casucci
Associate Librarian, J. Willard Marriott Library, University of Utah

License

What is Evidence-Based Practice? Copyright © by Emma Day. All Rights Reserved.

Share This Book