19 “I See Research Questions Everywhere”: Developing Metacognitive Skills in an English-Major Research Methods Course

In addition to Joyce Kinkead, Authors includes Non-USU Faculty John Draeger & Jill K. Singer

How many ways can sticky notes—branded as Post-it Notes and introduced in 1980 by 3M—be used by college students? According to the undergraduate who investigated this topic, 31. In the report she produced as a result of a study conducted in English 3470: Approaches to Research in English Studies, the researcher found that “many stationery products have died with advancements in technology, but the Post-it Note has thrived and continues to play a role in productivity in the workplace, continuing at the top of the supply list” (Eralie, 2019). To come to this magic number of 31, Megan conducted a case study and used tools such as a Qualtrics survey. She shared the results with the campus community through a poster during the annual Student Research Symposium (Eralie, 2019). Megan did not arrive in class thinking, “Oh, I’m going to focus on uses of sticky notes for my project”; the topic came to the forefront when I noticed that Megan had what I thought was an atypical way to store information from class lectures—on sticky notes, which could then be organized and re-organized easily, as she pointed out to me. This was an analog strategy for a digital native.

This brief example is but one of dozens of students who have engaged in research that is meaningful to them, and it exemplifies several Habits of Mind. First, it demonstrates how both teacher and student can draw on the 16 Habits of Mind identified by the Institute for Habits of Mind. As a teacher, I have structured this empirical research methods course so that students identify questions they are interested in, allowing them autonomy to follow their curiosity, but it also means that I can help them channel these questions into researchable topics. As a research-minded scholar, I bring an open attitude that encourages questioning and posing problems. This often results in asking students to re-see ideas that they take for granted—like the sticky notes. Megan had not thought deeply about sticky notes before this experience but approached the task of identifying their functions with a sense of wonderment and awe, and I share that sense of wonderment. As she compiled more and more uses of sticky notes, she was open to data and how they would accumulate to get a “big picture” of a ubiquitous and often taken-for-granted office item.

Other students in English 3470 have investigated how the hook-up dating app Tinder differs rhetorically in Utah; the effect of a high school one-on-one laptop policy on the quality and quantity of writing; what support Latinx students can find on websites of four-year Utah colleges; and what parents look for in faith-based children’s literature. At the conclusion of the semester, students complete a research report, an oral presentation (termed a “lightning talk”), and a research poster. They present either this “lightning talk” or poster at a campus symposium. These research outputs demonstrate how students have gathered data from multiple sources that must be communicated accurately and clearly to the public. It is no small task to complete a complex research project within a 15-week course. They must persist and be tenacious, particularly when encountering barriers in the research process.

Clearly, this course supports multiple Habits of Mind. It does so in part as it is based in a philosophy of making “research-based learning the standard,” a recommendation that arose from the influential Boyer Report, Reinventing Undergraduate Education: A Blueprint for America’s Research Universities (1998). The transformative power of students engaging in authentic research has been well documented (Kinkead, 2003; Kuh, 2008; Lopatto, 2009; Seymour et al., 2004). Undergraduate research has, in fact, been identified by the Association of American Colleges and Universities (AAC&U) as one of 11 high-impact practices. It enfolds many of the Habits of Mind. Classes such as mine are also labeled “CUREs,” that is, course-based undergraduate research experiences.

In this chapter, I describe how an empirical research methods course for English majors draws on Habits of Mind. One of the primary habits to be considered is metacognition—thinking about thinking—as the course itself has been part of an National Science Foundation-funded pilot project to examine deliberate integration of metacognitive activities in CUREs. Other instances of Habits of Mind are addressed within metacognitive exercises.

But first, a word about the student population and the context of the field of English studies. Not one monolithic discipline, English studies is comprised of multiple subfields: literature, writing studies, creative writing, technical and professional writing, English education, linguistics, and folklore. While English is generally considered part of the humanities, some of these fields draw on social scientific approaches. Even though the students are all considered English majors, the mode of inquiry can vary significantly. English education students engage in teacher action research, conducting case studies or even experiments. Folklorists organize field studies identifying culture and traditions. Technical communicators conduct usability studies, while literature students analyze, and creative writers create texts. As a result, a common interest among all groups is text, which becomes a way in for a course in empirical research that requires both qualitative and quantitative tools for such diverse subfields of English studies. The textbook Researching Writing: An Introduction to Research Methods (Kinkead, 2016), reinforces this emphasis on text. Thus, a creative writer can analyze sales of fantasy titles over several decades or conduct a rhetorical analysis of TikTok BookToks—both grounded in texts. Students undertake research that is considered RAD: replicable, aggregable, and data-supported (Haswell, 2005). One other commonality exists for these students: apprehension about their ability to use math or statistics.

Metacognition Writ Large

EvaluateUR-CURE was developed and pilot-tested with funding through the National Science Foundation (NSF) Advanced Technological Education (ATE) program. It is an extension of a previous NSF-funded project that resulted in a method known as EvaluateUR, which provided statistically reliable assessments of student growth as they engaged in research (Singer et al., 2022; Singer & Zimmerman, 2012). This expansion was designed to meet the needs of entire classrooms of students who conduct course-based research in two-year college programs preparing students to enter the STEM technician workforce, but the method is also broadly applicable to any institutional type or field of study. While EvaluateUR and EvaluateUR-CURE focused on STEM fields, both methods are appropriate for broader disciplinary areas. For instance, in engineering, Grinberg and Singer (2021) demonstrate how ETAC ABET student outcomes are well aligned. On the other hand, humanities fields have been less likely to engage in these types of projects. I took up that challenge to see how a humanities class would fare, answering an open call for participation.

A distinguishing feature of EvaluateUR-CURE is the emphasis on metacognition, encouraging learners to be aware of what they are doing and why and then to use that awareness to learn even more effectively. Metacognition is a key Habit of Mind. The goal is to provide students with ongoing opportunities to reflect on their own learning process to build metacognitive habits (Scharff et al., 2017).

I enrolled the English 3470 course, which fulfills a quantitative requirement for graduation (known as the “QI” course), into the EvaluateUR-CURE study. (For a more complete description of the class, see Kinkead, 2021; Kinkead 2019.) Every department at Utah State University is to offer a class that integrates mathematics and/or statistics to show their application to the field. English majors learn how numbers can be applied to words and texts, the primary focus of all the department’s subfields (e.g., literature, English education, writing). Admittedly, many of these students are numbers adverse.

Two research projects provide the work of the semester: the first, a whole-class project to demonstrate and model steps in the research process; the second, individual projects of students’ choosing. For the whole-class project, students have investigated high-impact practices in our department, producing a report submitted to department leadership; studied the writing lives of English majors; measured apprehension among English majors about quantitative research; and delved into the history of the Blue Book (Duersch et al., 2018). The 14 students enrolled in the fall 2020 course answered a CFP from College English, a top journal in the profession, that focused on undergraduate research and conducted a study of the effects of an empirical research methods course on English majors by surveying alumni who had completed the course (Stringham et al., 2022). Spoiler alert: the researchers found that students’ view of research changed dramatically because of their immersion in research. They “undergo huge changes in confidence levels, research skills, and understanding of the research process” (p. 562).

Although reflection had been a component in the course in previous iterations, E-CURE intensified this meta-thinking, amplifying students’ analysis of their work. The project includes a menu of metacognitive activities, designed by John Draeger, part of the central E-CURE team, from which faculty collaborators may choose. They include the following:

• Thinking about how to ask good questions
• Thinking about how to ask good research questions
• Reading with a purpose
• Reading for research
• Building resilience
• Building research resilience
• Learning from past projects
• Developing project-management skills
• Working and collaborating as a team member
• Building effective communication

A course could include all these metacognitive exercises, but for this class, students reflected on five: evaluate research questions, read and analyze research literature, communicate findings effectively, manage projects, and work as a team member.

Within each of these topics, exercises provide a set of questions for the students to engage in structured reflection. Through these questions, students take note of what is working, what is not working, and possible adjustments to their thinking and research processes. Within these metacognitive exercises, other Habits of Mind come to the surface. For instance, when students get stuck, they need to think flexibly, change perspective, generate alternative paths, and consider options. For researchers, this ability to think broadly is essential. Students may enter a research project believing they know what will result but then be surprised.

Here is an example of that principle in action. An English education student was sure from her own experiences that automated writing evaluators (AWE) like Turn-It-In are bad ways to evaluate secondary school writing; however, through her own research exploration that involved interviews with teachers and surveys of recent high school graduates, she came to have a more nuanced view.

Let us look at the questions students answer for one of these metacognitive activities to get a sense of how they are intentionally structured to help students examine their own thinking and behaviors. For building project-management skills, students answer these queries:

• Are you confident that you have the knowledge and skills to complete your research project? What is your greatest strength? Are you struggling with any aspect?
• Have you used your time effectively? If so, what is the secret to your success? If not, what would you do differently?
• Have you experienced setbacks? What are they? Have you found ways to overcome them, or are you still struggling?
• Have you used all the resources at your disposal? Where might you find resources that will help you work more effectively?
• What’s the biggest surprise about the research process so far?
• What’s one adjustment that you can make to effectively complete your research project?

This kind of exercise provides a “check-in” during the research process, but it should not be the first time project management is addressed. Early in the term, the student researchers are provided with a timeline checklist with target dates for completing steps in the process. These projects engage complex activities, but by breaking them into steps, students feel that they are more doable. A research project of this complexity cannot be a “midnight wonder” that is developed overnight. Rather, it requires careful attention at each stage, beginning with the students being certified in human-subjects research and understanding institutional review board (IRB) policies.

The following is the timeline that students receive to track progress on their projects. I note when assigning this checklist that authentic research does not always proceed as planned. Disruptions happen. Some parts of the research project may be completed sooner than expected while others trail. On the other hand, some due dates are “drop dead,” such as when students must have a high-quality poster or presentation ready for public dissemination. (This is a sample timeline submitted mid-term for “checking in on progress” with notes that are anonymized from various student responses.)

English 3470

Research Project Checklist

Order	Task	Projected Date of Completion	Actual Date of Completion	Notes
1	Completion of IRB Certification	September 11	September 11	I learned a lot getting this certification, principles I otherwise wouldn’t have ever known.
2	Annotated Bibliography	September 17	September 17	I found some insightful sources, and a clear gap has been identified.
3	Research Proposal Draft	September 24	September 24	I feel confident and excited in where my research is headed. I’m eager to find out results.
4	Research Proposal Final With Review of Literature	October 1	October 1	I continue to feel confident that my project is going to provide some insightful data. It is a relatively new topic, which makes me happy that I thought of it.
5	Qualtrics Survey, Recruitment Script & Letter of Information	October 9, October 4, and October 12	October 9, 4, and 12	The Qualtrics survey questions turned out well, and at the moment, I don’t think I missed important points. The recruitment script and letter of information have all been approved. I have fallen behind in not yet sending out surveys. I need to follow my own timeline and get them out. It’s been a busy week, which is no excuse.
6	Interview Questions, Recruitment Script & Informed Consent Letter	October 4, October 5, and October 15	October 4, 5, and 15	Getting to interviews is exciting. Ready to go: recruitment script and consent letter. Just like the Qualtrics survey, I am a bit behind in actually extending the invitation but will send tonight. My target date to complete the interview is October 30.
7	Draft of Research Project	November 1
8	Abstract	November 3
9	Research Poster	November 22
10	Final Research Project	November 22
11	Prezi or PowerPoint for Oral Presentation	November 24

Checking in on progress is crucial to build essential skills. As noted in Learning and Leading With Habits of Mind (Costa & Kallick, 2008),

Efficacious people stick to a task until it’s completed. They don’t give up easily. They are able to analyze a problem, and they develop a system, structure, or strategy to attack it. They have a repertoire of alternative strategies for problem solving, and they employ a whole range of strategies.…They have systematic methods for analyzing a problem, which include knowing how to begin, what steps must be performed, what data must be generated or collected, and what resources are available to assist. (p. 18)

The framework of the course, as already mentioned, divides the complex research project among multiple assignments. These are the “steps [that] must be performed.” This overarching framework ensures that students can finish with success as long as they keep pace. It is this kind of approach that removes the mystery of “research” and makes the process transparent. Likewise, it provides students with a framework to complete future projects independently.

Thinking About Thinking and Its Assessment

With the overlay of E-CURE, the Canvas learning-management system (LMS) illustrated below shows how its elements were integrated into the 15-week semester plan (Figure 19.1). A quick aside: E-CURE can be integrated into a shorter timeframe. These stages help keep students on track as they pause and reflect on their progress and any setbacks.

Figure 19.1

Screen Capture of CURE Assignments Embedded in English 3470

The course begins with an assessment of students’ past research experiences and skills, showing them that they can apply past knowledge to new situations. For instance, since these students are typically adept at critical analysis of literature, I explain that empirical research is much like a story with its own characters, plot, conflict, and narrative. This image begins to replace their pre-conceived idea that Research (capital letter intentional) takes place in laboratories.

Students in English 3470 are also apprehensive about the QI requirements, including developing infographics. How do we approach graphing data? Easy. For a week, they track every kind of writing they do, from text messages and class notes to browser searches. This accomplishes three objectives: first, they see how writing pervades their lives; second, they begin to question how to quantify data (e.g., volume, time); and third, they learn how to use an Excel spreadsheet and turn those data into a pie chart or bar graph. Seriously, when students first see the data in a spreadsheet turned into a graph, audible gasps are heard in the classroom. It is almost like magic to many of them. This simple exercise also gets at how data are portrayed, and as students experiment with Excel’s many chart options, they see that appearances can be helpful or even deceiving. This is an important lesson in being savvy about reading infographics, a skill that they will carry past the end of the semester.

Once that baseline of previous experience is established, students are asked to choose their top five outcomes from a list of 10 provided by the project (Figure 19.2). According to the PI of this grant, typically, the faculty member chooses the outcomes; however, in this case, I felt that the students would feel greater ownership of the project if they had a say. Perhaps not surprisingly, autonomy ranked in the top five; their other choices included communication, creativity, critical thinking, and intellectual development. While I would not necessarily have chosen creativity, they believed it essential to conducting research and presenting it in meaningful—even aesthetically pleasing—modes.

Figure 19.2

Outcome Categories

CURE faculty can add up to three course-specific outcomes. Source: Evaluate UR-CURE, “E-CURE Orientation Spring 2022 Presentation,” https://serc.carleton.edu/evaluateur/cure/resources_using_e-cure.html. Used with permission.

On the other hand, I chose the metacognitive activities. Developing research questions is a primary skill that occurs early in the term. Students in this class generate a list of three possible topics, and through individual conferences, they settle on one. The first metacognitive activity, then, focused on “asking good research questions.” Students can reflect on the intensive but productive give-and-take with the instructor as they refine their thinking about their topics. The goal of the activity is not only to develop good research questions but to learn how good research questions can be formulated.

Following closely on, students develop a review of literature, which is good fodder for the second metacognitive reflection, “reading as researchers.” English majors are accustomed to summarizing secondary literature, but the “literature” that they are asked to read and summarize for this course is research literature such as scholarship in higher education. This is not literary criticism of Pride and Prejudice but, more typically, articles such as “Lexile Reading Growth as a Function of Starting Level in At-Risk Middle School Students” (Archer, 2010) or “Evaluation of Charm Factors of Short Video User Experience Using FAHP—A Case Study of Tik Tok App” (Feng et al., 2019). The metacognitive activity prompts students to become aware of differences in these types of reading contexts and thus the potential need to alter their reading strategies.

Because students work collaboratively on the whole-class research project that provides scaffolding of the skills to undertake in their individual projects, an apt metacognitive activity was to ask about how they would structure teams effectively. Students responded to these questions:

• In your view, what does effective teamwork look like?
• What are the signs that things are going well?
• What are three ways you might distribute tasks between team members? Which do you consider the best? When conflicts arise between team members, what’s the best way to resolve them?
• What’s the best part of being on a team?
• What do you try to get out of the research team experience?

One student responded to the first question like this:

Effective teamwork not only involves communication, but also responding. There’s no point in communicating your ideas to someone if that other person doesn’t take them into account. An effective team doesn’t have a dictator as a leader. “My way or the highway” does not apply in team activities as that approach prevents the final product from reaching its full potential.

While only a brief illustration, this response suggests the potential for influencing future workplace interactions. Likewise, successful project management is as crucial for a research project as it is for a career. Students develop timelines for stages of their individual project to “hit their marks.” When asked how they deal with “setbacks,” one student demonstrated how she analyzed her own work. “Usually, I just keep going, and the answer becomes clear because of other factors in the assignment,” she wrote. “I just keep trying versus stopping with indecision and that gets me further.” The metacognitive activities reinforce and also interrogate students’ own thinking. From the faculty perspective, answers to “what was your biggest surprise?” were thought-provoking. The students were “stunned” by what they were able to accomplish and the skills they had acquired. By the end of the semester, they had completed a research report, developed a poster, and created a lightning talk. They present either the poster or the oral presentation at a campus symposium.

Finally, following the public Student Research Symposium on campus, students responded to a set of questions about “communicating research,” which is in line with the Habit of Mind of thinking and communicating with clarity and precision. Students found the dissemination of their work with a “real audience” empowering. Judges and others looked seriously at their projects with genuine interest. They asked questions and suggested additional lines of inquiry. As the students put it, “We felt like real researchers.” One said, “I didn’t want to do the symposium because it was a little terrifying, but I felt on top of the world when I was presenting.” Metacognitive questions asked them to evaluate their own learning, what “worked” in communicating findings and what “needed work.” They pinpointed that the “steps” (or “stages”) approach to a complex research project made it “doable,” and they realized that they could use these strategies for future projects. They were also pleased to have acquired skills in how to conduct research ethically, protect human subjects, structure surveys, and conduct interviews.

How did this course affect the students’ Habits of Mind? Perhaps one of the most evocative comments was from the student who said, “I see research questions everywhere.” Much better than seeing dead people! For this student, the world became a different place as she asked why, what, and how? Students noted that they became “systematic observers,” skills honed during an ethnographic exercise in taking field notes, watching a particular scene for 30 minutes, describing and analyzing.

English majors are not known for their skills in quantitative measurement; however, the consistent work with spreadsheets, data, and infographics led one student to move from being “daunted” by numbers to turning into a “statistics skeptic.” They believed that they moved from being cowed by data to being able to make data-informed decisions. After enrolling in the course, they know to use multiple statistical indicators, such as median, mode, mean, and range.

Identifying as researchers, these students are empowered by understanding responsible conduct of research (RCR) and being certified in human-subjects research. They have learned to frame questions to obtain the most applicable information through surveys and interviews. These are skills that will serve them well, whether pursuing graduate studies or entering the workforce. Accustomed to the rhetoric of research with its emphasis on shared structures and vocabulary, these students can make a smooth transition to graduate work. As one student, a practicing secondary school English teacher, wrote a couple of years following her experience, “Yesterday, I had my first graduate education research course. I felt transported to our research class. I was so pleased to be familiar with almost all of the terms we covered on the first day.”

Although this English class used reflection in previous iterations, as the examples above indicate, it was never as structured or in such depth as the metacognitive element of EvaluateUR-CURE. This pilot study is still gathering data but suggests that well-timed and efficient questioning of research processes for students can have an impact on learning and retention of information and processes. Pilot testing continued in the 2022–2023 academic year, and I signed up for a second round. Extensive resources and training are provided for faculty participants. They, too, engage in metacognitive activities as the project calls for formative and summative assessment from them. For each metacognitive exercise, I submitted feedback on “what worked” and “what did not work.” Individually, I was influenced by student reflections and modified assignments based on my analysis of their responses. I appreciated a question that put them in a role as “expert,” offering advice to a more novice researcher. Likewise, the question “What has been your biggest surprise in the research process?” elicited interesting responses.

Habits of Mind are seen by Allen and Allen (2003) as the “cultivation of proficient humanity” (p. 19) and draws on the ancient concept of “liberal education” in which liberal meant liberated, as when free citizens of Rome could study the quadrivium and then the trivium—the seven liberal arts. AAC&U has also taken up this clarion call to emphasize the importance of a liberal education in its LEAP initiative: Liberal Education and America’s Promise. Habits of Mind do not exist in a vacuum but are part and parcel of assuring that students, through higher education, have access to ways of being and thinking as citizens and workers. The metacognitive activities described here can help those Habits of Mind stick.

References

Allen, W. B., & Allen, C. M. (2003). Habits of Mind: Fostering access and excellence in higher education. Routledge.

Archer, L. E. (2010). Lexile reading growth as a function of starting level in at-risk middle school students. Journal of Adolescent & Adult Literacy, 54(4), 281–290.

The Boyer Commission on Educating Undergraduates in the Research University. (1998). Reinventing undergraduate education: A blueprint for America’s universities. Carnegie Foundation for the Advancement of Teaching.

Costa, A. L,. & Kallick, B. (2008). Learning and leading with Habits of Mind: 16 essential characteristics for success. Association for Supervision and Curriculum Development.

Duersch, M., Bennett, M., Bess, G. Z., Bitner, J., Cook, S., Dotson, J., Edgar, D., Fleming, S., Jordan, K., Lyle, C., Pedersen, T., Peterson, J., Rudd, D., Simmons, M. Tovey, C., & Watson, N. (2018). The rise and fall of the blue book: An examination of essay exam books. Young Scholars in Writing, 15, 102–113.

Eralie, M. (2019, December 5). To stick or not to stick: A look into sticky notes as a study aid. USU Fall Student Research Symposium.

Feng, Y.-L., Chen, C.-C., & Wu, S.-M. (2019). Evaluation of charm factors of short video user experience using FAHP – a case study of Tik Tok App. IOP Conference Series: Materials Science and Engineering, 688, 055068. https://doi.org/10.1088/1757-899x/688/5/055068

Grinberg, I. Y., and Singer, J. K. (2021). ETAC ABET and EvaluateUR-CURE: Findings from combining two assessment approaches as indicators of student learning outcomes. Journal of Engineering Technology. https://d32ogoqmya1dw8.cloudfront.net/files/evaluateur/cure/etac_abet_evaluateur-cure_findings_from_combining_two_assessment_approaches.pdf

Haswell, R. (2005). NCTE/CCCC’s recent war on scholarship. Written Communication, 22(2), 198–223. https://journals.sagepub.com/doi/10.1177/0741088305275367

Kinkead, J. A. (2003). Valuing and supporting undergraduate research. Jossey-Bass.

Kinkead, J. A. (2016). Researching writing: An introduction to research methods. Utah State University Press.

Kinkead, J. A. (2019). An empirical research project in English and writing studies. Prompt, 3(2), 39–68. https://www.thepromptjournal.com/index.php/prompt/article/view/40/86

Kinkead, J. A. (2021). The empirical strikes back: A RAD research methods course for undergraduate English students. CEA Forum, 49(1), 118–145. https://journals.tdl.org/ceaforum/index.php/ceaforum/article/view/7202

Kuh, G. D. (2008). High-impact educational practices: What they are, who has access to them, and why they matter. Association of American Colleges and Universities. www.neasc.org/downloads/aacu_high_impact_2008_final.pdf

Lopatto, D. (2009). Science in solution: The impact of undergraduate research on student learning. Council of Undergraduate Research.

Scharff, L., Draeger, J., Verpoorten, D., Devlin, M., Dvorakova, L. S., Lodge, J. M., & Smith, S. (2017). Exploring metacognition as a support for learning transfer. Teaching and Learning Inquiry, 5(1), 78–92. https://doi.org/10.20343/teachlearninqu.5.1.6

Seymour, E., Hunter, A.-B., Laursen, S. L., & Deantoni, T. (2004). Establishing the benefits of research experiences for undergraduates in the sciences: First findings from a three-year study. Science Education, 88, 493–534.

Singer, J., Weiler, D., Zimmerman, B., Fox, S., & Ambos, E. (2022). Assessment in undergraduate research. In H. A. Mieg, E. Ambos, A. Brew, D. M. Galli & J. Lehman (Eds.), The Cambridge handbook of undergraduate research (pp. 158–171). Cambridge University Press.

Singer, J., & Zimmerman, B. (2012). Evaluating a summer undergraduate research program: Measuring student outcomes and program impact, CUR Quarterly, 32(3), 40–47.

Stringham, S., Kirkham, A., Jenkins, K., Piscione, A., McMurray, A., Chapman, K., Horning, A., Kent Topalian, B., Mack Ingram, C., Mair, A., Satterthwaite, S., Sorensen, M., & Brockbank, T. (2022). Formalized curiosity: Outcomes of an empirically based research methods course for English majors. College English, 84(6), 546–569.