24 Revisioning a Bibliography Assignment to Center Discovery and Critical Source Engagement

In her response to college students’ growing access to ChatGPT, Annette Vee (2023) differentiated between the kind of “research” that large language models can produce and the research process that students undergo in classes like first year writing. In “Large Language Models Write Answers,” Vee emphasizes how the answers that these models produce diverge from the discovery process that we hope students will experience during a research unit. Afterall, our focus in teaching scholarly research is not really on the writing or the answers so much as the experiences of critical inquiry and productive uncertainty that teach students what it is like to be immersed in a research process. Vee emphasizes how her first-year writing program’s goals at the University of Pittsburgh included writing for critical inquiry and productive uncertainty, outcomes that we share in our program at Marquette University, a mid-sized private university in the Midwest of the United States. In fact, the first course outcome for our program is that students, “Develop ideas and arguments shaped by a process of inquiry.” And like Vee describes, over time we have shifted away from the emphasis on arguments to highlight the importance of evolving ideas.

However, during Spring 2023, when Lilly and Jenna—Director and Assistant Director of our first-year writing program—sat down to review our curriculum and consider what assignments might be most vulnerable to AI-generated writing, our bibliography assignment was an obvious contender. This assignment, while oriented around a discovery process and an evolving research question, still focused heavily on summary of individual sources, something ChatGPT can do quite well. Thus, we worked with our research librarian liaison, Nicole, to reimagine this assignment, guided by a conversation about learning goals. After reviewing relevant scholarship on source use and plagiarism, this chapter describes our process for revising the assignment, which we believe could be transferable to other projects, and then introduces our revisions and shares initial responses to the revised assignment from students and instructors.

Literature Review

Programs like ChatGPT both reinforce questions that writing teachers have been asking for years about student source use and introduce new perspectives and challenges. In Pluralizing Plagiarism (2008), Howard and Robillard argue that plagiarism poses three distinct problems for instructors. The moral problem is that plagiarism is often equated with cheating (fueled by “gotcha” computer programs like TurnItIn) and thus associated with bad moral character. The interpersonal problem is that instructors often see plagiarism as a challenge to their own intelligence or expertise – “Did you think you could pull one over on me?!” – which can undermine student-teacher relationships. And finally, the equity problem: students who are caught plagiarizing are often the most academically vulnerable on campus—international or first-generation students—and plagiarism can be used as evidence that they do not belong.

In all three cases, ChatGPT reinforces and complicates these problems. To the moral problem, TurnItIn has already created an AI detector that works very similarly to its plagiarism detector, highlighting portions of a student’s paper that may have been written by AI and spitting out a percentage for instructors. This once again positions the instructor in a policing role, also foregrounding the interpersonal problem. However, there is also a complicating factor that “proving” AI use is not nearly as clear-cut as proving plagiarism. There is no original text that instructors can reference as evidence; they can only go off the percentage produced by another algorithmic program. Meanwhile, the equity problem is complicated by questions of both dominant discourse and access. As the CCCC/MLA Working Paper on Writing and AI (2023) explains, “Students may face increased linguistic injustice because LLMs promote an uncritical normative reproduction of standardized English usage” (p. 7). At the same time, students’ unequal access to LLM tools can replicate social inequity.

In his final essay in the Pluralizing Plagiarism (2008) collection, Horner argues that we must move beyond developmentalist and archipelago views of plagiarism and towards a community assimilation view. While the developmentalist view recognizes plagiarism as a possible stage in a students’ writing development (i.e., patchwriting), the archipelago view interprets differences between accepted and unaccepted writing practices as differences in literacies and language practices. Both perspectives, Horner argues, do not fully recognize that “there is a politics to how plagiarism is defined and to the levying of charges of plagiarism, shaping what does and doesn’t get identified as plagiarism with what consequences” (p. 171). To fully attend to these politics, we must recognize that learning to work with sources is a process of community assimilation, often facilitated by the assistance of community members in the real world. In this way, we can both value the agency of incoming members, who make choices about their assimilation and can create change and acknowledge the power dynamics between new/old members, resulting in shared responsibility for the transition.

But what would it mean for a research writing assignment to take a community assimilationist view of source use and plagiarism, especially in relation to AI? How can we center the process of inquiry that we believe is key to a research experience in the academy while not completely disregarding the writing? These were questions that guided our revision process as we sat down together to consider our research journal assignment. In the next section, we briefly overview our first-year writing curriculum and this assignment before delving into our revision process.

Course Context

Our first-year writing course, which about 70% of our incoming first-year students take, is housed within the English department and is the only required English class in the university’s core curriculum. In Summer of 2020, we revised course assignments to better meet the needs of a diverse student body by creating an anti-racist curriculum that includes a shared first unit that focuses on cultural rhetorics. Our initial curricular revisions were aimed at supporting incoming first-year students in having complex conversations about racial justice as they developed college writing skills.

The course curriculum currently features four separate units. All instructors are required to adopt the anti-racist cultural rhetorics framework for Unit 1 and assign three one-page analyses. In Unit 2, instructors choose their own theme, but all students work to write an academic synthesis essay using texts curated by the instructor. During Unit 3, classes work alongside an assigned university librarian to learn about academic research, particularly finding and evaluating sources. Students can choose their own research question and sources as they catalog their learning in a research journal. In Unit 4, students utilize their findings from their research journal to compose a creative project with a critical reflection. Students are tasked with demonstrating their rhetorical knowledge by selecting an appropriate genre to convey their findings to a particular audience. Popular student project genres include websites, brochures, infographics, videos, podcasts, and social media campaigns. Additionally, all instructors are asked to engage their students in sustained critical reflection throughout the semester, including an initial reflection on the first day of class; a reflection at the end of Unit 1, about a month into the semester; and a final course reflection.

Prior to the Spring of 2023 semester—and before the public release of ChaptGPT—the course’s third unit, entitled “Research as Discovery,” engaged students in writing a research journal. Students posed a research question of their choosing (sometimes in line with the course’s theme), worked with support from their class librarian to learn how to find and evaluate both academic and popular sources, and wrote annotated bibliography-style summaries of approximately ten sources. In these entries, students were asked to provide a citation and write a summary of each text in their journal. They were encouraged to create a journal that provided a variety of sources—both in perspective and genre—to foster a more comprehensive, holistic understanding of their research question.

While it confused some students whose high school careers had prepared them for argumentative writing, we did not assign a thesis-driven academic research paper. Echoing Vee’s (2023) goals of building critical inquiry and productive uncertainty, we aimed to help students listen to others’ perspectives, question their own researching processes and practices, and reflect on their own positionalities before arriving at a conclusion. The unit concluded with a reflective assignment where students narrated their learning, responding to questions such as, “How did you find your sources? How did you evaluate your sources? What did you learn while reading (and rereading) your sources? How did the sources’ different positions affect your own thinking over time?” Some instructors also assigned students to visually map relationships between sources to help them better contextualize information and synthesize ideas across sources.

Revision Process

After deciding the research journal assignment would need to be revised in response to growing access to ChatGPT, the authors met to clarify assignment learning outcomes. We worked together to think through what we hope students are learning to do throughout their lives as researchers and writers: assess the original context of a source and its potential fit for their own purposes as composers/writers/rhetors. Through this conversation, we posed the question to ourselves: “What reflection or connection making can students do while deprioritizing the essential skill of summary, which ChatGPT does so well?” Ultimately, we revised the research journal assignment to integrate higher-order thinking and evaluation skills.

Students at the first-year level are usually unfamiliar with potential strengths and weaknesses of sources—sources are either “good,” in that they fulfill assignment requirements, or “bad,” in that they do not. As students develop beyond this dualistic thinking, they begin to understand all sources are valuable ways to build their understanding of a topic. We hope to give them experiences and encounters with a plurality of perspectives, including those that corroborate, contradict, or complement each other, to build deeper knowledge of a problem or issue.

This exposure to the necessity and utility of all types of sources positions students to progress toward course learning outcomes (refer to Appendix 1 for course outcomes). Considering this knowledge of where students are when they start the course and where we intend them to be by the end of the course, we chose to invite students to focus on assessing the credibility and authority of sources they find. We brainstormed an initial checklist of potential positive and negative characteristics of sources. Then we met with the cohort of instructors who would be incorporating the revised assignment into their courses. The instructors brainstormed additional, neutral characteristics of sources and organized them into three categories: characteristics related to the author, the evidence presented, and the dissemination or publication platform. This ensured the assignment did not feed into dualistic thinking about “good” and “bad” sources.

To encourage students to make connections between their sources and their thinking, we added five reflection questions for them to answer. The questions require students to draw on their personal experience conducting research, qualitative assessment of the experience, what is going well and what is not, and begin assessing sources for eventual inclusion in the final project. While some instructors were already using the research map (Appendix 2) with a culminating reflection for Unit 3, we also wrote this map into the assignment prompt and provided some sample maps for instructors to show students, which led to more uptake of the map across course sections.

Revised Assignment

The revised version of the assignment kept the structure of having several journal entries, each describing a set of 3 sources. Students were asked to provide specific information for each source and draft a synthesizing paragraph for each “set” of sources (typically 3). After completing 4-6 entries, they also produced a final Research Map and Reflection (refer to Appendix 2 for full assignment). There were 3 parts for each source entry:

1. An MLA citation
2. A series of checkboxes or list of possible hashtags for students to indicate the author, type of source, and type of evidence used in their source
3. A set of findings from their source including a quote, a fact, and a “bonus” finding

We gave students the following example, which uses hashtags:

1. Brandt, Deborah. “Sponsors of Literacy.” College Composition and Communication 49.2 (1998): 165-185.
2. #scholar #scholarlyjournal #qualitativefindings
3. Findings

• • “Sponsors, as I have come to think of them, are any agents, local or distant, concrete or abstract, who enable, support, teach, model, as well as recruit, regulate, suppress, or withhold literacy—and gain advantage by it” (166)
  • Theory based on 100 interviews with people born between 1900-1980 (167)
  • Raymond Branch and Dora Lopez—both pursuing literacy on their own in Madison, WI in the 1980’s; Branch in computer science and Lopez in Spanish.

Then, we asked students to include a 250-word synthesis paragraph for a set of sources found during one search period. These paragraphs considered source context, key take-aways, reliability, and next steps. Rather than asking for a summary, the synthesis paragraph was specifically oriented towards their emotional reaction to information in the sources (“What was interesting? Surprising? Maddening? Awesome?”) as well as the sources’ relationship to their research question. We asked students to hone in on how this information would build their ongoing learning about their subject as well as help them evaluate whose voices were being centered in their research. This was a frequent topic of conversation in class, given the first unit’s focus on Cultural Rhetorics.

After completing all of their journal entries—typically 4-6 entries and 12-18 sources depending on an instructor’s goals and pacing—students would create a research map of all of their sources to work on identifying relationships between them. We encouraged the use of Bubl but also gave students the option to hand-draw their maps or make use of another computer program for their designs. Based on this research map and all of their journal entries, students wrote a 2-page double-spaced reflection considering the evolution of their research question, how sources connected to one another (with specific reference to their maps), and what they learned and would do differently in the future. Again, our focus was not on summarizing findings but on reflecting on the experience of research itself and synthesizing connections across sources. Given the quantity of sources at play and the need to focus their findings around their research question, this is a task that students will be much more equipped to take on while large language processing models likely will not.

Student Work

We piloted these revisions in several sections in Spring 2023, and during Summer 2023 we introduced our whole instructor team to the updated assignment parameters and sample work. As of this writing, we have integrated the assignment into all our sections for Fall 2023. Overall, our shifts in the assignment structure meant that we saw students explicitly identifying source types and reflecting on authors and types of evidence. While these had always been questions in the research journal assignments, the basic source information was often overshadowed by bigger questions about relevance or findings. The checkbox/hashtag model really encouraged students to address these questions directly. In Lilly’s class, however, some students seemed confused on whether they should include a hashtag from each of the 3 broader categories; instead, many started inventing their own hashtags. The checkboxes, although they took up more space on the page than hashtags, were therefore more straightforward in helping students to categorize information clearly. Another clear advantage of putting these categories in the forefront was that it helped students make direct connections to the Unit 4 assignment. Explicitly identifying the genre they were reading, as well as considering how information was communicated in that genre, prepared them for choosing the genre for their project.

Of course, it’s hard for us to say whether students engaged less with large language-based processing models than they would have before our assignment revisions, since we implemented the revisions alongside the rise of ChatGPT in Spring 2023. However, since we did have students who tried to use AI on the synthesizing paragraphs of the journals, we could observe how this did not work well for the assignment. In Lilly’s class, one student who used AI for these paragraphs wrote broad generalizations about the value of public sources (since one entry was focused on public sources) to research that included little direct connection back to his research question or even to specific findings. We suspect that the kinds of questions we focused on in the synthesis and summary did not lend themselves well to ChatGPT.

Meanwhile, thinking back to the critical thinking skills we hoped to target with this assignment, specifically evaluation of sources strengths and weaknesses, we did see improvement in students’ ability to assess their sources and a move towards more holistic assessment. For example, in Lilly’s class, a number of students noted in their final reflections that they had not considered how first-person narratives could be part of scholarly research. However, since that category was on the checklist and was specifically targeted in one of the class’ journal entries, students were encouraged to recognize the value of including perspectives from those who had personally experienced a topic even if they were not scholarly experts in the field. A number of them chose to find more first-person sources in their final “grab-bag” entry as well.

From a practical standpoint, it is impossible for instructors to read and be familiar with every source in students’ journals because each student selects their own research questions and sources. Most sections enroll 19 students, and most instructors teach two to four sections; hence, the teaching load is too large for instructors to read and monitor each student’s sources. This approach gives students agency and authority over their learning and research process; often the students will know more about their topic than their instructors, rendering the student an expert who can help teach the whole class.

This structure also nudges instructors away from acting as authoritative, omniscient gatekeepers, which helps combat the interpersonal plagiarism conflicts Howard and Robillard describe (2008). As Johnson (2023) explains, “much of the discourse centered on plagiarism and academic integrity positions student writers as purposely deceitful and mercilessly unethical. This has been concerning in terms of the types of assumptions made about students as well as the implied thrill of catching a student using AI and LLM-generated writing” (p. 172). The student-instructor relationship Johnson (2023) describes does not help motivate students to deeper understanding of their research skills nor awareness of their rhetorical agency.

Reviewing initial student work, Jenna experienced tensions between evaluating student process and product. As Vee (2023) reminds, the aim of such inquiry assignments is not a polished product, but critical self-awareness of processes and “the pleasurable difficulty of writing and reading” (p. 177-178). Some students were very much learning how to synthesize and engage with academic texts; the assignment stretched their skills resulting in uneven success among students. While some grappled with complexity and synthesis across sources, other students struggled to explain their learning and connections that they noted in their research. This observation does not dissuade us from our assignment approach but reminds us that researching is challenging and a new skill for many of our students. Like all learners, our students need time and practice to succeed at critical inquiry, source evaluation, and synthesis, often more time and practice than just one semester can afford.

Conclusion

While we do not claim to predict every way the AI and LLM models will influence student research, writing, and learning going forward, we are encouraged by how these revisions helped students achieve the course learning outcomes. Particularly, the shift away from summary-based bibliography entries toward more source evaluation and synthesis across information is aiding students in engaging with their own research practices and processes, becoming more aware of their own positionalities, and examining their own rhetorical contexts and agency. Meanwhile, we encourage ongoing collaborations between writing instructors and librarians, who can both bring valuable perspectives to assignment evaluation and redesign. Despite the unknowns that the advent of AI brings, we contend that a writing curriculum that encourages critical thinking, investigation of research practices, source evaluation, and reflection can continue to help students navigate inquiry through reading and writing.

References

Horner, B. (2008). Afterword: Plagiarism, difference, and power. Pluralizing Plagiarism: Identities, Contexts, Pedagogies. Portsmouth, NH: Heinemann, 171-178.

Howard, R. M., & Robillard, A. E. (2008). Pluralizing Plagiarism: Identities, Contexts, Pedagogies. Portsmouth, NH: Heinemann.

Johnson, G. P. (2023). Don’t act like you forgot: Approaching another literacy crisis by (re)considering what we know about teaching writing with and through technologies. Composition Studies, 51(1), 169-218.

Modern Language Association/Conference on College Composition and Communication Joint Task Force on Writing and AI (2023). Working Paper: Overview of the Issues, Statement of Principles, and Recommendations. https://aiandwriting.hcommons.org/working-paper-1.

Vee, A. (2023). Large language models write answers. Composition Studies, 51(1), 176-181.

Appendix 1

Course Learning Objectives
Upon completion of this course, students will be able to:

1. Develop ideas and arguments shaped by a process of inquiry and understanding of sources, including academic and nonacademic texts that represent both dominant and underrepresented perspectives​

2. Demonstrate rhetorical knowledge through their ability to analyze contexts and audiences and create tailored multimedia texts that represent information accurately and accessibly​

3. Engage in a process of writing including overlapping phases of invention, synthesis of ideas and information, and revision undertaken in response to others’ feedback and self-evaluation​

4. Reflect on their performance and growth as ethical rhetors striving for effective written and spoken communication given the rhetorical situation and standards set by this course.

Appendix 2

Unit 3: Searching

Research—the genuine search for truth and knowledge—is a cornerstone of Jesuit education. We need to be knowledgeable in our disciplines before we can discern actions and enact the changes we need to transform our world. While you may have written a research paper before, this assignment departs from that genre because it asks you to question your own researching processes and practices.

First, you will choose a problem related to literacy and/or education that you genuinely want to know more about. Then you’ll provide a narrative of how you went about trying to understand that problem. How did you find your sources? How did you evaluate your sources? What did you learn while reading (and rereading) your sources? How did their different positions affect your own thinking over time?

As you seek to learn more about your chosen research question. Your challenge is to understand new information while also rhetorically analyzing your sources with particular attention to personal, local and global contexts. This not a thesis-driven essay. You don’t start by reaffirming your final, considered opinion. Try starting with rhetorical listening, intellectual curiosity, and an open mind.

**The main assignment for this unit is NOT a fully composed research paper. Instead, it is a research journal that tracks the development of your thinking and mentions at least 12 new sources you found on the way.**

PART I: SOURCE ENTRIES & SYNTHESIS (80 points total/20 points each)

Your research journal will consist of four journal entries based on 3 sources each (totaling 12 sources) and then a final summative reflection. Entries should be single-spaced and include the following information:

1. SOURCES: Find at least 3 useful sources that emerged from this search. Keep in mind that sources can be “useful” even if you don’t plan to include them in your project, because they can help you understand what to do next.

a. Provide an MLA citation and provide at least one hashtag for each category below (ie #firstperson #scholarlyjournal #qualitativefindings)

• • • • Author: First-person account; Journalist; Scholar (advanced degree) in the field; Advocacy Organization; Other
      • Source: Scholarly journal; General Interest Magazine/Journal; Newspaper or Online News Site; Non-profit organization (.org); Government organization (.gov); Educational organization (.edu); For-profit company (.com); Other
      • Evidence: Historical Context; Personal Experience; Statistical Evidence; Qualitative Findings (interviews, focus groups, etc.); Quantitative Findings (experiments, statistical modelling, etc.); Other

b. Include at least one of each of the following from all three sources

• • • • A quote
      • A fact
      • A “bonus” thing

2. SYNTHESIS: Write a 250 word paragraph (not a bulleted list) summarizing your experience with the search and how your three sources contribute to your project. Consider:

• • Context: When and where did you conduct the search? What search engine and keywords did you use? Any difficulties with the search itself?
  • Take-aways: What were the most useful things you learned from these sources? What was interesting? Surprising? Maddening? Awesome? Which one was your favorite?
  • Reliability: How reliable are these sources and whose voices are amplified? Do you have perspectives from people with first-person experience with your topic?
  • Next steps: How well are your sources answering your research question? Where might you go next to continue answering your question?

Part II: Research Map (20 points)
After you have completed your four research entries with 12 sources total, create a research map to show the connections you see between your sources. You can use internet tools or even hand draw your map if you’d like. I’m recommending Bubbl, but you are not required to use it. Bubbl will allow you to create three (3) maps for free. Be sure to register for the free version.

**The purpose of the map is to create a visual depiction of the sources you found. You should include ALL sources you plan to cite.**

PART III: SUMMATIVE REFLECTION (50 points)

Finally, you will write a 2-page double-spaced summative reflection of your research experience. The goal of this reflection is to articulate to yourself and to me how your research process proceeded during this unit, what you have learned, and where you would like to head next.

Consider the following questions to inform your response:

• What is your research question? How has it changed since you began?
• How did you choose your sources? Would you choose differently in the future?
• How and why do these sources connect with one another? (reference your map as evidence of these connections)
• What did you learn? What did you find most surprising, interesting or confusing?
• What do you still need to know? In other words, what research might you still need to do?

Keep in mind: Even though they are engaging with academic sources both the journal entries and summative reflection should be written in a conversational tone. The goal here is to get comfortable talking about academic topics and discussing sources in a way that feels authentic to you. You are welcome (and even encouraged) to use personal pronouns, slang, and words or phrases from other dialects or languages. Of course, I still expect your language to be respectful and appropriate for a classroom context, but I hope you will use these entries and reflection to make the source material your own. Next unit, we’ll practice translating these findings for an audience of your choice.

About the authors

name: Lillian Campbell

institution: Marquette University

Lillian Campbell (she/her) is Associate Professor of English and Director of Foundations Instruction at Marquette University. Her research focuses on rhetorics of health and medicine, feminist rhetorics, and embodied writing pedagogy and can be found in Written Communication, Technical Communication Quarterly, and Composition Forum, among others. She is currently working on a book that examines how newcomers learn rhetorical body work in the fields of nursing, physical therapy, and tele-observation.

name: Jenna Green

institution: Marquette University

Jenna Green (she/her) is a Teaching Assistant Professor and Assistant Director of Foundations Instruction at Marquette University. Her teaching and research focuses on digital and multimodal composing, literacy studies, and student success.

name: Nicole Bungert

institution: Marquette University

Nicole Bungert (she/her) is the Student Success Librarian at Marquette University Raynor Memorial Libraries. She has supported first-year writing programs since 2014. Her research focuses on information literacy teaching, learning, and assignment design.