26 Working Alongside, Not Against, AI Writing Tools in the Composition Classroom: a Dialectical Retrospective

The release of ChatGPT by OpenAI in November 2022 has generated intense discussion both within and outside of academia regarding the potential impact of Generative AI on higher education. ChatGPT is a Generative AI technology, powered by a Large Language Model (LLM), that can generate human-like text in response to prompts and queries. Within just a few days of its release, over one million users had experimented with ChatGPT, making it one of the fastest-growing software products ever released (Rudolph et al., 2023). Some have hailed ChatGPT as a revolutionary technology that could enhance learning and teaching in profound ways, while others view it as a threat to academic integrity that enables new forms of cheating and plagiarism (Xiao et al., 2023).

Amidst the fervent debate sparked by ChatGPT’s release, as writing instructors we pondered a pressing question: How could we thoughtfully incorporate this potentially transformative yet concerning technology into our classrooms? As specialists at UCSB dedicated to teaching writing and rhetoric, we recognized the need to actively experiment with and critically examine how generative AI could impact our pedagogy and students. Since ChatGPT emerged in November 2022, we have begun integrating it and other LLMs into our curriculum and lesson plans. While Dan Frank has long been engaged with researching the intersections of emerging technologies and writing instruction, Jennifer Johnson has until now had a tendency to proceed cautiously with new technology yet has quickly come to acknowledge the importance of grappling with this advancing AI firsthand. Despite our contrasting backgrounds, we shared a commitment to exploring ChatGPT as thoughtfully and ethically as possible, and this shared commitment led us to collaboratively develop activities and assignments aimed at fostering critical thinking skills and transparent AI use among our students. We knew it was our duty as educators to confront both the potential benefits and the risks as we sought to incorporate and evaluate LLMs’ evolving role in higher education.

We’re certainly not the only ones who have been reflecting on these classroom possibilities. Chan and Hu (2023) argued that understanding student perceptions of AI can ensure responsible use and complement traditional teaching (p. 13). Rahman and Watanobe (2023) noted that the natural language capabilities of ChatGPT make it “intuitive and learner-friendly” and allow students to “receive interactive help from ChatGPT anytime and anywhere” (p. 5). They suggested that ChatGPT can help develop skills like reading, writing, problem-solving, language learning, and report writing through customized explanations, exercises, and feedback. Domenech (2023) explained how ChatGPT can act as a “personal tutor” by tailoring guidance and study plans to individual students’ needs and learning styles (p. 342). In addition to personalized learning, generative AI shows promise for enhancing teaching approaches and content creation. Domenech (2023) gave examples of how instructors can prompt ChatGPT to generate custom quizzes, open-ended discussion questions, explanations of concepts, and educational games to engage students across disciplines (p. 342). Rahman and Watanobe (2023) added that AI can rapidly generate quizzes at varying difficulty levels, freeing up educators’ time (p. 8). Above all the others, we’ve gained tremendous insight from Dr. Ethan Mollick, who with a strong presence on Twitter has been a pioneering thought leader in educational takes on GPT technology. In two published papers, Mollick and Mollick (2022, 2023) outlined pedagogical approaches such as using AI to provide feedback on student work, act as a tutor or coach for metacognition, and create opportunities for students to “teach” the AI as a powerful learning technique. The consensus here is that Generative AI and LLM technology has tremendous pedagogical potential, if it is handled, discussed, taught, and critically considered carefully in the classroom.

Based on these findings, we decided several key principles must guide our curricular approach:

1. Use Generative AI as one supplemental tool among many, not as a replacement for student writing. Combine it with peer feedback, instructor comments, and other resources (Berdanier & Alley, 2023).
2. Promote academic honesty by clearly communicating expectations for ethical AI use. Students should verify AI information, acknowledge AI assistance, and maintain their own voice (Halaweh, 2023).
3. Foster critical perspectives by facilitating discussions on AI biases, inaccuracies, authorship issues, and more. Understand and teach that Large Language Model technology such as that which powers ChatGPT does not actually “think” or “hold knowledge,” but instead simply repeats word-patterns based off of the context of its immense training data (Collins et al., 2022).
4. Exercise severe caution with AI assessment tools, which can be inaccurate, easily fooled, and unfairly accuse students with false positives. Rely more on holistic evaluation strategies (Rudolph et al., 2023).

These points have served as guidelines for embracing LLMs as new educational aids while bringing them “into the light” via ethical usage policies and critical perspectives. We aim to foster students’ understanding of how Generative AI works, what it excels at versus where it falls short, and the importance of verifying its output. With this pedagogical framework in mind, we each developed experimental activities using ChatGPT and other LLM tools in our classrooms. Here we share these lesson plans and reflect on how they were received based on student feedback.

Dan: In classroom activities, I ask students to experiment. I invite them to choose among the following prompts. They play with the tool, generate, collaborate, discuss, and evaluate what they get from the back-and-forth with the machine:

1. Choose at least three of the following tasks to complete as a group (or come up with other ones), utilizing ChatGPT (or another LLM):

• • Brainstorm a list of potential topics for an upcoming writing assignment.
  • Ask ChatGPT to develop a compelling opening paragraph for a chosen topic. Include specific commands for thesis statements, academic vocabulary, etc. Ask it for a few iterations.
  • Generate a fictional dialogue between two characters with distinct voices.
  • Blend, transpose, and/or synthesize content/genres; translate a poem or a passage from one genre to another (e.g., from prose to poetry or from a news article to a short story).
  • Use ChatGPT to suggest improvements or revisions to a paragraph of writing.
  • Experiment with different prompts to see how ChatGPT responds to variations in input phrasing.
  • See how ChatGPT responds to requests for writerly verbs: (“Add a hook”, “Defend this position”, “Use figurative language”, “Use Pathos”, etc.)
  • See how ChatGPT iterates on and revises multiple versions of a piece of writing through your instructions.
  • Ask ChatGPT to explain why it made the rhetorical / writerly choices it made
  • Ask ChatGPT to find two grammar issues in a piece of writing and talk about them; Ask ChatGPT to write a paragraph that includes a grammar mistake.

2. As you complete each task, make note of any strengths and limitations you observe while working with ChatGPT.

3. After completing the tasks, have a group discussion about your experiences. Consider the following questions:

• • What were the most helpful aspects of using ChatGPT?
  • What limitations did you encounter, and how did they impact your tasks?
  • How did the AI’s output differ based on the instructions and prompts you provided?
  • In what ways could ChatGPT be useful for classroom activities and lessons?

4. Finally, each group will present their findings and experiences. Share the tasks you completed, the strategies you employed, and your thoughts on the overall usefulness of ChatGPT in the context of academic writing and creativity.

I piloted this activity paired with a brief lecture on the nature of LLMs, how they work, what they can and cannot do, what to be careful about, and how they can be (potentially) useful, in several classes. Students are overwhelmingly engaged and interested in the content. It is worth restating that while it is essential to educate teachers to help them understand the tool, students also need this knowledge. LLM technology is easy to use but difficult to understand. By offering direct guidance on how to be critical of LLM output and how to think of the technology best as a remixing tool, and then pairing that information with hands-on experimentation, students can first learn the theory and then experience it through application. The prompts listed above are all designed to promote evaluation and critical response. This, I believe, is a crucial factor when working with the technology: it’s easy to get content, but it’s still on the student to think it through, practice discernment, consider multiple choices, and stay in control of the rhetorical purpose if that content is going to be effective.

I found that, when prepared in such a way, students approached LLM technology with a surprising sense of critical awareness and discernment. Some of my students produced content across a range of tools (such as ChatGPT 3.5 and then Claude) and compared the quality of their answers. Others asked for content in multiple forms (poem, scene, and dramatic narrative, for example), and used that as fodder for comparative and evaluative discussion. I was pleased to discover that by the end of the activity, students overwhelmingly preferred the cultivation of their own voices, but appreciated that the tool could help them move ahead if they felt stuck. There’s a certain sense of demystification happening in here that I think is really important.

Jennifer: Dan does an awesome job of inviting his students to both engage with and reflect on their experiences with the technology. Thus far, my approach has mainly centered on reflection, as I’ve been thinking that this metacognitive work will enable my students to see the affordances and limitations of LLMs and their output.

In my most recent classes, I have first opened a class discussion about ChatGPT, then invited students to engage with it throughout the course, and I’ve let them know up front that at the end of the course I will be asking them how and to what extent they used LLMs in their course work. I explain to them that I believe that the real learning happens in the articulation of how these tools help them be better writers or how they engage them in thinking about their writing and how it is operating. I tell them that they are welcome to use the tools at any stage of the writing process, but I want them to maintain authorial control and agency over their writing, i.e. I strongly discourage them from just copying and pasting whatever the LLMs produce, and instead I encourage my students to consider, revise, and expand upon the tech’s output until they feel that they have a rhetorically effective response to whatever prompt they are working with. At the end of the course, I asked them to answer the following prompt in their journals:

To what extent did you use generative AI such as ChatGPT to help you with this project or the activities for this class? How did it work for you? What worked well or what was frustrating? Do you feel skilled at using these tools or do you feel like you still need to develop your ability to use them effectively?

While about a third of the students in my most recent classes responded that they had not used the tech at all in their work for the class, I have been pleased and surprised by the range and depth of responses from those who indicated they had used it. I had imagined that they would tell me they had primarily asked it to either brainstorm or to refine their prose, but it turns out that their engagement with it is often a lot more nuanced than I had expected, as the following student responses will demonstrate.

Unsurprisingly, some reported that they had used it to make their prose more concise or to switch from passive to active voice. For example, one student wrote, “I used Generative AI such as ChatGPT to review my grammar and sentence structure throughout the course. [As] a storyteller at heart, concision has never been my strong suit. Thus, AI was extremely helpful in deleting unnecessary additives and achieving a more professional tone.” Yet that same student went on to note that,

One must know not to rely on AI alone. The technology is helpful, yet robotic, and often asserts an inhuman, mechanical writing tone when aiming for a professional writing tone. Thus, I found it better to use what I had learned in the course, referencing [the course] teachings frequently, and consulting Generative AI for finer details. Overall, I am confident in my ability to use technology in addition to, not in replacement of, my personal voice.

Other students indicated that they had prompted ChatGPT to generate several versions of a paragraph in a business writing class as a means of identifying the expected features of the text and mix-and-matching the sentences that seemed most effective in order to develop their work on the main assignment of the course, which in this case was a collaboratively written business plan:

I got ChatGPT to write three drafts of the marketing section. I was curious what it would come up with. I did it by asking it to “write in the style of a business plan a marketing section for a company that…” and then fed it the checklist you gave us for the marketing section. It gave a response in a bulleted form that didn’t really fit what I was going for, and it pulled a bunch of statistics out of nowhere, but it actually was really useful. I didn’t copy paste anything from ChatGPT into my draft, but it acted as sort of another sample marketing section that was customized to our business. It really helped give me an idea of what sort of claims and language was expected by the prompt.

In almost all of the affirmative responses, the students articulated their sense that AI-generated prose is often “repetitive and shallow,” at least until or unless they engage with it in the type of back-and-forth manner that Dan articulates above. Indeed, one student concluded that “one thing that was frustrating is its lack of versatility and flexibility. I have to write my prompts in a specific manner to make it work well. Therefore, the ability of writing effective prompts is the skill I need to develop to help me use it more efficiently.”

I’ve been heartened to discover in these responses that given the opportunity to engage with and reflect upon their use of ChatGPT, those who choose to employ it seem to be thinking in smart ways for themselves about how and when to use it most effectively as they strive to produce writing that works. I have not seen any evidence in my students’ submitted work or their reflections that they are mindlessly relying on LLMs to produce their work for them. Instead, I am seeing evidence that they are engaging with the tools in useful and productive ways and that in doing so they are learning and growing as writers.

Dan and Jennifer: In our talks with colleagues, we find that the first impression and worry about this technology stems from the concern that students are using the technology to shortcut the learning process. If they can get their homework done quickly without having to work hard or think about it, if it is “good enough” to pass them along, they would be tempted to do so. We have discussed the roots of this temptation in our faculty meetings: the traditional educational apparatus is a business that exists within a capitalistic system. Such a system privileges efficiency and transaction: those who get better results with less work are more successful. This model—that of grades and points as a transaction—distracts from the point, purpose, and method of education: to struggle, work, and think, and in doing so, learn, grow, develop. We believe that the transactional model must be resisted with innovation, collaboration, and discussion, whether or not tools such as ChatGPT are in play.

LLM technology can be used to short circuit the writing and learning process. But we think that many teachers take that fact and jump to the wrong conclusion: they frame it as simple cheating software and attempt to prohibit its use. This perspective and approach, however, simply relegates student engagement with LLMs to the shadows. Research reveals that LLM writing cannot be consistently or reliably differentiated from human writing by either human readers or detection software (Sadasivan et al., 2023; Weber-Wulff et al., 2023). In addition, the technology continues to grow in ubiquity. With companies including Google, Microsoft, and even Snapchat now offering their own LLMs in their software, students will continue to have access to, and be tempted by, the promise of the shortcut. If they are not taught how the technology works, what it can do, what it can’t do, and the ethical and critical considerations of using the tools, they will rely on the technology, in the dark, in all the wrong ways.

References

Berdanier, C. G. P., & Alley, M. (2023). We still need to teach engineers to write in the era of ChatGPT. Journal of Engineering Education, 112(3), 583–586. https://doi.org/10.1002/jee.20541

Chan, C. K. Y., & Hu, W. (2023). Students’ Voices on Generative AI: Perceptions, Benefits, and Challenges in Higher Education (arXiv:2305.00290). arXiv. https://doi.org/10.48550/arXiv.2305.00290

Collins, K. M., Wong, C. C. L., Feng, J., Megan, W., & Tenenbaum, J. B. (2022). Structured, flexible, and robust: Benchmarking and improving large language models towards more human-like behavior in out-of-distribution reasoning tasks. arXiv. https://doi.org/10.48550/arxiv.2205.05718

Domenech, J. (2023). ChatGPT in the Classroom: Friend or Foe? 9th International Conference on Higher Education Advances (HEAd’23), 339–347. https://doi.org/10.4995/HEAd23.2023.16179

Halaweh, M. (2023). ChatGPT in education: Strategies for responsible implementation. Contemporary Educational Technology, 15(2). https://doi.org/10.30935/cedtech/13036

Mollick, E. R., & Mollick, L. (2022). New modes of learning enabled by AI chatbots: Three methods and assignments. SSRN Electronic Journal. https://doi.org/10.48550/arXiv.2205.05718

Mollick, E. R., & Mollick, L. (2023). Assigning AI: Seven approaches for students, with prompts. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.4475995

Rahman, Md. M., & Watanobe, Y. (2023). ChatGPT for education and research: Opportunities, threats, and strategies. Applied Sciences, 13(9), 5783. https://doi.org/10.3390/app13095783

Rudolph, J., Tan, S., & Tan, S. (2023). ChatGPT: Bullshit spewer or the end of traditional assessments in higher education? Journal of Applied Learning and Teaching, 6(1). https://doi.org/10.37074/jalt.2023.6.1.9

Sadasivan, V. S., Kumar, A., Balasubramanian, S., Wang, W., & Feizi, S. (2023). Can AI-Generated Text be Reliably Detected? (arXiv:2303.11156). arXiv. https://doi.org/10.48550/arXiv.2303.11156

Weber-Wulff, D., Anohina-Naumeca, A., Bjelobaba, S., Foltỳnek, T., Guerrero-Dib, J., Popoola, O., Šigut, P., & Waddington, L. (2023). Testing of Detection Tools for AI-Generated Text. ArXiv. https://doi.org/10.48550/arXiv.2306.15666

Xiao, P., Chen, Y., & Bao, W. (2023). Waiting, Banning, and Embracing: An Empirical Analysis of Adapting Policies for Generative AI in Higher Education (arXiv:2305.18617). ArXiv. https://doi.org/10.48550/arXiv.2305.18617

Yancey, K. B. (2009). Writing in the 21st century: A report from the National Council of Teachers of English. Urbana, IL: National Council of Teachers of English. https://literacy.wonecks.net/2009/03/16/writing-in-the-21st-century-a-report-from-the-national-council-of-teachers-of-english/