30 Revising LLM Text to (Re)Discover Rhetoric in a Graduate Seminar

In a recent post on X, statistician Nate Silver remarked that among his “weird takeaways” about large language models is “that language is more strategic than was generally assumed and extremely rich in implicit information.” Michael Trice, who teaches writing at MIT, responded to humorous effect, “Rhetoric welcomes all its new found fans.”

Articles both journalistic and scholarly have pointed out that text from LLMs tends to replicate what is considered good school writing: “Smooth, bland, correct, formulaic, compliant, voiceless, not-wrong” (Morrison) with “surface-level fluency” (Warner). We might think of it as writing that is not rhetorical, prioritizing alleged objectivity and clarity above other attributes. LLM writing, in its recognizable difference from writing by people, makes Silver realize the real and present human motives, ulterior and otherwise, behind language use.

Morrison writes of LLM text that it “pretends that there can be writing without writers, which is as nonsensical as suggesting there can be swimming without swimmers, or breathing without breathers.” She goes on to say that “The generated text is boring, or derivative, or both, and often factually incorrect. When I write, it’s because I have something to say, and I know that if I do it persuasively, I can change some part of the world in some small way” (160). Academic writing should be rhetorical, interesting, persuasive, and intended toward change; it shouldn’t be incompatible with those goals. However, students still internalize rules about writing that erase them – their writing selves, or voices. Some instructors have been un-teaching these rules for many years, insisting that students use first-person pronouns, state their opinions and feelings unabashedly, address real audiences of readers, and experiment with language. Now, with LLMs, this project of unlearning takes on a new urgency.

The assignment I describe here is for a special topics graduate seminar on composition pedagogy, with the topic “Learning to Live with LLMs.” Along with scholarship in composition studies specifically about LLMs, I am situating LLMs in earlier scholarship about plagiarism and authorship, plagiarism detection services (or what I prefer to call automated plagiarism detection), and automated essay scoring, also called machine scoring. The students in the class were in a variety of areas within English Studies, mostly literature, but also linguistics and rhetoric and composition. Most of them were teaching or were preparing to teach first-year writing, so I approached the course with this in mind.

The final project in the course was a portfolio of work consisting of four reading responses that I’ve assigned for the semester: a conference-length essay on a topic of their choice, an end-of-semester reflection on what they learned, and what I am calling an LLM Revision Project, which is the subject of this essay. The assignment asks students to think of a prompt and enter it into a LLM, then save the unedited output text. Then students had to make revisions to the output text, resulting in a new version that contained a mix of their writing and the LLM output. They also had to document these revisions and explain why they made each change (the rhetorical component, Silver’s “language is more strategic than was generally assumed and extremely rich in implicit information”), as well as reflect on the process of, essentially, collaborating with an LLM.

My purpose for this assignment was to provide students with a way to experiment with LLM writing – as some of them had not played around with LLMs on their own – and to prompt them to think about authorship, writing processes, and revision. The assignment includes a changelog showing each revision that the student made.

What is a changelog?

Changelog is a term from software development, listing the changes made from one version of a software application to the next. For writing, a changelog is a simple table that has a reader’s comments, explanation of the revision made in response to each comment, and where in the document the change can be found. I have used changelogs in my own writing, teaching of undergraduate and graduate writing courses, and thesis and dissertation directing for many years. I find that changelogs are good tools for making revision clear and visible, showing how the writer revised for each comment of feedback from a reader. Though our graduate students are not required to teach changelogs in their First-Year Writing courses, some have told me that they have their students include changelogs with revised versions of writing projects.

In this assignment, the changelog is showing the student’s own revision decisions; in other words, they are the readers of the LLM output text, they decide themselves what needs to be revised, and then they do the revisions. The LLM Revision Project’s changelog has five columns: one that numbers the changes (I required at least 20 changes), one that shows the excerpt of original LLM output text that they decided to change, one that briefly names the change (“added a sentence,” “substituted a word,” “deleted two sentences,” etc.), one that explains the reason for the change, and one that shows where the change is in the new version (“third paragraph,” “item 5 in numbered list,” etc.).

What follows is the student-facing description of the assignment, then a model of the assignment that I did myself. The prompt I entered into GPT-4 was, “How can we use Rita Dove’s poem “Parsley” to understand social and economic conditions in the Dominican Republic?” The output text was a brief explanation of some very general qualities of the poem and some historical information about Rafael Trujillo’s regime, presented in a numbered list.

What was my process of revising the output?

I heavily revised the GPT-4 output, adding a lot of my own writing to it, changing some words and phrases, and deleting some of the output text. Almost all of what I changed was with rhetoric (purpose, audience, persuasion) in mind. I explained my own personal motivation for writing about Dove’s poem and my first experience reading the poem as an undergraduate student in the 1990s, and I explained why I see this connection of Dove’s poem and understanding conditions in the Dominican Republic as making a case for the value of studying the humanities. Procedurally speaking, I toggled back and forth between my revised version and the changelog, entering each change as I made it, but I made it clear to students that they didn’t have to do that – they could make all the changes and then create the changelog afterward.

***

Student-Facing Prompt

LLM Revision Project

Deliverables of the project:

• Prompt for LLM
• Text generated by LLM
• Revised version of LLM-generated text (make at least 20 changes, minor or major)
• Changelog of revisions with explanations
• Reflection on the process of doing the project (500 words)

LLMs are part of the writing landscape from this point forward. Our readings in class have expressed a variety of positions on LLMs, with some wanting nothing to do with LLMs, others pointing out the language discrimination and privileging of Standard Written English in LLMs, and others claiming that we’ll all be using them in the future in some capacity. This project is an experiment with intentional use of LLMs.

The Prompt:

First, you’ll select an LLM to use: ChatGPT, Google Bard, or another tool. Then, think of a prompt to enter. Your prompt can be about a topic pertaining to your research interest (in this class or in your primary research area), or it can be a topic about current events or a political issue, with the understanding that the corpus text in the LLM may not have updated news. For example, ChatGPT’s corpus doesn’t have articles published past 2021. [Note: this was at the time of the assignment; it now has information up to April 2023.] Your prompt could be a creative one, if you like, to generate a short piece of fiction or a poem, or a letter to a friend or family member. For best results, the subject matter should be something you care a lot about.

The Changes:

Once the LLM has generated its response to your prompt, copy and paste both your prompt and the generated text into a document file. Then, paste the LLM-generated text into the document again (this will be the second version, the one you revise) and make at least 20 changes to it. Changes can vary from substituting a different word from the one in the LLM text, to inserting or deleting a phrase or sentence, to adding or deleting whole paragraphs. The most important thing is explaining why you make each change, which you’ll do in the changelog.

The Changelog:

A changelog is a useful tool for any writing project you do, especially when you are getting feedback from readers, like reviewers for scholarly journals, or your thesis or dissertation committee. For this project, your changelog should be in table form (see the changelog in the sample assignment).

The Reflection:

This should be a short reflection (500 words maximum) about what your experience was like doing this project. Consider these questions (no need to answer them all):

• Would it have been easier to write this from scratch?
• Was this process different from how you normally revise a draft of your own writing?
• What did you think about doing the changelog?
• Do you see the changelog as being a useful tool for future writing projects – why or why not?
• If you had the chance to redo the project, would you have changed the prompt you gave the LLM? Why or why not?
• Will the experience of doing this project change your writing process – do you think you’ll use LLMs in your writing? Explain.

***

Model of the assignment that I wrote for students to review

Sample LLM Revision Project

My prompt:

How can we use Rita Dove’s poem “Parsley” to understand social and economic conditions in the Dominican Republic?

Unaltered Response from GPT-4:

Rita Dove’s “Parsley” provides a chilling reflection on one of the darkest moments in the history of the Dominican Republic, namely the Parsley Massacre (also known as “El Corte”) of 1937. Through this poem, we can delve deep into the social and economic conditions of the Dominican Republic, particularly under the regime of the dictator Rafael Trujillo.

1. Background: Before diving into the poem, it’s essential to understand the historical context. In 1937, under the orders of Trujillo, the Dominican army killed thousands of Haitians living near the Dominican-Haitian border. The soldiers would hold up a sprig of parsley to a person, asking them to pronounce its name in Spanish: “perejil”. Haitians, with their French-based Creole accent, had difficulty pronouncing it in the Spanish way, which made it easy for soldiers to identify and massacre them.

2. Language and Identity: The poem uses the word “parsley” as a metaphor for the thin lines that separate communities and individuals based on race, language, and nationality. The arbitrary nature of using pronunciation to determine one’s right to live speaks to the constructed and, often, arbitrary distinctions that have real-world implications in terms of social inclusion, discrimination, and violence.

3. Economic Desperation: The massacre was, in part, a result of economic tensions. Haitians often crossed into the Dominican Republic for better work opportunities, especially in the sugar cane fields. This migration was seen as a threat by some Dominicans. The poem, while not directly focusing on the economic conditions, touches upon these tensions by emphasizing the differentiation and dehumanization of the ‘other’, which can be exacerbated in times of economic strain.

4. Power and Oppression: Trujillo’s regime was marked by brutality, fear, and oppression. The poem touches upon the unchecked power of a single man and the horrors that can ensue when that power is used without restraint. Such unchecked authority can distort the social fabric, and economic imbalances can become even more pronounced as the ruling class or individual consolidates wealth and power.

5. Memory and Trauma: The haunting tone of the poem and its reflections on death and life reflect the collective trauma experienced by a nation. This trauma goes beyond the immediate violence, seeping into the social consciousness, affecting interpersonal relationships, trust within communities, and the nation’s overall psyche.

6. Environmental Imagery: Dove uses nature – the cane fields, the greenery of the parsley – as symbols in her poem. They serve as silent witnesses to the violence and draw a contrast between the natural world and the human-inflicted brutality, possibly suggesting the violation not just of human life but of the land itself.

By studying “Parsley,” readers can gain insights into the complexities of the Dominican Republic’s history, the depths of human cruelty, and the socio-economic tensions that can underlie and instigate such violence. The poem stands as a potent reminder of the past’s reverberations in the present and the importance of memory in shaping our understanding of social and economic conditions.

Changelog:

Revised Version:

Over the last decade or so, enrollment in the humanities has declined, prompting humanities scholars to explain to prospective students and the public why studying literature, history, and other humanities disciplines is valuable. Often, we do this by remarking that through the humanities fields, we teach students how to comprehend difficult texts, solve complex problems, and communicate effectively in writing: important skills for any career that students may have after college. The problem is, we don’t always model how to apply knowledge gained from discussing literature (or history, philosophy, etc.) to solving complex problems. I think that if we’re going to make these claims for the value of humanities, we ought to take responsibility for this knowledge transfer by having students apply what they learn to hypothetical or real problems that students may encounter.

In the spirit of teaching literature with an eye to knowledge transfer, I am doing a thought experiment using Rita Dove’s poem “Parsley.” It provides a chilling reflection on one of the darkest moments in the history of the Dominican Republic: the Parsley Massacre (also known as “El Corte,” or “the cutting”) of 1937. The poem is twelve stanzas written in free verse. The speaker of the first section (six stanzas) is a collective “we” of workers in the sugar cane fields. The speaker in the second section is General Rafael Trujillo (though he is not named), and his inner monologue reveals a man who is angry about the death of his mother and haunted by memories of the battlefield. The poem imagines that this anger drives his cruelty toward the Haitian people. Through this poem, we can delve deep into the social and economic conditions of the Dominican Republic, particularly under the regime of the dictator Rafael Trujillo.

1. Background: In 1937, under the orders of Trujillo, the Dominican army killed thousands of Haitians living near the Dominican-Haitian border. The soldiers would hold up a sprig of parsley to a person, asking them to pronounce its name in Spanish: “perejil”. Haitians, with their French-based Creole accent, had difficulty pronouncing it in the Spanish way, which made it easy for soldiers to identify and massacre them.

2. Language and Identity: The poem uses the word “parsley” as a metaphor for the thin lines that separate communities and individuals based on race, language, and nationality. The arbitrary nature of using pronunciation to determine Haitian people’s right to live speaks to the constructed and, often, arbitrary distinctions that have real-world implications in terms of social inclusion, discrimination, and violence.

3. Power and Oppression: Trujillo’s regime was marked by brutality, fear, and oppression, but anti-Black racism already existed in the Dominican Republic before he came to power. According to Busey and Cruz (2017), “Trujillo promoted blanqueamiento (whitening), encouraging the immigration of single Europeans to the island and offering refuge to Jews during World War II because they were considered white—thus attempting to mejorar la raza or ‘improve [whiten] the race’ of the Dominican Republic.” He also wore foundation makeup to lighten his own skin. Racism and classism can become even more pronounced as the ruling class or individual consolidates wealth and power, allowing for this kind of genocidal violence.

4. Memory and Trauma: The haunting tone of the poem and its reflections on death and life reflect the collective trauma experienced by a nation. The death of the speaker’s mother, and the death of a soldier who is killed before his eyes, are relevant. Could this have gone differently had he not experienced these traumas? We will never know, but the poem challenges us to consider the question. This trauma goes beyond the immediate violence, seeping into the social consciousness, affecting interpersonal relationships, trust within communities, and the nation’s overall psyche.

5. Environmental Imagery: Dove uses nature – the cane fields, the parrot, the greenery of the parsley – as symbols in her poem. They serve as silent witnesses to the violence and draw a contrast between the natural world and the human-inflicted brutality, possibly suggesting the violation not just of human life but of the land itself. The land is an important part of the context; the Dominican Republic was first colonized in the sixteenth century and has been under Spanish and French rule, which has forever marked the land: an island ecosystem that was destroyed to form sugar cane plantations, making European settlers rich.

I first read “Parsley” when I was a junior in college, and it was a gut punch. It has stayed with me ever since (almost 30 years now). One way I think the humanities can help people understand real-world complex problems is simply by piquing our curiosity. Would I have done research about the history of the DR if I hadn’t encountered this poem? I doubt it. This poem was a point of entry, and other poems can do this too. This poem made me care about the DR, when I hadn’t before, because I just didn’t know what had happened there. Do the specific lines in the poem help me understand conditions in the DR? Maybe they don’t, at least not current conditions. But I have remained attuned to news about the DR because of both this poem and Julia Alvarez’s novels, also centered on Dominican people. Recently I attended a talk by a Black United States diplomat who’d had postings in several different countries. He said there were two countries he would not accept a post in: one was Russia, and the other was the Dominican Republic. He explained that racism is common, and tolerated, in the DR, where he couldn’t be sure that, if he took his family to a restaurant, they would allow him to eat there.

Faculty in the humanities, especially faculty members who teach literature courses, may object to this approach to talking about literature. This is not a proper close reading, they may say. Or they may oppose on principle the teaching of literature in this utilitarian or instrumental way. My response would be: then don’t make the claim that studying the humanities helps people solve complex problems – find a new argument. Unfortunately, we do have to make the case for our value.

By studying “Parsley,” readers can gain insights into the complexities of the Dominican Republic’s history, the extremes of human cruelty, and the racism and settler colonialism that can underlie and instigate such violence. The poem stands as a potent reminder of the past’s reverberations in the present and the importance of memory in shaping our understanding of social and economic conditions. Trujillo neither started nor ended the complex problem of racism in the DR; in 2023, people are still living with the legacy of slavery and genocide in palpable ways.

Reflection:

Even though I used an LLM for part of this piece of writing, I still put a lot of time and thought into my changes. I added a lot of writing to it and removed some; in some places, I made word-level changes. For each change, I had my own intent in mind and what impressions I wanted to make on my reader. I put in some of my personal experience, which an LLM can’t do for me, and I tried to make the analysis more meaningful by giving examples. Using the changelog in this way was an interesting challenge that, on the one hand, forced me to make twenty changes, but on the other, kind of limited me to twenty changes. I think that’s a good thing – I could have made more than twenty changes, and this limit helped me to know when I was finished with the project. If I had made fewer changes, I’m not sure they would have been transformative enough. The prompt I gave to GPT-4 was kind of vague, now that I look back at it; I wonder what it would have generated if I had added something like “Explain how this is an example of how studying literature can help people understand and solve real-world complex problems.” I think if I’d added that to the prompt, the generated text would have had even less to do with the poem than this version did. I’m surprised that it didn’t quote any lines from the poem or have much about the poem other than the idea that things in nature were used as symbols. Right now, I don’t really see myself using LLM text very much in my own writing process, but I guess that could change depending on the task.

***

End of student-facing material

Students’ Response to the Project

When I reviewed the students’ projects, I was pleased: it was clear that they understood the prompt – I think my model helped to demonstrate what the assignment was. I asked them in class how they ended up doing the revisions and documenting them in the changelog: did they toggle back and forth like I did, or do all the revisions (visually distinguishing them, for example, in a different color font) and then create the changelog? Almost all of them went back and forth. I also asked them in class: what would you think if your first-year writing students were to use LLMs this way for writing assignments in your class? They responded, with some hesitation, that they thought it would be acceptable and would give some transparency to students’ use of LLMs, but they didn’t seem to be strongly supportive of LLM use. I asked: if students choose to use LLMs this way, generating a draft and then making changes and documenting them in a changelog, should they be able to get an A on that assignment? The responses were interesting — some students replied with an emphatic no, that they could get a passing grade of C, but not higher than that. Others were open to the idea, especially because a maximum grade of C for transparency in use of LLMs could make it more tempting for students to use LLMs in a more deceptive way.

Of the ten students in the seminar, five of them chose to do the project about one of their research interests. The other five did a combination of personal interests (one did a post-date letter to a person they didn’t want to go on a second date with). None of the students had done this kind of work with LLM output before: integrating their own writing with the generated text. All the students added sentences and paragraphs to the initial output. In their changelogs, they demonstrated that they had been thoughtful about why they made each change, and their reasons revealed that they were reading the LLM output critically; many of their changes corrected what they considered to be information that was not quite accurate. They were fairly split about whether or not they envision themselves using LLMs in their writing in the future. Several students noted that the LLMs did not provide enough context about their subject and added this – an important rhetorical, reader-focused concept. They tended to agree that LLMs are handy brainstorming tools but that the generated text should not stand alone without human revision.

Conclusion

The LLM Revision Project was an assignment for a special topics composition studies graduate seminar about LLMs: situating them in rhetoric and composition scholarship about authorship, plagiarism, and automation in writing pedagogy (plagiarism detection services, automated essay scoring). This assignment was an opportunity for them to do some quasi-collaboration with an LLM “author” sourced from text scraped from the internet. I intended it to help students reflect on their own authorship and use of rhetoric, as well as their own evolving positions on appropriate use of LLMs. If I could go back and do something differently, I would have asked students to take more risks in their revisions and reflect specifically on the school-style of LLM output text: that objective, flat affect and stakelessness that merely gestures toward information without infusing it with personal meaning and emotion. Only a few of them made those kinds of human experience-based revisions.

The project itself was not subject-matter specific; the students in the course were studying linguistics, literature, popular culture, and other topics. Rather, the focus was on the process: giving the LLM a prompt, evaluating the output, making and documenting revisions, and reflecting on the experience. The process could apply to any discipline: any area in English Studies, or other fields: undergraduate or graduate level. It could even use a different document instead of LLM text – it could be a rhetorical revision of any text. I can imagine students making at least twenty changes to a different source text: a Wikipedia article, an archival document from the distant past, or a famous short story. I hope to hear from any teachers who try it in their classes.

References

Morrison, A. (2023). Meta-writing: AI and writing. Composition Studies, 51 (1), 155-161. https://compstudiesjournal.com/current-issue-spring-2023-51-1/

Warner, J. (2023, April 21). ChatGPT and writing assessment, an old problem made new. InsideHigher Ed. https://www.insidehighered.com/opinion/blogs/just-visiting/2023/04/21/chatgpt-and-writing-assessment-old-problem-made-new

About the author

name: Clancy Ratliff

institution: University of Louisiana at Lafayette

Clancy Ratliff is Professor of English and Associate Dean of the College of Liberal Arts at the University of Louisiana at Lafayette. Her research and teaching interests are in feminist rhetorics, environmental rhetorics, writing program administration, and copyright and authorship. She has published research in Women’s Studies Quarterly, Kairos, Pedagogy, and other journals and edited collections. She is involved in several community advocacy organizations, including Sierra Club Delta Chapter, Move the Mindset, Citizens Climate Lobby, and the Acadiana Regional Coalition on Homelessness and Housing.