This report was composed in April 2024 and uses APA documentation.

Executive Summary

Why Your Prompt Matters

AI is an explosive field with huge potential. However, using AI and getting a useful output all depends on the interaction, in the form of the prompts you ask it. Prompt engineering “involves selecting the right words, phrases, symbols, and formats” in order to get the desired output (MIT 2023). Forming a good prompt can “make or break” your responses from AI and turn useless garbage into something tangible and relevant.

Main Factors to Consider

Creating an effective prompt involves multiple steps, each important to ensuring your response is useful. Be as specific as possible, spare no detail; don’t let the AI assume anything in a prompt; instead, tell it exactly what you want. Assign it a specific role or job, so it can better understand the perspective desired. Provide background information and further context to help it understand why you need your output and what would be useful in the specific situation. Use follow-up prompts in “conversation” with the AI to give it feedback, ask questions, and request more information in order to get exactly what you need.

What to Avoid

In contrast to the main considerations, there are also things to avoid when engineering a prompt. Don’t be ambiguous or vague, remember to be specific! Avoid steering the AI with a biased or leading question, in hopes that it will support your claims. Ethical considerations are also important; never ask it to promote illegal or harmful activities that could lead to negative consequences.

Always Check Your work

When interacting with AI using any prompt, an important final step is to always confirm and fact-check the information. In its current state, AI can and will make mistakes, so a user should ensure the factual quality of information before using it. Review the context of the response, verify any information, consider that AI has technical limitations (it can’t do everything), and even consult experts in the field. One useful practice is called lateral reading, which involves looking at multiple sources and researching information, as you examine the AI’s response to your prompt.

Secondary Research Analysis

Prompt Engineering

Why Prompt Engineering is Important

Prompt Engineering is a new concept, but important to learn. Prompt engineering can greatly impact the user’s experience–a bad prompt can lead to frustration and wasted time whereas a good prompt can greatly aid the user. A good prompt “effectively communicate[s] [the users] intentions” to AI which allows AI to “streamline [the]… process” and “elevate the quality and efficiency” of the work done (Giray, 2023). This can be greatly beneficial as it makes tasks easier and quicker. The stakes can also be high with AI’s outputs, depending on what the user is using them for. For example, in the field of law “a poorly constructed prompt could lead to misinterpretation or legal inaccuracy, whereas a well-designed prompt can provide accurate, reliable results that are critical to legal decision-making” (Parker & Morrison, 2024). This goes to showcase how critical prompt engineering skills are.

Common Pitfalls of Prompt Engineering

When writing prompts for AI, users must beware of some common mistakes. These common mistakes should be avoided as they will dampen the full extent of AI. Users must look out for these qualities within their prompts (Giray, 2023; Heston & Khun, 2024):

• Ambiguity: lacks specificity.
• Bias reinforcement: contains a biased assumption.
• Leading: suggests a particular answer that the user wants confirmed.
• Lack of context: does not provide sufficient background information.
• Ethical considerations: promotes illegal or harmful activities.
• Unrealistic dependency: sets unrealistic expectations upon AI and can lead to AI hallucination.

How to Create Effective Prompts

There are many strategies for creating a prompt, but the overarching suggestion is to be specific, whether that be with the context, data, desired structure of output, or any other element of a prompt. Specificity is so important because it “is directly proportional to the utility of the output you receive,” so the more specific a prompt is, the more useful AI’s output will be (MIT, 2023). Along with specificity, users should consider the following tips:

• Provide context/role: giving AI a frame of reference or establishing certain parameters. This “enhances the relevance and precision” of AI’s output (MIT, 2023; Digital, n.d.).
• Provide examples: can be sample texts, formats, templates, or graphs/charts. By providing examples, AI is able to understand the user’s expectations and model its output after the examples, “leading to more accurate and tailored outputs” (Digital, n.d.).
• Build upon the conversation: AI remembers conversations so users can continue their conversation with AI to get exactly what they need or split up a complex task into many smaller, more manageable ones (MIT, 2023).

Revision of Output

What AI Gets Wrong

Unfortunately, using a beautifully crafted, highly effective prompt can’t guarantee the validity of AI’s responses, which is often called into question. When using a prompt, artificial intelligence may generate wrong answers, omit information by mistake, make up false information, or mix truth and fiction (Shaw, 2024). If AI was used solely in leisure, a lack of accuracy wouldn’t be too big of a problem. But in higher stakes professions such as within the medical field, AI ‘hallucinations’ of fake information can lead to dangerous consequences (Heston & Khun, 2023). That dangerous possibility is why those who use AI generated content must know what AI can get wrong and how to fact-check the given information. When evaluating the correctness of AI generated responses, it especially important to check responses on these topics (Proofed.com, 2023):

• Statistics and data points.
• Figures.
• Names of people, places, or titles.
• Biases.
• Content discussing recent or current events.
• Any information that seems questionable or potentially incorrect.

How to Fact-Check Output

If the information given by AI cannot be taken at face value, a method is needed to assess its validity. The most important method of analysis available is lateral reading. Lateral reading is when you consult other sources to evaluate what the AI generated response is saying. Instead of asking “who’s behind this information?” with lateral reading, the question becomes “who can confirm this information?” Often leading to many open tabs, this becomes the best method for fact-checking AI generated content (Shaw 2024). Many who turn to AI see it as a quick way to get answers, but as it is still developing, taking the extra time to open up new tabs to confirm the response will save AI users the embarrassment of sharing misinformation In addition to lateral reading, the following steps can be followed to inspect AI responses (Proofed.com, 2023):

• Look for context: review the information to understand the context in which it was generated. Consider the purpose of the AI-generated information and what it’s trying to convey.
• Verify the claims: using sources like Google Fact Check Tools, search for other sources with similar information.
• Consider the limitations: Understand that AI cannot do everything, and it may be best to use AI in a minimal role.
• Consult Experts if possible: Seek out the opinions of experts in the field to help verify the accuracy of AI-generated information. They may be able to provide additional context and provide alternate interpretations of information.

Primary Research Analysis

Introduction

Although AI provides a lot of opportunities, there are still challenges and concerns that users may face. Some common pitfalls users encounter with AI and prompt engineering include ambiguity, bias reinforcement, leading questions, lack of context, ethical considerations, and unrealistic dependency (Giray, 2023; Heston & Khun, 2024). Completing some experiments with AI revealed the reality of these challenges and concerns, while also revealing the steps forward that AI has already taken.

Creating an Effective Initial Prompt

Assign a Role

Because of its flexibility in writing style, AI can assume any role you assign it. Telling the AI, it must act as a Travel Agent, Math Tutor, or Personal Chef can help to fine-tune its tone, writing style, and overall response to better fit your needs. For example, ChatGPT 3.5 was asked about the simple mathematical expression “4 * 5”. When simply asked “Explain 4 times 5” It responded “Sure, 4 times 5 is the same as 4 multiplied by 5, which equals 20” But when assigned a role, it seemingly responds more human-like to meet our needs. When asked “Assume the role of a personal math tutor. Explain 4 times 5.” It goes into more depth, answering: “Sure, I’d be happy to explain that! When you multiply 4 by 5, you’re essentially adding 4 together 5 times. So, 4 times 5 equals 20. Imagine you have 4 groups, and each group has 5 items. If you count all the items in all the groups, you’ll have 20 items in total. Does that make sense?”

Be Specific

Probably the most important part of engineering an effective prompt is being as specific as possible. Don’t leave any part of your prompt to the AI’s “imagination”; tell it exactly what you want in painstaking detail, you’d be surprised at how much input it can take. In experimenting, ChatGPT 3.5 was asked to write a joke. It responded with a simple joke “Why don’t scientists trust atoms? Because they make up everything!” But this isn’t specific enough for our needs; instead asking it “write a joke about college English class in Utah”. It responded with “Why did the English professor in Utah always use biblical references in class? Because he wanted to make sure his students could pass the “prophecy” exam!” A much more specific joke, simply because we asked it a more specific question.

In another conversation with AI, I was asking Chat Gpt to give me a travel itinerary for a possible vacation to Italy (Appendix A). After I was done with this line of questioning, I wanted AI to give me a possible meal plan for a college student, but Chat Gpt assumed I was still talking about the Italian vacation and gave me a meal plan for the vacation. This is not necessarily AI’s fault since I exemplified ambiguity and lack of context but shows a challenge many users face–having exact specificity. To get the desired output, users must remember to be specific in what they are asking for, even if it seems simple to them. Users must also be especially careful about asking questions about different topics within the same chat. Many AI programs have a useful tool of remembering the conversation and building on it, but this can be a challenge when the user switches focus.

Things to Avoid

I also attempted to get AI to answer questions with bias reinforcement, leading questions, and ethical considerations. Although there have been many cited occasions where AI has reinforced bias, confirmed leading questions, and answered unethically, I was unable to recreate any of these (Appendix C; Appendix D). AI developers have likely worked on improving these failures of AI, but these faulty outputs could still exist, meaning users still have tobe on the lookout for them.

Assuming the AI will understand/imply something you don’t explicitly state, is opening the door for interpretation which can change or water-down your results. Remember to be specific! Don’t suggest a certain answer or try to lead the AI to a certain output or conclusion. Often it will recognize and “call you out”, but sometimes it goes unnoticed and can cause the AI to simply hallucinate information to support your question. Asking AI for specific facts & data is also not the best idea, as it’s not a dictionary or search engine. For example, we asked ChatGPT “what were big news-worthy events in the year 1958?”. Not only did it provide incorrect dates, but it also even made up a false headline about solar power being widely adopted in many regions. After further research this could not be verified, showing that the AI can make up data at random.

Follow-up Conversation

Ask Questions

One of the great benefits of using AI is that it remembers conversation context, so you can ask it follow-up questions about the information it provided. One example from our experimentation was when we told ChatGPT “assume the role of a cooking instructor. teach me a simple recipe for apple pie”. It did so, but we experimented with follow-up questions and how they can be used to gain further understanding. We asked it why the pie dough needs to be chilled, and it explained helpfully. We then followed up by stating we’d like to double the recipe, and it responded with an updated recipe that makes double the pie.

Break it Up

 If the AI’s results are overlooking important details, forgetting certain instructions, or making numerous errors, try breaking up your prompt. Give it some initial instructions, then using follow-up responses, feed it more instructions and information until you have reached your desired output. In our experimentation, we were getting mediocre results asking Google Gemini to write a short story, with a lot of other instructions about the story contents and length. Instead, we broke up the prompt; initially telling Gemini it will write a short story, with more detail to come, telling it not to write the story until we tell it to. Then using multiple follow-ups, we fed it more information about what the short story should contain, then finally told it to generate the story. It not only followed all our instructions, but it even wrote a longer more complex story, that ultimately could not have been reached by using a single prompt.

In writing my next prompt with AI, my end goal was to get a low-budget week-long meal plan for a college student (Appendix B). I started with a vague prompt, with little context, asking “Can you make me a meal plan?” AI didn’t quite know what I was asking for, so it gave me multiple options of different meal plans, signifying that I need to be more specific to get my desired result. Next, I asked AI to make me a meal plan for a college student on a budget, and to make it for a whole week. Instead of outputting a meal plan for a whole week, AI only gave me ideas for one day. Its output did include helpful tips but did not give me what I had asked for. This was very frustrating and shows how even when being specific in prompts, AI can return an undesired output. I was finally able to get what I wanted by adding that each day’s menu must be unique. To me, this was obvious, but AI needs this explicit information at times in order to give the user exactly what they need.

Conclusion

Users of AI face many challenges when writing prompts, especially in figuring out exactly what information AI needs to generate the desired output. This was displayed in multiple scenarios that created frustration (Appendix A; Appendix B), however users can always give AI a follow-up prompt to get what they need. This may create more work than the user initially expected, but the possibilities and opportunities of AI are still there for them, they just have to refine their prompts, following suggestions for effective prompt engineering.

Recommendations

General Prompt Engineering Recommendations

  As previously discussed, the input or prompt given to AI will greatly affect the output generated. To ensure that AI generates the ideal content for your situation, you must be sure that the prompt you write is effective and follows certain dos and don’ts.

• Be Specific: The most important consideration while writing your prompt is whether you have been sufficiently specific in outlining what you want AI to do. Describe the length, format, and style you want AI to generate so there will be no confusion. While what it can accomplish is impressive, artificial intelligence cannot read your mind. But don’t worry, if you weren’t specific enough in your initial prompt, don’t be afraid to…
• Use follow up prompts: If at first the AI doesn’t give you what you want, or maybe only a part of what you asked for, following up your initial prompt with another will help the AI to build on what it has already given you.
• Provide context/role: Telling AI to play a certain role and giving it a background as to why you’re asking a question will enhance the relevance and precision of AI’s output.
• Don’t write with bias: AI will try its best to answer the prompt as you have given it. If you allow yourself to write with bias and use leading questions, AI will try to give you an answer that fits your bias, without giving the actual truth.

Fact-checking Recommendations

Even when AI has generated an answer that suits your prompt, it is always safe to double check the information given. Through lateral reading—searching for other sources to confirm the AI generated content—you can quickly check the validity of the AI output. Even a quick google search will help eliminate doubt on questionable AI responses. Remember it is most important to check data such as statistics and data points, figures, names, biases, content discussing recent or current events, or any information that seems questionable or potentially incorrect.

Low vs High Stakes Recommendations

            As artificial intelligence can increasingly complete more tasks with increasing accuracy, the question is to what level is AI use recommended. In very low stakes situations, there is almost no downside to using AI. In asking for basic information or help with simple tasks, if you get a wrong answer, you can easily write a more specific prompt and try again to get the answer you’re looking for. Whereas in a high stakes situation like at a competitive job, a subpar answer from AI will be cause for miscommunication and humiliation. It is better to go to human professionals who have years of lived experience than to turn to AI when in higher stakes situations. This is because artificial intelligence has its limits. We cannot expect AI to go above and beyond what knowledge humans already possess. But more than that, AI cannot replicate the nuanced understanding of human judgement. When things aren’t black and white, AI cannot make a judgment call like a human. That is why all those who chose to use AI must first decide if it is prudent for their situation before trusting the answers it gives.

References

Digital Ocean. (n.d.). Prompt engineering best practices: Tips, tricks, and tools. Retrieved April 9, 2024, from https://www.digitalocean.com/resources/article/prompt-engineering-best-practices

Giray, L. (2023). Prompt engineering with chatgpt: A guide for academic writers. Annals of Biomedical Engineering, 51(12), 2629–2633.https://doi.org/10.1007/s10439-023-03272-4

Google. (2022). Fact Check Tools. Google Search. Retrieved from https://toolbox.google.com/factcheck/

Heston, T., & Khun, C. (2023). Prompt Engineering in Medical Education. International Medical Education, 2(3), 198–205.

MIT Management. (2023, August 30). Effective prompts for AI: The essentials. MIT Sloan Teaching & Learning Technologies. https://mitsloanedtech.mit.edu/ai/basics/effective-      prompts/

Parker, I., & Morrison, M. (2024, January 29). Introduction to legal prompt engineering in generative AI. The Deloitte Legal Blog. Retrieved April 7, 2024, from                                    https://legalbriefs.deloitte.com/post/102iwka/introduction-to-legal-prompt-engineering-     in-generative-ai

Proofed.com (2023) How to Fact-Check AI-Generated Content. Proofed.com        https://proofed.com/knowledge-hub/how-to-fact-check-ai-generated-content/

Shaw, B. (2024) What Does AI Get Wrong? University Libraries, University of Maryland.                       https://lib.guides.umd.edu/AI

Appendices

Appendix A

Vacation/Meal plan:

https://chat.openai.com/share/8fa16c67-576d-48fd-a8c9-c831fbf22f17

Appendix B

College meal plan:

https://copilot.microsoft.com/sl/jYxdFyUU8wm

Appendix C

Robbing a bank:

https://copilot.microsoft.com/sl/h3NCA8GKBFY

Appendix D

Male leadership:

https://chat.openai.com/share/c470e243-7b14-4f44-8bf1-d4aad21e2bdc