Dispense With The Pleasantries? New Study Asks Whether Manners Matter In AI Prompts

For all the talk about AI’s jockeying for position in the Search world and what to make of the results it creates, the thought for many has been that, when it comes to those results, you get what you give.

It’s the idea that the effort put forth in a prompt directly correlates with the reciprocated AI result given back. Sort of like that whole “the love you take is equal to the love you make” sort of thing, but with AI prompting. 

And while a new report on AI prompts doesn’t promise the latest tip to kill it with kindness, it instead sheds some light that, well, the love you make (and the niceties in your questioning) actually might not make a consistent difference in the results you get.

“It is hard to know in advance whether a particular prompting approach will help or harm the LLM’s ability to answer any particular question,” says the authors of the new “Prompting Science Report 1: Prompt Engineering is Complicated and Contingent” study. “Specifically, we find that sometimes being polite to the LLM helps performance, and sometimes it lowers performance. We also find that constraining the AI’s answers helps performance in some cases, though it may lower performance in other cases.”

The study found no universal value to “particular prompting formulas or approaches” following its 19,800 test-runs (per prompt, for each of OpenAI’s GPT-4o and GPT-4o mini models), using the GPQA Diamond dataset of multiple-choice PhD-level questions as its benchmark.

(Photo Source: ”Prompting Science Report 1: Prompt Engineering is Complicated and Contingent”)

While many of the ensuing, more-nuanced results are as complicated as the questions themselves, The Neuron offers a bit of clarity and takeaway from the study.

“This has huge implications for how we use and evaluate AI tools,” The Neuron says. “Companies relying on benchmarks alone might completely miss how inconsistent these models can be in real-world use. Also, this inconsistency problem probably explains why genAI hasn’t been fully rolled out yet, despite the obvious gains it can unlock. Consistency is key.”