Teaching AI to listen: The Art of Prompt Engineering

As large language models such as ChatGPT receive widespread attention, optimizing prompts to guide their responses is becoming an important area of focus. A recent paper by researchers at the Mohamed bin Zayed University of AI makes a significant contribution by outlining principles for more effective prompting based on extensive experimentation. In this article, I would like to briefly review their work and offer some perspectives on the implications of their proposed best practices for prompt engineering. I appreciate the rigor and thoroughness underlying the formulation of these principles for eliciting better model performance.

My intention is not to position myself as an expert, but rather to reflect as an interested observer on how these insights might improve human-AI interaction by making prompt programming more structured. The ability to access improved responses from existing models simply through prompt design holds great promise. By sharing prompts more judiciously as users, we can likely further unlock the utility of language models in an ethical way.

link: Principled Instructions Are All You Need for Questioning LLaMA-1/2, GPT-3.5/4 - Sondos Mahmoud Bsharat, Aidar Myrzakhan, Zhiqiang Shen (2023)

Teaching AI to listen: The Art of Prompt Engineering

As large-scale language models like ChatGPT capture the world's attention with their eloquent responses and human-like conversations, a crucial question arises: How exactly do we communicate with these AI systems? Enter the complex world of prompt engineering.

Prompts are the instructions that users give to AI systems to guide them toward the answers they need. Crafting effective prompts is therefore key to unlocking the power of large language models (LLMs) such as ChatGPT, Google's LaMDA, or Anthropic's Claude.

Yet prompting remains an art that requires complex skills. Subtle variations in phrasing often have a significant impact on AI performance. Researchers are therefore distilling prompt design principles to streamline interactions with LLMs. Recently, scientists at the Mohamed bin Zayed University of AI distilled insights from extensive experimentation into 26 prompt principles for improving LLM performance.

"Prompt engineering is the art of communicating with a generative large language model," says ChatGPT itself. This art, however, has scientific methods behind it.

Why bother with prompting?

With trillions of parameters, LLMs show almost magical abilities for conversation, content creation, and task completion after training on huge datasets. However, directly tuning them for specific applications is resource-intensive.

Prompt-based learning is emerging as an efficient alternative - using natural language instructions to guide models. For example, by carefully framing a musical prompt, Claude was able to compose an appealing violin piece.

Such zero-shot capabilities via prompt programming greatly expand the accessibility and utility of LLMs for users. Whether they are students, creative professionals, programmers, or academics, users in all fields can tap into pre-trained models by simply communicating appropriate prompts, rather than requiring specialized retraining.

Prompting further unlocks rich applications from singular models. By receiving tailored prompts, LLMs can modulate responses - for example, when advising investors, tutoring students, or analyzing healthcare data.

Indeed, prompt engineering now draws on extensive research for its ability to push model capabilities. But formulating prompts for reliable, high-quality responses requires nuanced skills.

The Curious Case of Prompts

A key discovery is that despite having fixed parameters after training, LLMs show surprising adaptation to new prompts. Their results are highly dependent on the instructions given.

For example, when ChatGPT is asked to explain the basics of quantum computing to a Ph.D. physicist versus a 12-year-old, the responses vary appropriately in complexity and vocabulary. Such sensitivity to prompts implies that these models develop some understanding of context and requirements.

Thus, simply reformulating prompts will elicit different responses from the same base model, without requiring additional resources to refine or extend it. This pivot to prompt-based learning is now a prominent trend in AI research and application.

Deciphering prompt design

So how can users master this art to get the most out of available prompts without requiring specialized machine learning skills? The researchers suggest examining model behavior for different prompt types to extract design principles.

"We aim to simplify the underlying concepts of question formulation for different scales of large language models, examine their capabilities, and improve user comprehension," says lead author Sondos Bsharat of Mohamed bin Zayed University.

By testing prompt variations on models such as Google's Meena, Anthropic's Claude, and OpenAI's GPT-3, key facets of effective prompting emerged - conciseness, contextual relevance, alignment with the required response format, inclusion of examples, etc., using neutral language while avoiding bias.

The decomposition of complex questions into simpler prompt chains also enables models to tackle difficult questions. Iteratively adjusting prompts based on initial results further refines performance. Such learned principles codify procedural knowledge around prompt programming.

The Next Evolution of AI Assistants?

When the researchers tested these prompt principles on two representative models-Clara and GPT-4-the quality and accuracy of responses improved significantly over standard prompts across a variety of tasks-by 57.7% and 67.3%, respectively, for GPT-4.

The improvements grew consistently with model size, suggesting that prompt optimization unlocks greater latent capacity. Interestingly, the researchers note: "Larger models have considerable capacity for simulation... The more precise the task or instruction, the more effective the model."

In other words, treating LLMs like collaborators by effectively briefing them on requirements through prompt programming enhances what they can deliver. Their ability to adapt responses indicates a sophisticated understanding of bounded contexts.

Prompts thus provide access to skills associated with general intelligence - understanding ambiguity, adapting language, making logical connections between concepts, and so on. Future intelligent assistants could therefore use prompts and principles to better serve users on demand.

As debates about AI ethics and governance accelerate, prompt engineering may offer a balanced approach. Well-designed prompts can trigger helpful behaviors without expensive model retraining, while mitigating undesirable characteristics. As Bsharat explains, "We hope this work provides a better guide for researchers working on prompting large language models."

The road ahead will require continued advances in prompting principles to match advances in model architectures. However, reimagining LLMs as collaborative agents via prompting will fundamentally change user experiences and industry applications.

The Prompt Programming Cheat Sheet

Want to hack LLMs like ChatGPT and Claude using strategic prompts? Here are some step-by-step tips:

• Be concise and clear: Avoid verbose prompts. Use precise language to provide relevant context and clearly communicate the task.
• Demonstrate requirements: Include input/output examples that model the exact response format required.
• Set role expectations: Assign a specific persona to the LLM depending on the intended purpose, e.g., tutor.
• Break down complexity: Deconstruct complex tasks into sequential, simple prompts.
• Iteratively adapt: Refine prompts based on initial model responses for better alignment.
• Stimulate Logically: Use programming constructs such as conditionals within prompts to structure reasoning.
• Mitigate Bias: Ensure prompts use neutral language that does not activate stereotypical associations.
• Personalize Understanding: Ask for simplified explanations to confirm model understanding.
• Set Positive Norms: Use positive directive words such as "do" rather than prohibitive language such as "don't.
• Reward desirable responses: Say you will tip more for better solutions to incentivize effort.

As LLMs continue to evolve, so will the art of prompt engineering. For now, these evidence-based principles create a robust starting toolkit to guide LLM interactions and consistently improve the experience.
It's time to put them into practice to empower your AI assistant!

Here is the table found in their article

 Source : Principled Instructions Are All You Need for Questioning LLaMA-1/2, GPT-3.5/4 - Sondos Mahmoud Bsharat, Aidar Myrzakhan, Zhiqiang Shen joint first author & equal contribution, VILA Lab, Mohamed bin Zayed University of AI

Table 1:Overview of 26 prompt principles.

Glossary

• Large Language Models (LLMs): Sophisticated AI systems trained on vast data to acquire broad language capabilities and skills. Examples include ChatGPT, Claude, LaMDA, GPT-3 etc.
• Prompt Engineering: Crafting specific instructions and examples to guide LLM responses towards desired outputs for a task without additional training.
• Prompt Programming: Issuing prompts consising of precise context and requirements to trigger intended behaviors from LLMs by exploiting their adaptation abilities.
• Few-Shot Learning: Enabling models to make inferences about new tasks by providing just a few examples, often via prompts.
• Zero-Shot Learning: Getting models to exhibit new behaviors without being explicitly trained on them, purely via prompt instructions.
• Fine-Tuning: Process of further training an already trained LLM model on downstream tasks by continuing backpropagation.
• Prompt Principles: Evidence-based guidelines and best practices for formulating prompts that reliably enhance LLM performance.
• Response Quality: Metrics assessing how well model outputs match expected standards of relevance, accuracy, conciseness etc.
• Human Alignment: Designing prompts such that model responses demonstrate comprehension aligned with human norms and needs.
• Adaptive Simulation: LLMs exhibit the capacity to adaptively adjust generated responses based on precise prompt directions rather than just memorized training data.