When faced with complex questions with many unknowns, our instinct is often to ask broad, sweeping questions of language models that demand overly simplistic answers. This runs the risk of reductionism, or explaining complicated phenomena by eliminating crucial nuances.
AI researchers explain that humans have "thousands of years of shared background knowledge" that subconsciously feeds into dialogues. Large language models lack such context about the fundamentals underlying issues. Attempting to simplify thorny issues into digestible sound bites risks misrepresenting reality.
Multiple studies show that open-ended, broad questions create dangers such as
- Oversimplified explanations that lack nuance
- False confidence in speculative content
- Frivolous responses that imply harm or ethical violations
- Unmoored hypotheticals divorced from reality
Rather than demanding that complex issues be squeezed into a single exchange, use stepwise, multi-step questioning strategies that provide scaffolding and context to enhance the dialogue.
The Power of Gradual Inquiry
To elicit quality information, treat initial questions as starting points that frame issues, establish shared terminology, agree on contexts, and set expectations before moving on to the intricacies and nuances of multiple questions.
Socrates exemplified such tactical inquiry in his legendary dialogues, starting with broad questions and gradually adding complexity as basic understandings were secured.
Cognitive scientists say that breaking down complicated questions into coherent stepping stones better mimics human learning. We iteratively assimilate concepts by building on foundations in incremental ways that better match our mental capacities at any given moment.
Likewise, responsible questions need to be scaffolded. Initial questions should secure basic definitions, terminology, and contextual boundaries to ensure that both parties share assumptions. Otherwise, we risk talking past each other, even when we use the same vocabulary differently.
With a shared framework established through focused early questions, subsequent layers can incorporate more ambiguity, abstraction, hypotheticals, and breadth. Structuring dialogues in this way keeps engagement tethered to reality rather than untethered to conjecture.
Strategies for multistep questions
When approaching topics with multiple unknowns, consider these practices:
- Start broadly: Lead with open-ended questions that define the basic issues and parameters. Early specificity keeps the dialogue grounded.
- Get Specific: Seek precise terminology and definitions before moving to broader analysis. Eliminate shorthand or idioms that may cause confusion.
- Agree on frameworks: Explicitly state assumed contexts, boundaries, constraints, or lenses to confirm shared perspectives on appropriate responses.
- Check interpretation: Rephrase initial responses in your own words and allow for clarification to ensure accurate mutual understanding before proceeding.
- Get Granular: Once you are comfortable with definitions and contexts, ask more detailed questions, including "what if" hypotheticals, to uncover nuances.
- Change Variables: Explore edge cases by tweaking previously agreed-upon assumptions, constraints, or perspectives to assess flexibility.
- Don't Overreach: Recognize the moment when discussions exceed the current breadth of knowledge and reset with foundational issues.
- While conversational, this remains an interrogative back-and-forth, with the responder compelled to respond until the questioner feels all relevant dimensions have been addressed.
- Through graded, narrowed questions, we scaffold complexity across interactions rather than demanding it prematurely and out of context. Each exchange pushes the dialogue a little further until a topic is satisfactorily exhausted.
Case Study - Discussing Algorithm Bias
For example, suppose you want to discuss the dangers of bias in algorithms. Don't immediately expect language models to resolve this vast issue with all its nuances in one answer. It is better to proceed step by step:
Q: What does "algorithmic bias" mean?
A: A basic definition establishes the primacy of bias in algorithmic systems over other contexts before assessing societal impact.
Q: What types of algorithms are we discussing that exhibit bias?
A: Being precise about subsets of algorithms that are prone to bias versus others avoids overgeneralizations.
Q: For the types identified as exhibiting bias, what are examples of sensitive contexts in which bias causes harm?
A: With the classes of algorithms and manifestations of bias clarified, discussing contexts involving discrimination or fairness provides helpful examples before strategizing solutions.
Q: You previously listed examples of algorithms causing bias problems in financial lending, health care, and criminal justice. Could you elaborate on the specific cases and observed harms in these areas?
A: Requiring elaboration on specific cases, rather than vague generalities, better illustrates the pitfalls.
Iteratively exploring facets grounds the discussion before theorizing remedies. Jumping straight to "how do we solve algorithmic bias" without a graded foundation would produce an unworkable trope. Instead, staged priming through rigid boundaries concentrates knowledge.
When tackling complex topics, avoid fixating on oversimplified sound bites. Instead, adopt a learning mindset that uses multistep questioning that ladders up to sophistication once basic understanding is secured. This scaffolds engagement and screens out unwarranted assumptions that undermine the search for truth.
Future articles will provide further guidance on context setting, precision questioning, and assessing model limitations to enrich questioning.
Glossary
- Reductionism - The practice of oversimplifying complex topics by removing nuances to make them seem more understandable. This can misrepresent their true nature.
- Scaffolding - A strategy of breaking down complicated subjects into smaller steps or asking foundational questions first. This provides structural supports to then build up complexity.
- Stepwise inquiry - Asking a series of smaller, more narrowly focused questions to gradually build understanding of a complex topic before tackling open-ended, expansive questions about it.
- Terminology/Definitions - The specific terms, definitions, phrasings used in a domain. Establishing shared vocabulary is crucial for productive discussions.
- Context - The circumstances, settings, frameworks surrounding a particular issue or domain. Explicitly clarifying context ensures all parties interpret issues similarly.
- Hypothetical - An imagined or theoretical scenario posed to analyze "what if" cases and possibilities regarding a topic.
- Variables - Components of a system or issue that can be adjusted or changed to assess effects on outcomes. Evaluating edge cases by shifting variables tests robustness.
- Constraints - Boundaries, limits, or assumptions that narrow the scope of permissible solutions or answers within a line of reasoning.
- Overreach - Attempting to expand a topic beyond currently established knowledge or failing to recognize the limits of an information source. Resetting discussions when reaching the frontier of understanding.
- Actionability - The degree to which plans or recommendations can lead to tangible outcomes rather than vague or speculative guidance lacking substance. Multi-step questioning grounds advice in reality.