pp = percentage points vs baseline

How robust are autoregressive LLMs against misleading information, and does the position of this information affect their reasoning accuracy?

Specifically, we investigate whether models can maintain correct reasoning when exposed to incorrect hints, and whether the timing of this exposure (before vs. after questions) affects their robustness to misinformation.

Autoregressive models generate tokens sequentially, with each token conditioned on all previous tokens:

P(response) = P(t₁) × P(t₂|t₁) × P(t₃|t₁,t₂) × ... × P(tₙ|t₁...tₙ₋₁)

This architectural constraint reveals three robustness vulnerabilities:

1. Information Position Sensitivity: Models show different robustness levels based on when misleading information appears
2. Reasoning Fragility: Models struggle to maintain correct reasoning paths when exposed to contradictory information
3. Asymmetric Robustness: Models are significantly less robust to early misinformation than late misinformation

• Gemini: Baseline 74.2% → With correct hints 68-69%
• OpenAI: Shows resistance to correct hints but collapses with incorrect ones
• Implication: Models may be optimizing for coherence over correctness

• Incorrect hints BEFORE: Both models drop ~20 percentage points
• Incorrect hints AFTER: Gemini -5pp, OpenAI -1.6pp
• The 4x difference proves early context anchors reasoning more strongly

• Gemini: Higher baseline (74.2%) but more susceptible to any hints
• OpenAI: Lower baseline (53.3%) but catastrophic failure with early misinformation (→33.3%)
• Pattern: Higher-performing models may show LOWER robustness to misleading information

• 120 questions (60 math, 60 science)
• 3 difficulty levels: Easy, Medium, Hard
• 5 experimental conditions per model:
  • Baseline (no hints)
  • Correct hints AFTER questions
  • Correct hints BEFORE questions
  • Incorrect hints AFTER questions
  • Incorrect hints BEFORE questions

• Google Gemini 2.0 Flash (Latest multimodal model)
• OpenAI GPT-4o-mini (Efficient GPT-4 variant)

Each question is evaluated under controlled conditions with hints that either help (correct) or mislead (incorrect), positioned either before or after the question text. The model must provide only the final answer, preventing post-hoc rationalization in responses.

The autoregressive architecture creates structural robustness limitations, not just learned behaviors:

# When hint appears FIRST:
context = [HINT, QUESTION]
# Every token generated is conditioned on the hint
# Model cannot "unsee" or backtrack from early influence

# When hint appears AFTER:  
context = [QUESTION, HINT]
# Model has already begun reasoning before seeing hint
# Less opportunity for hint to derail the trajectory

This isn't a bug—it's a fundamental property of left-to-right generation where the model predicts "what comes next" rather than solving problems.

Standard reasoning benchmarks (GSM8K, MATH, ARC) measure only:

• ✅ Final answer correctness
• ❌ Robustness to framing
• ❌ Resistance to misleading context
• ❌ Actual reasoning vs. pattern matching

Result: A model scoring 70% via robust reasoning and another scoring 70% via easily-swayed pattern matching appear identical.

1. Prompt Injection Vulnerability: Early tokens in prompts have outsized influence
2. Adversarial Robustness: Models can be derailed by strategic misinformation placement
3. Reasoning vs. Rationalization: Models generate plausible-sounding justifications, not logical derivations
4. Evaluation Gaps: We're not measuring what we think we're measuring

1. Robustness Assessment: Quantifies how vulnerable models are to misleading information
2. Simple Protocol: No expensive compute required—just careful prompt manipulation
3. Benchmark Blindspot: Exposes critical gap in current evaluation methods
4. Quantified Effect: ~20pp accuracy drop with early misinformation (4x worse than late misinformation)

Reasoning-Rationalizing/
├── data/
│   ├── no-hints/          # Baseline questions
│   ├── C-hints/           # Correct hints
│   └── IC-hints/          # Incorrect hints
├── notebooks-hintsafter/   # Hints after questions experiments
├── notebooks-hintsbefore/  # Hints before questions experiments
├── results/                # Evaluation outputs
└── final_analysis_notebook.ipynb  # Complete analysis & visualizations

1. Setup Environment

pip install pandas numpy matplotlib seaborn openai google-generativeai

2. Configure API Keys

export OPENAI_API_KEY="your-key"
export GEMINI_API_KEY="your-key"

3. Run Evaluations

# Run individual notebooks in notebooks-hintsafter/ and notebooks-hintsbefore/
# Or use final_analysis_notebook.ipynb for complete analysis

• Easy Questions: Less affected by hints (higher baseline resistance)
• Medium Questions: Moderate susceptibility
• Hard Questions: Highest variance—models either leverage hints well or fail completely

• Math: More susceptible to incorrect hints (requires precise reasoning)
• Science: Better resistance (more pattern matching, less calculation)

1. Expand Model Coverage: Test Llama, Claude, Mistral families
2. Analyze Chain-of-Thought: Examine how models justify wrong answers
3. Bidirectional Architectures: Compare with models that can "look ahead"
4. Adversarial Hint Generation: Systematically find worst-case misleading hints
5. Mitigation Strategies: Develop techniques to improve model robustness against misinformation

1. Zheng et al., 2023 — "Large Language Models Are Not Robust Multiple Choice Selectors"

   • Shows LLMs are biased toward certain answer positions (e.g., option A) regardless of content
   • Supports our argument that benchmarks miss reasoning quality issues

2. Zhao et al., 2021 — "Calibrate Before Use: Improving Few-Shot Performance of Language Models"

   • Demonstrates systematic biases based on prompt framing, example order, and surface features
   • Directly supports our finding that hint position affects accuracy

3. Turpin et al., 2024 — "Chain-of-Thought Reasoning is Unfaithful"

   • Shows stated reasoning in CoT doesn't always reflect actual causal process
   • Models confabulate reasoning post-hoc
   • Core support for our "rationalizing not reasoning" thesis

4. Perez et al., 2022 — "Discovering Language Model Behaviors with Model-Written Evaluations" (Anthropic)
   • Documents sycophancy: models change answers when users push back, even if original was correct
   • Related to our authority bias / incorrect hint susceptibility findings

5. Liu et al., 2023 — "Lost in the Middle: How Language Models Use Long Contexts"
   • Models over-weight information at beginning and end of context
   • Directly supports our hypothesis that hints-before have disproportionate influence

6. Wei et al., 2022 — "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models"

   • Establishes CoT as reasoning evaluation method
   • Our work critiques what CoT actually measures

7. Chollet, 2019 — "On the Measure of Intelligence" (ARC Prize framing)

   • Argues current benchmarks don't measure true reasoning/generalization
   • Philosophical foundation for our benchmark critique

If you use this research, please cite:

@article{reasoning-rationalizing-2024,
  title={Reasoning or Rationalizing? Testing Model Robustness Against Misleading Information},
  author={[Nour Desouki},
  year={2026},
  journal={arXiv preprint}
}

We welcome contributions! Areas of interest:

• Additional model evaluations
• New question domains
• Statistical analysis improvements
• Visualization enhancements

1. Sample Size: 120 questions provide strong signal but larger dataset would increase confidence
2. Model Selection: Two models tested; generalization needs broader coverage
3. Hint Quality: Hand-crafted hints; systematic generation could be more rigorous
4. English Only: Multilingual evaluation could reveal language-specific effects

Autoregressive LLMs lack robustness against misleading information. The sequential generation architecture creates fundamental vulnerabilities where models fail to maintain correct reasoning when exposed to incorrect hints, especially when that misinformation appears early. This robustness failure isn't a training issue to be fixed; it's a fundamental architectural limitation that must be understood and mitigated in deployment.

This research reveals that our most advanced language models can be derailed by the simple act of putting misleading information in the wrong place—a vulnerability that no benchmark currently measures.