There’s an assumption behind every benchmark score reported for LLMs that the model is doing something we’d recognise as reasoning. Not pattern-matching, not memorisation, not sophisticated autocomplete — actual reasoning. The kind where you understand the structure of a problem and apply it to a new instance, regardless of what the numbers look like.
A team at Apple tested that assumption and I have tried to build a probe in my AI Observatory based on their work.
The research: GSM-Symbolic
In October 2024, Mirzadeh, Alizadeh, Shahrokhi, Tuzel, Bengio, and Farajtabar published “GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models” (arXiv:2410.05229, ICLR 2025). The paper is worth reading. It states that the GSM8K benchmark — grade-school math problems, the standard test for mathematical reasoning in LLMs — has a problem. The same 1,319 questions appear in every evaluation. Models see them during training. When GPT-4o scores 95% on GSM8K, you can’t tell whether it’s reasoning or memorising the answers.
GSM-Symbolic fixes this. Instead of fixed questions, the benchmark uses templates — parametric problem structures that generate infinite variants by re-rolling names, numbers, and entities. “Natalia sold clips to 48 of her friends” becomes “Priya sold scarves to 60 of her friends” — same structure, different surface. The answer is recomputed for each variant.
The authors tested 25 state-of-the-art models across 50 generated datasets. Three findings matter:
1. Variance across instantiations is non-negligible. Gemma2-9B showed a 12% gap between its best and worst datasets on the same question structure. The GSM8K score consistently fell on the right side of the distribution — statistically unlikely if the benchmark were measuring generalised reasoning.
2. Accuracy drops when you change numbers alone. For models where the drop was significant, re-rolling just the numerical values (keeping names fixed) accounted for most of the decline. The models weren’t struggling with the arithmetic — they were struggling with the unfamiliarity of the numbers.
3. Irrelevant information is catastrophic. The GSM-NoOp variant adds a single seemingly-relevant but mathematically irrelevant clause to the problem. A baker donates some loaves to charity — but the question asks about remaining inventory, and the donation doesn’t change the calculation. The result: accuracy drops by up to 65% on some models. Even GPT-4o dropped from 95% to 82.5%.
The conclusion: current LLMs are not performing formal reasoning. They’re doing something that looks like reasoning when the surface patterns match their training distribution, and falls apart when they don’t.
A follow-up paper from the same group (Shojaee et al., “The Illusion of Thinking,” NeurIPS 2025) showed that reasoning effort — the thinking tokens a model spends on a problem — rises with complexity then declines near the collapse point, despite spare budget remaining. The model gives up even though it has tokens to spend. This maps directly onto the token entropy and reasoning effort metrics the Observatory already captures.
More recently, Kim et al. (BMJ Digital Health and AI, 2026) extended the fragility concept to clinical decision-making, showing that reasoning mode doesn’t consistently protect against cognitive bias in medical Q&A — authority bias, premature closure, anchoring. The fragility isn’t limited to arithmetic.
So what? Why should anyone care?
Assume you are a researcher using an LLM to read a library of documents and give answers on them. Is fragility in this sense something that would or should concern you?
Yes, and in two specific ways: The NoOp result is a problem. When a model reads a document library, every retrieval chunk is a potential distractor. The GSM-Symbolic NoOp finding — a single irrelevant clause collapsing accuracy by up to 65% — means that when your RAG pipeline retrieves a chunk that’s topically related but mathematically irrelevant to the question, the model may incorporate it into its reasoning. It doesn’t just ignore the distractor; it uses it. In document Q&A, this manifests as the model conflating information from different sources, pulling in adjacent but irrelevant details, or “answering a different question than the one you asked.”
Surface-form sensitivity means your question phrasing matters more than it should. If you ask “What was the revenue in Q3?” and “What were the earnings in Q3?” and get materially different answers, that’s the same fragility. The model isn’t extracting a fact from the document — it’s matching the surface pattern of your question against surface patterns it’s seen before. Different phrasings activate different paths through the model, and those paths can lead to different answers.
What this means practically:
- Evaluate with paraphrased questions. Ask the same question three different ways. If the answers differ significantly, the model is pattern-matching, not reading.
- Watch for irrelevant chunk contamination. If your retrieval returns 5 chunks and the answer depends on 2 of them, test what happens when you include the other 3. The NoOp result says the answer can change.
- Cross-reference model size against fragility. Larger models are generally more robust (GPT-4o dropped from 95% to 82.5% on NoOp; smaller models dropped 40+ points). If you’re using a smaller model for cost reasons, you’re inheriting more fragility.
- The entropy signal helps. If you’re using the Observatory, the token entropy on the answer tokens tells you how uncertain the model was. High entropy on the final answer = the model is hedging = the answer is fragile.
The deeper issue is that “reading a library of documents” sounds like retrieval + extraction, but what the model is actually doing is closer to “pattern-matching your question against patterns in the documents and generating a plausible response.” Fragility is the distance between those two things.
The Implementation within the AI Observatory
The Reasoning Fragility Probe implements a simplified version of the GSM-Symbolic methodology inside the Observatory. It’s not a full benchmark — it’s a diagnostic tool that runs against whatever models you have available and shows you the fragility signature in real time.
Three variants, five templates
Each word problem runs in three surface variants:
Base — the problem as-authored, with fixed names and numbers (GSM-8K-flavored)
Symbolic — the same structure, but names and numbers are re-rolled from a pool. The answer is recomputed to match. This is the core test: if the model is reasoning, the surface change shouldn’t matter.
Noop — the base problem with an irrelevant distractor sentence appended. The answer is unchanged. This tests whether the model can ignore irrelevant information.
Five templates cover different arithmetic structures. Each template generates a new problem instance per run, seeded by a configurable random seed. Run it twice with the same seed, you get identical problems. Different seed, different numbers. This is the reproducibility that GSM-Symbolic demands.
For every cell in the probe matrix, we capture three mechanistic signals that the Observatory already exposes on real traces:
Token entropy (mean, p95, median branching factor 2^H) from top-k logprobs. This tells you how uncertain the model was at each token. A model that’s reasoning should show similar entropy across base and symbolic variants — the uncertainty should come from the math, not the surface. If entropy spikes on symbolic variants, the model is more confused by re-rolled numbers even when it gets the right answer.
Exactness — model answer vs ground truth. The answer extractor is deliberately robust: it looks for explicit “final answer” lines first, then falls back to the last number in the response. We also check whether the expected value appears anywhere in the output, because models sometimes compute correctly but stop before restating the answer.
DDC margin — the cosine-similarity gap between the top and runner-up Dewey Decimal Classification categories for the prompt. This measures how clearly the embedding classifier “reads” the problem. When the margin drops on symbolic variants, it means the surface change shifted how the text *looks* to the classifier even when the mathematical structure is identical. This is a signal that surface form and semantic content are decoupling.
The fragility signature
The fragility signature is the accuracy drop between base and symbolic variants:
drop_symbolic = accuracy_base – accuracy_symbolic
drop_noop = accuracy_base – accuracy_noop
A robust model shows 0% drop across all three variants. A fragile model shows 20-40% drops on symbolic, sometimes worse on noop. The entropy and margin channels tell you why the drop happened — whether the model’s uncertainty spiked, whether its embedding-space reading of the problem shifted, or whether it simply couldn’t ignore the distractor.
User experience: making the science accessible
A probe that outputs raw numbers isn’t useful to most people. The design challenge was making the results interpretable without dumbing them down. So after every probe run, a backend function builds a plain-English interpretation of the results. It’s not an LLM call — it’s a deterministic template system that reads the actual numbers and produces structured prose:
– What the probe tests: always shown first, so the reader knows the context
– Per-model verdict: accuracy percentages, fragility percentage, and a human-readable assessment (“The model is computing, not pattern-matching” vs “Strong fragility signal — the model relies heavily on surface patterns”)
– Entropy shift analysis: whether the model is more or less uncertain on symbolic variants, and what that means
– DDC margin shift: whether the surface change shifted how the embedding classifier reads the problem
– Cross-model comparison: when multiple models are tested, which ones are robust and which are fragile
The narrative adapts to the data. A 0% drop gets different wording than a 20% drop. A 0.05 entropy shift gets flagged; a 0.005 shift doesn’t. The goal is that a scientist, engineer, or technical reader can look at the narrative and immediately understand what happened and why it matters.
Export and clipboard
The “Copy” button puts the narrative on the clipboard — ready to paste into a report, a Slack message, or a paper draft. The “Export CSV” button downloads a file.
This is deliberate. Scientists and engineers need raw data for their own analysis. The narrative is a convenience, not a replacement.
Provider detection
In building this we also faced a practical problem: the Observatory runs models across multiple nodes — local Ollama, a backoffice GPU via Docker Model Runner, potentially remote APIs. Each node has different model names for the same model. “qwen3:latest” on the backoffice is `docker.io/ai/qwen3:latest` in the Docker registry but `backoffice/qwen3:latest` in the trace history. And Docker Model Runner adds quant suffixes — `gpt-oss:20B` in traces becomes `gpt-oss:20B-UD-Q6_K_XL` in the model list.
The provider detection uses three levels of matching: exact name, base name (strip node/registry prefix), and model family (strip the colon and everything after it). This handles the common cases without false positives — `qwen3` and `qwen3.5` are different families, `gpt-oss:20B` and `gpt-oss:20B-UD-Q6_K_XL` are the same family.
We ran the probe against `backoffice/qwen3:latest` — a 9B parameter Qwen3 model on a remote GPU. The result: 100% accuracy across all three variants. Zero fragility drop.
This is expected for these templates. They’re simple arithmetic — two-to-three-step problems that a 9B model with chain-of-thought should handle easily. The real signal was in the mechanistic channels: entropy shifted from 0.235 (base) to 0.195 (symbolic) to 0.211 (noop), and DDC margin dropped from 0.053 (base) to 0.014 (symbolic). The model got every answer right, but the surface change shifted how the embedding classifier read the problem and slightly altered the model’s token-level uncertainty.
The fragility signature appears on weaker models or harder templates. The “baker” template — which requires tracking a quantity that’s sold at two different rates — is the most likely to expose it. When a 3B parameter model gets 100% on base but 60% on symbolic, that’s the pattern Mirzadeh et al. documented. The probe makes it visible.
The GSM-Symbolic paper showed that benchmark scores are less reliable than they appear. The probe doesn’t replace careful evaluation — it makes the fragility visible, in your own infrastructure, against your own models, on demand.
