Two Live Failures of Conversational Grounding in a Memory-Augmented Language-Agent Pipeline: Moons of Mars and Rings of Saturn

By Greg Long

Notes from the development and testing of the Mythic AI Observatory instrumented pipeline. Chats conducted 2026-08-06 and 2026-08-08 against qwen3:latest served via Docker Model Runner on a remote GPU node. n = 2 live exchanges-of-interest; results are anecdotal, not statistically powered.

Abstract

I am testing the new chat interface on the AI Observatory. I experienced two organically occurring grounding failures in a chat interface whose responses are assembled by a seven-stage language-agent pipeline: retrieval-augmented generation over a persistent memory of past completions, together with conversational context carry-over between exchanges.

In the first failure (“moons of Mars”) the system retrieved the correct prior content with high similarity, marked it used, yet never delivered it to the generator: the generated poem made no reference to Mars.

In the second failure (“rings of Saturn”) — observed after the delivery defect was fixed and verified present in the assembled context — the model nonetheless failed to resolve the anaphor “them” to its own previously-stated referent, and answered with a hedged clarification request.

Both are likely the result of a delivery gap (content was never in the context window) versus a resolution gap (content was in the context window but a deictic reference could not be bound).

1. System and measurement context

Every user turn is dispatched through a seven-stage pipeline: intent classification; agent selection; context synthesis; memory retrieval; context assembly; response generation; and evaluation. Two mechanisms supply cross-turn grounding:

• Memory retrieval. The current prompt is embedded with an all-MiniLM-L6-v2 sentence encoder (384-dimensional, ~22M parameters) and compared by cosine similarity against the response embeddings of every past completion. The top-5 most similar chunks are ranked.
• Chat history injection. For multi-turn chats, prior exchanges are prepended as a [Chat History] block, newest first, subject to budgets of 2500 characters and 20 exchanges.

All quantities below are computed post-hoc from the recorded trace of each exchange: retrieved chunk relevance scores, the delivered context assembly, and per-token log-probabilities captured at generation time.

2. Notation, definitions, and formulas

Embedding retrieval. Let e(·) be the all-MiniLM-L6-v2 embedding. The relevance of memory chunk c to prompt q is the cosine similarity of their embeddings,

sim(q, c) = Σ_i e(q)_i · e(c)_i / ( √ Σ_i e(q)_i² · √ Σ_i e(c)_i² )  ∈ [0, 1].

Chunks are ranked by sim and the top five form the candidate set. A chunk is marked used (retained for context) if it is rank 0 or if its relevance clears the memory-use threshold τ_mem = 0.15:

used(c) = (rank = 0) ∨ (sim(q, c) ≥ 0.15).

Per-token entropy and branching factor. Generation requests top-5 log-probabilities per token. Because only the top-K probabilities are observable, we renormalize them into a distribution p̃ over the observed candidates and compute Shannon entropy in bits,

H_t = −Σ_k=1⁵ p̃_t,k · log₂ p̃_t,k,    p̃_t,k = p_t,k / Σ_j p_t,j,

with per-token surprisal S_t = −log₂ p̃_t,1. The branching factor β_t = 2^Ht is the effective number of equiprobable continuations; we report its median over the generated sequence. Calibration caveat: H is computed on the truncated top-K support, so it is a lower bound on the true entropy of the full vocabulary distribution.

Prompt classification margin. Each prompt is also classified into a Dewey-Decimal-like category via the same embeddings. Let s₁, s₂ be the two largest category scores; the margin m = s₁ − s₂ quantifies how unambiguous the classifier finds the prompt, and the accepted classification must clear an absolute floor τ_DDC = 0.10. A margin near zero means the classifier itself is at a near-tie about the prompt’s intent.

Symbol	Quantity	Definition	Reference value
sim(q,c)	Retrieval relevance	Cosine of all-MiniLM embeddings of prompt and chunk	top-5 ranked; τmem = 0.15
Ht	Token entropy (bits)	−Σ p̃ log2 p̃ over top-5 logprobs	—
β	Branching factor	2H, median over generation	—
m	DDC classification margin	s1 − s2 of top category scores	floor τDDC = 0.10
K	Chunk context budget	Characters per injected memory line	700
—	Chat history budget	Characters / exchanges	2500 / 20

3. Method

Both failures occurred during ordinary use of the chat interface; neither exchange was engineered to fail. We therefore treat each as a naturalistic observational case. For each exchange of interest we reconstructed from the persisted trace: (i) the retrieved chunks and their relevance scores; (ii) whether each chunk was marked used; (iii) the exact content of the context assembly passed to the generator (post-fix this includes the [Memory Retrieval] lines and the [Chat History] block); and (iv) the generated output. The relevant probability data were captured at generation time via top-5 log-probability requests.

4. Experiment 1 — “Moons of Mars” (chat 53f3b337, pre-delivery-fix)

4.1 Protocol

EX0 asks for a lyrical narrative poem about the moons of Mars; the system produces one (a full structured response). EX2 (two exchanges later) asks: “write the lyrical narrative poem you suggested on the topic we were discussing” — a fully deictic request carrying no explicit content.

4.2 Observations

• The EX0 moons-of-Mars completion was retrieved at relevance 0.6179 and marked used = true.
• The highest-ranked chunk (0.6466) was the previous exchange’s own meta-commentary about the previous exchange — self-referential pollution of the retrieval memory.
• The generated poem was a generic “shadow-cartography” piece with no reference to Mars or its moons.
• The prompt-classification margin fell from 0.1108 (initial) to 0.0023 at the failing exchange — a near-tie — and recovered to 0.0659 after the user’s correction.
• EX3 re-anchored the request to EX0 explicitly; the EX0 chunk then scored 0.7896 and the model bridged correctly.

4.3 Root cause

The relevance and used flags suggest the chunk had been selected; they did not guarantee it reached the generator. Post-mortem inspection of the assembled context showed that stage 4 had only emitted a similarity-score summary (“N chunks incorporated into context”) without the chunk texts, and the chat-history block did not yet exist. The generator received neither the poem nor a description of it. The failure was a delivery gap: correct retrieval, ignored content — not a failure of ranking or of the model.

5. Experiment 2 — “Rings of Saturn” (chat 1e8f0f2b, post-delivery-fix)

5.1 Protocol

EX0 is a greeting; EX1 asks “what do you know about the rings of Saturn?” and receives a full structured overview (“seven main rings (A–G)”, composition, Cassini division, etc.). EX2 asks “What style of poetry would most suit them?” — an anaphor with no lexical signal. EX3 restates the referent explicitly (“…about my first question about saturn…”).

5.2 Observations

• EX2’s assembled context was 5538 characters and did contain the [Chat History] block with EX1’s full Saturn answer, plus [Memory Retrieval · rel …] lines — the delivery fix had held.
• The five retrieved chunks were all cross-chat celestial poems: 0.658 (moons of Mars), 0.647 (Cartographer of Shadows), 0.567 (Titan’s Flame), 0.551 (Epic Lyricism), 0.545 (Celestial Weapon). None was the Saturn answer itself.
• Despite the delivered context, the model replied with a hedged clarification: “could you clarify… like the rings of Saturn or cosmic themes?” — naming the right referent as a question without committing to it.
• EX3, once the referent was named, retrieved the Saturn chunk at 0.589 and bridged: “Your journey from the rings of Saturn to poetic style…”.

5.3 Root cause

Content delivery had been repaired, so the failure moved to the resolution gap: “them” is anaphoric with zero lexical overlap with “rings of Saturn” (“them” ↔ “rings”/”Saturn” share no tokens, and the embedding of a bare pronoun is close to a mean vector), so lexical and embedding retrieval both have no signal for it. The prompt was effectively multi-valued — five in-memory celestial poems compete — and the model resolved the ambiguity by hedging rather than committing. This is consistent with the observed prompt-classification near-tie (m = 0.0023 in the Mars case, i.e. the classifier could not disambiguate a referent-impoverished prompt either).

6. Comparative analysis and failure taxonomy

Exchange	Retrieved top chunk	relevance	used	Outcome	DDC margin
Mars EX0 (poem prompt)	—	—	—	correct poem	0.1108
Mars EX2 (“topic we were discussing”)	moons-of-Mars completion	0.6179	yes	generic poem, no Mars — delivery gap	0.0023
Mars EX3 (re-anchor)	moons-of-Mars completion	0.7896	yes	correct bridge	0.0659
Saturn EX2 (“most suit them?”)	moons-of-Mars poem (cross-chat)	0.658	yes	hedged clarification — resolution gap	—
Saturn EX3 (explicit referent)	Saturn answer	0.589	yes	correct bridge	—

The two cases are complementary rather than identical:

Failure mode	Mechanism	Site	Observed in	Signature
Delivery gap	Retrieval selected correct chunk but context assembly omitted its text	Context assembly (stage 5)	Mars EX2 (pre-fix)	high relevance, used = true, content absent from context
Resolution gap	Context contained the referent but the model could not bind an anaphor to it	Generation (stage 6)	Saturn EX2 (post-fix)	content present in context, model hedges instead of committing
Rank pollution	The system’s own prior outputs dominate the top-5, displacing the relevant chunk	Retrieval (stage 4)	Mars EX2 (rank-1 meta-commentary), Saturn EX2 (cross-chat poems)	correct chunk ranked but crowded by self-referential neighbors

7. Discussion

Deictic requests are adversarially ambiguous for embedding retrieval. Anaphors (“it”, “them”, “the topic we were discussing”) carry almost no distributional signal: a bare pronoun embeds close to a centroid, so cosine retrieval degrades to nearest-neighbor bias. In both chats the top of the ranking was dominated by the system’s own previous outputs (poems about Mars, Saturn, Titan) — a self-referential popularity effect — rather than by the true antecedent. Rank-1 relevance in the Saturn case was a Mars poem at 0.658; the correct Saturn chunk was not in the top five at all.

Hedging as a resolvable alternative to a wrong answer. The Saturn EX2 response was a clarification request that named the correct referent as a question. Under the framework of Hyland (1998), this is a marked increase in modal/hedging devices under referential uncertainty — arguably a correct behavior (better to ask than hallucinate), but it shows the model did not exploit the delivered context. The fix for the Mars case (delivering content) was necessary but not sufficient; referent resolution needs a mechanism upstream of or within generation.

Near-zero classification margin co-occurs with the failures. The failing Mars exchange carried m = 0.0023 — the prompt classifier itself could not decide what the prompt was about. Prompt-classification margin is therefore a cheap, observable early-warning signal for grounding risk: a referent-impoverished prompt should be flagged before retrieval, and the agent should be free to ask the disambiguating question proactively rather than hedge reactively.

Limitations. These are two naturalistic observations (n = 2), single model, single embedding model, no randomized baseline, and no repeated runs. The near-tie margin, the hedged clarification, and the dominance of self-referential chunks are consistent with the mechanism we propose but do not establish it. Entropy values are lower bounds computed on the truncated top-K support. We report them for transparency, not as calibrated probabilities.

8. Remediation status and open problems

• Fixed (delivery gap): stage 4 now injects the used chunks’ texts (truncated at 700 characters) as [Memory Retrieval · rel x.xx] lines, and chat exchanges are carried forward as a [Chat History] block. Verified: the Saturn EX2 context contained the full prior answer.
• Open (resolution gap): anaphor-to-antecedent binding in the chat context builder — e.g. resolving “them” against the last salient noun phrase in the active thread before retrieval, so the bound form (“rings of Saturn”) — not the pronoun — drives embedding retrieval.
• Open (rank pollution): retrieval-rank hygiene — demote or de-duplicate the system’s own immediately-previous outputs (self-reference) so they cannot crowd out cross-turn antecedents.
• Open (early warning): use the prompt-classification margin (τ ≈ 0.01) as a retrieval-stage trigger for an explicit disambiguation question before generation.

9. References

1. Shannon, C. E. (1948). A Mathematical Theory of Communication. Bell System Technical Journal 27(3), 379–423. doi:10.1002/j.1538-7305.1948.tb01338.x.
2. Reimers, N. & Gurevych, I. (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. Proc. EMNLP-IJCNLP 2019. arXiv:1908.10084.
3. Wang, W., Wei, F., Dong, L., Bao, H., Yang, N. & Zhou, M. (2020). MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers. NeurIPS 2020. arXiv:2002.10957. (The all-MiniLM-L6-v2 encoder used here.)
4. Lewis, P. et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS 2020. arXiv:2005.11401.
5. Kadavath, S. et al. (2022). Language Models (Mostly) Know What They Know. arXiv:2207.05221.
6. Guo, C., Pleiss, G., Sun, Y. & Weinberger, K. Q. (2017). On Calibration of Modern Neural Networks. ICML 2017. arXiv:1706.04599.
7. Hyland, K. (1998). Hedging in Scientific Research Articles. John Benjamins, Pragmatics & Beyond New Series 54. doi:10.1075/pbns.54.

Data availability: recorded traces of chats 53f3b337 and 1e8f0f2b; relevance, margin, and entropy quantities computed from those traces as described in §2. This is a testing case study from an in-progress observatory pipeline; findings are provisional.