The Half-Life of an Instruction
Comments on “The Half-Life of an Instruction”
Claim-checked. Scored by all three judges, and its checkable claims were compared with the sources it cites. How far stories are checked
A benchmark run on DeepSeek models finds that standing instructions erode fast: with thinking off, two earlier turns in French made V4 Pro ignore a "translate to Russian" rule in 90 of 90 trials, while thinking mode held at 89–100% through 50 turns. A single reminder beside the question also restored most adherence. A sliding-window memory cost 54% more than append-and-compact on a 12-turn replay, though that figure and the cache rules are DeepSeek-specific and come from the author's own runs. Rules that must hold belong in code.
Why it ranked
No. 9 of 124 with a score of 0.72. The panel rated it highest on technical depth (0.90) and practical relevance (0.89), lowest on significance (0.61).
Every answer behind this is in the table under Score.
Score
0.72total under rubric v2.2
- Significance0.61
- Reader interest0.84
- Audience breadth0.64
- Practical relevance0.89
- Substantiated0.81
- Technical depth0.90
- New information0.80
- Counts against: Promotional0.08
- Counts against: Overclaim0.33
Each judge’s answer per question (0–1), their mean and spread, and what the question adds to the score (weight × mean).
| Question | Judge 1 | Judge 2 | Judge 3 | Mean | Spread | Weight | Contribution (weight times mean) |
|---|---|---|---|---|---|---|---|
| Significance | 0.62 | 0.50 | 0.72 | 0.61 | 0.22 | +0.24 | +0.147 |
| Reader interest | 0.75 | 0.85 | 0.93 | 0.84 | 0.18 | +0.26 | +0.219 |
| Audience breadth | 0.60 | 0.60 | 0.72 | 0.64 | 0.12 | +0.13 | +0.083 |
| Practical relevance | 0.94 | 0.85 | 0.88 | 0.89 | 0.10 | +0.22 | +0.196 |
| Substantiated | 0.90 | 0.70 | 0.82 | 0.81 | 0.20 | +0.05 | +0.040 |
| Technical depth | 0.99 | 0.85 | 0.86 | 0.90 | 0.14 | +0.05 | +0.045 |
| New information | 0.90 | 0.60 | 0.90 | 0.80 | 0.30 | +0.05 | +0.040 |
| Promotional | 0.06 | 0.15 | 0.02 | 0.08 | 0.13 | −0.19 | −0.015 |
| Overclaim | 0.40 | 0.35 | 0.24 | 0.33 | 0.16 | −0.11 | −0.036 |
| Sum of contributions | +0.721 | ||||||
| Claim-check penalty | 0.000 | ||||||
| Total, as published | 0.721 | ||||||
Spread is the highest minus the lowest judge answer. marks a spread above 0.40: the judges disagree, and the story is marked contested.
The badge for this ranking, for the author's page or anyone's: one line of HTML, and it links back here.
<a href="https://rubric.hrih.si/editions/2026-10-27/9/"><img src="https://rubric.hrih.si/badge/2026-10-27/9.svg" alt="Ranked #9 on Rubric, 27 Oct 2026" width="238" height="28"></a>Claim check
Claim check: 3 matches source · 2 no source given · 3 couldn’t check. Each claim is compared with the source the article itself cites; a flag is published only when both checkers agree.
Matches sourceBenchmark result
Camassa and Shiller's paper reports instruction half-lives of 31 turns for Claude Opus 4.6, 22 for Claude Sonnet 4.6 and 3 for Gemma-3 12B.
The cited source says the same thing.
The article says “Claude Sonnet 4.6 22 paper Claude Opus 4.6 31 paper”
The source says “In these conditions, GPT-5.2 maintains the latest transition point ( N{=}43 ), followed by Claude Opus 4.6 ( N{=}31 ), then Claude Sonnet 4.6 and Llama 3.3 70B ( N{=}22”
Source cited by the article: arxiv.org/abs/2605.20382
Couldn’t checkBenchmark result
In the paper, GPT-5.2 without reasoning follows the instruction 47% to 66% of the time at every N, and with reasoning averages 88% and never drops to half within 50 turns.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “GPT-5.2 without reasoning follows the instruction between 47% and 66% of the time at every N; with reasoning (OpenAI’s name for thinking mode) switched on, it averages 88%”
The source says No passage could be compared.
Source cited by the article: arxiv.org/abs/2605.20382
No source givenNumber
With thinking mode on, DeepSeek V4 Pro followed the Russian-translation instruction between 89 and 100 times in 100 at every number of turns tested, up to 50, whereas with thinking off two French examples made it answer in French 90 of 90 times.
The article gives no link or citation for it.
The article says “Two, and it answered in French 90 times out of 90. Switch on thinking mode”
The source says No source to compare with.
Couldn’t checkNumber
DeepSeek's off-peak price for Flash on 26 October 2026 was $0.003 per million cached input tokens and $0.15 per million uncached, a 50× gap.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “DeepSeek’s off-peak price for Flash was $0.003 per million cached input tokens and $0.15 per million uncached: 50×”
The source says No passage could be compared.
Source cited by the article: api-docs.deepseek.com/quick_start/pricing
No source givenNumber
On DeepSeek Flash, a sliding-window memory strategy cost 54% more than append-and-compact over the same 12-turn conversation.
The article gives no link or citation for it.
The article says “it cost 54% more than append-and-compact for the same conversation”
The source says No source to compare with.
Matches sourceNumber
Microsoft's Fork-and-Flush runs of the same autoresearch agent plateaued at different scores between 68 and 100, and forking into fresh contexts improved the normalised score by 66% over a single run across 13 tasks at equal compute.
The cited source says the same thing.
The article says “improved the normalised score by 66% over a single run across 13 tasks, at equal compute”
The source says “Their final best scores range from 68.3 to 100.0, and the gaps remain wide even after considerable additional compute”
Source cited by the article: arxiv.org/abs/2610.07447
Matches sourceBenchmark result
A JetBrains team found that on SWE-bench Verified, simply masking old tool output halved cost and matched LLM summarisation.
The cited source says the same thing.
The article says “simply masking old tool output halved cost and matched LLM summarisation”
The source says “We find that a simple environment Observation Masking strategy halves cost relative to the Raw Agent while matching, and sometimes slightly exceeding, the solve rate of LLM-Summary”
Source cited by the article: arxiv.org/abs/2508.21433
Couldn’t checkGeneralization
DeepSeek's cache stores whole units, so a request only hits if it matches a stored unit completely, which is stricter than other providers' prefix caching.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “DeepSeek is stricter still: its cache stores whole units, and a request only hits if it matches a stored unit completely”
The source says No passage could be compared.
Source cited by the article: api-docs.deepseek.com/guides/kv_cache
Also covered by
No other coverage was clustered with this story.
Elsewhere
Elsewhere: HN 2 pts, not on front page
Recorded for context only, checked 27 Oct 2026, 05:46 UTC. It played no part in the ranking.