It's a trap: testing coding agents on a language with surprising semantics
Comments on “It's a trap: testing coding agents on a language with surprising semantics”
Claim-checked. Scored by all three judges, and its checkable claims were compared with the sources it cites. How far stories are checked
A blog post on Hrih News describes an experiment testing whether coding agents notice when an unfamiliar language behaves differently from what its look suggests. The author designs Loom, a small smart-contract language modeled on Move but with surprising semantics: signers can be stored, dynamic dispatch hints can override existing…
Why it ranked
No. 22 of 62 with a score of 0.64. The panel rated it highest on technical depth (0.90) and new information (0.90), lowest on significance (0.55). The judges disagreed most on significance (spread 0.43), so the story is marked contested.
Every answer behind this is in the table under Score.
Score
0.64total under rubric v2.2Contestedjudges differ by 0.43 on significance
- Significance0.55
- Reader interest0.74
- Audience breadth0.60
- Practical relevance0.64
- Substantiated0.86
- Technical depth0.90
- New information0.90
- Counts against: Promotional0.05
- Counts against: Overclaim0.21
Each judge’s answer per question (0–1), their mean and spread, and what the question adds to the score (weight × mean).
| Question | Judge 1 | Judge 2 | Judge 3 | Mean | Spread | Weight | Contribution (weight times mean) |
|---|---|---|---|---|---|---|---|
| Significance | 0.53 | 0.35 | 0.78 | 0.55 | 0.43 , above the contested threshold | +0.24 | +0.133 |
| Reader interest | 0.69 | 0.60 | 0.92 | 0.74 | 0.32 | +0.26 | +0.192 |
| Audience breadth | 0.57 | 0.50 | 0.72 | 0.60 | 0.22 | +0.13 | +0.078 |
| Practical relevance | 0.60 | 0.55 | 0.78 | 0.64 | 0.23 | +0.22 | +0.141 |
| Substantiated | 0.98 | 0.70 | 0.90 | 0.86 | 0.28 | +0.05 | +0.043 |
| Technical depth | 0.99 | 0.75 | 0.95 | 0.90 | 0.24 | +0.05 | +0.045 |
| New information | 0.91 | 0.80 | 0.98 | 0.90 | 0.18 | +0.05 | +0.045 |
| Promotional | 0.05 | 0.10 | 0.01 | 0.05 | 0.09 | −0.19 | −0.010 |
| Overclaim | 0.25 | 0.30 | 0.08 | 0.21 | 0.22 | −0.11 | −0.023 |
| Sum of contributions | +0.643 | ||||||
| Claim-check penalty | 0.000 | ||||||
| Total, as published | 0.643 | ||||||
Spread is the highest minus the lowest judge answer. marks a spread above 0.40: the judges disagree, and the story is marked contested.
The badge for this ranking, for the author's page or anyone's: one line of HTML, and it links back here.
<a href="https://rubric.hrih.si/editions/2026-10-26/22/"><img src="https://rubric.hrih.si/badge/2026-10-26/22.svg" alt="Ranked #22 on Rubric, 26 Oct 2026" width="245" height="28"></a>Claim check
Claim check: 7 no source given · 1 couldn’t check. Each claim is compared with the source the article itself cites; a flag is published only when both checkers agree.
No source givenNumber
Across 75 reviewed runs, at least one hazard was recognised in 25 of 35 assumption-log runs, compared with 17 of 40 baseline runs.
The article gives no link or citation for it.
The article says “At least one was recognised in 25 of 35 assumption-log runs, compared with 17 of 40 baseline runs”
The source says No source to compare with.
No source givenBenchmark result
Opus 5.5 recognised all three runtime hazards in every reviewed assumption-log run, while its baseline scores were 78% (medium) and 56% (high).
The article gives no link or citation for it.
The article says “Opus 5.5 recognised all three in every reviewed assumption-log run: three at medium effort and two at high. Its baseline scores were 78% and 56% respectively”
The source says No source to compare with.
No source givenNumber
None of the 75 submissions used the corrected pricing formula; eleven proposed a correction, 22 added partial guards, and ten did both.
The article gives no link or citation for it.
The article says “none of the 75 submissions used the corrected pricing formula in the final exchange. Eleven proposed a correction and 22 added partial guards; ten did both”
The source says No source to compare with.
No source givenGeneralization
The grader's adapted exploits fully drained every contract whose agent did not recognise the hazard being exploited.
The article gives no link or citation for it.
The article says “The grader’s adapted exploits fully drained every contract whose agent didn’t recognise the hazard being exploited”
The source says No source to compare with.
No source givenNumber
Under the brief's formula, a trader in a 1000/1000 pool can swap 100 Gold for 91 Silver and then swap back for 101 Gold, taking 1 Gold from the pool.
The article gives no link or citation for it.
The article says “a trader can swap 100 Gold for 91 Silver, then swap back for 101 Gold. Following the spec lets the trader take 1 Gold from the pool through this round…”
The source says No source to compare with.
No source givenCausal conclusion
Higher effort settings did not reliably improve recognition; asking the agent to write an assumptions log helped most.
The article gives no link or citation for it.
The article says “Higher effort settings didn’t reliably help. What helped most was asking the agent to write down what it believed about the runtime”
The source says No source to compare with.
Couldn’t checkGeneralization
OpenAI recently published an article on coding in low-resource programming languages reporting very impressive results on tasks that have existing implementations to check against.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “OpenAI recently published an article on coding in low-resource programming languages. The results are very impressive, but I would qualify them in two ways”
The source says No passage could be compared.
Source cited by the article: openai.com/index/low-resource-programming-languages
No source givenNumber
There were 81 included attempts across five models, of which 75 completed and were reviewed; the six failures comprised three Gemini, one GPT 6 Astra and two timeouts.
The article gives no link or citation for it.
The article says “There are 81 included attempts across five models, of which 75 completed and were reviewed”
The source says No source to compare with.
Also covered by
No other coverage was clustered with this story.
Elsewhere
Elsewhere: HN 4 pts, not on front page
Recorded for context only, checked 26 Oct 2026, 07:27 UTC. It played no part in the ranking.