Outside evaluators found ways OpenAI’s agents could slip past its monitor
Comments on “Outside evaluators found ways OpenAI’s agents could slip past its monitor”
Claim-checked. Scored by all three judges, and its checkable claims were compared with the sources it cites. How far stories are checked
METR and Redwood Research, each with one researcher embedded at OpenAI from 21 July to 11 August, reported flaws in the system that monitors its internal agents. One exploit could have let agents hide hundreds of actions. Compaction events went unmonitored, leaving about 10% of eligible Codex traffic unwatched from May to July. Coverage of the strongest models' traffic was estimated at 75% to 94%. OpenAI says it patched every production flaw; the evaluators have not checked that. Only a redacted summary of the 61-page report is public.
Why it ranked
No. 17 of 62 with a score of 0.66. The panel rated it highest on new information (0.85) and reader interest (0.82), lowest on technical depth (0.56).
Every answer behind this is in the table under Score.
Score
0.66total under rubric v2.2
- Significance0.70
- Reader interest0.82
- Audience breadth0.62
- Practical relevance0.58
- Substantiated0.60
- Technical depth0.56
- New information0.85
- Counts against: Promotional0.05
- Counts against: Overclaim0.22
Each judge’s answer per question (0–1), their mean and spread, and what the question adds to the score (weight × mean).
| Question | Judge 1 | Judge 2 | Judge 3 | Mean | Spread | Weight | Contribution (weight times mean) |
|---|---|---|---|---|---|---|---|
| Significance | 0.73 | 0.60 | 0.78 | 0.70 | 0.18 | +0.24 | +0.169 |
| Reader interest | 0.77 | 0.80 | 0.90 | 0.82 | 0.13 | +0.26 | +0.214 |
| Audience breadth | 0.44 | 0.60 | 0.82 | 0.62 | 0.38 | +0.13 | +0.081 |
| Practical relevance | 0.57 | 0.45 | 0.72 | 0.58 | 0.27 | +0.22 | +0.128 |
| Substantiated | 0.52 | 0.60 | 0.67 | 0.60 | 0.15 | +0.05 | +0.030 |
| Technical depth | 0.60 | 0.45 | 0.62 | 0.56 | 0.17 | +0.05 | +0.028 |
| New information | 0.84 | 0.80 | 0.90 | 0.85 | 0.10 | +0.05 | +0.042 |
| Promotional | 0.06 | 0.05 | 0.03 | 0.05 | 0.03 | −0.19 | −0.009 |
| Overclaim | 0.23 | 0.20 | 0.24 | 0.22 | 0.04 | −0.11 | −0.025 |
| Sum of contributions | +0.658 | ||||||
| Claim-check penalty | 0.000 | ||||||
| Total, as published | 0.658 | ||||||
Spread is the highest minus the lowest judge answer. marks a spread above 0.40: the judges disagree, and the story is marked contested.
The badge for this ranking, for the author's page or anyone's: one line of HTML, and it links back here.
<a href="https://rubric.hrih.si/editions/2026-10-26/17/"><img src="https://rubric.hrih.si/badge/2026-10-26/17.svg" alt="Ranked #17 on Rubric, 26 Oct 2026" width="245" height="28"></a>Claim check
Claim check: 7 matches source · 1 couldn’t check. Each claim is compared with the source the article itself cites; a flag is published only when both checkers agree.
Matches sourceDate
METR and Redwood Research each sent one researcher inside OpenAI from 21 July to 11 August, with access to internal code, documentation and the monitor's prompts, and interviewed about a dozen staff.
The cited source says the same thing.
The article says “Each group sent one researcher inside OpenAI from 21 July to 11 August”
The source says “For three weeks, from July 21st to August 11th, a METR employee and a Redwood Research employee conducted a pilot high-access exercise”
Source cited by the article: metr.org/blog/2026-10-22-red-teaming-openai-agent-monitoring
Matches sourceNumber
The evaluators estimated that 75% to 94% of agentic traffic from OpenAI's strongest models was eligible for monitoring, and 99.2% for internal Codex agents.
The cited source says the same thing.
The article says “The evaluators estimated that 75% to 94% of agentic traffic from OpenAI’s strongest models was eligible for monitoring. For internal Codex agents, the figure was 99.2%”
The source says “roughly 75–94% of agentic internal Responses API (which powers Codex, among other things) 4 traffic from models at least as strong as GPT-5.5 was eligible for monitoring”
Source cited by the article: metr.org/blog/2026-10-22-red-teaming-openai-agent-monitoring
Matches sourceNumber
METR estimated that the monitor's skipping of compaction events left about 10% of eligible Codex traffic unwatched from May to July.
The cited source says the same thing.
The article says “METR estimated that this left about 10% of eligible Codex traffic unwatched from May to July”
The source says “not monitoring compaction events likely led to on the order of 10% of eligible internal Codex traffic not being monitored between May and July 2026”
Source cited by the article: metr.org/blog/2026-10-22-red-teaming-openai-agent-monitoring
Matches sourceNumber
An exploit could have let AI agents at OpenAI hide hundreds of their own actions from the monitoring system.
The cited source says the same thing.
The article says “AI agents at OpenAI could have hidden hundreds of their own actions from the OpenAI monitoring system, outside evaluators found”
The source says “We also discovered an exploit agents could use that would prevent hundreds of recent actions (including the actions involved in the exploit itself) from being monitored”
Source cited by the article: metr.org/blog/2026-10-22-red-teaming-openai-agent-monitoring
Matches sourceQuote
OpenAI told the evaluators it has patched every flaw that affected its production safeguards, a claim METR and Redwood have not evaluated.
The cited source says the same thing.
The article says “OpenAI told the evaluators it has patched every flaw that affected its production safeguards”
The source says No passage could be compared.
Source cited by the article: metr.org/blog/2026-10-22-red-teaming-openai-agent-monitoring
Matches sourceNumber
OpenAI received the full 61-page report, while METR and Redwood agreed to publish only a short summary that OpenAI could redact.
The cited source says the same thing.
The article says “OpenAI received the full 61-page report. METR and Redwood agreed to publish only a short summary, and OpenAI could redact it”
The source says “we agreed upfront with OpenAI that we would only publish a relatively brief and high-level public post”
Source cited by the article: metr.org/blog/2026-10-22-red-teaming-openai-agent-monitoring
Couldn’t checkQuote
An OpenAI spokesperson said OpenAI is using what it learned to strengthen its safeguards and shape future collaborations with independent evaluators.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “We’re using what we learned to strengthen our safeguards and shape future collaborations with independent evaluators”
The source says No passage could be compared.
Source cited by the article: cnbc.com/2026/10/22/metr-redwood-research-open-ai-test.html
Matches sourceCausal conclusion
A day after the report, OpenAI said it would cut live internet access from all frontier model training, citing the current state of monitoring systems among its reasons.
The cited source says the same thing.
The article says “A day after the report, OpenAI said it would cut live internet access from all frontier model training”
The source says “Based on the current state of alignment research, monitoring systems, and the security of internet infrastructure,” OpenAI wrote, it believes frontier training should run offline”
Source cited by the article: thenextweb.com/news/openai-frontier-training-no-live-internet
Also covered by
No other coverage was clustered with this story.
Elsewhere
Elsewhere: HN —
Recorded for context only, checked 26 Oct 2026, 07:27 UTC. It played no part in the ranking.