When the Kubernetes API went silent: A three-layer networking mystery
Comments on “When the Kubernetes API went silent: A three-layer networking mystery”
Claim-checked. Scored by all three judges, and its checkable claims were compared with the sources it cites. How far stories are checked
Google Cloud's support team describes a January 2026 GKE incident in which pods intermittently timed out reaching the API server over Private Service Connect. It traced the outage to three interacting faults: packet reordering in Private Service Connect's packet-switching layer, Envoy's half-close keeping sockets in FIN_WAIT_2, and a Linux 6.1 bug that discarded the client's FIN. A switching-layer change then dropped the resulting challenge ACKs. Rolling back that change restored service; the kernel fix ships in GKE 1.33.5 and later. The claims come from Google alone, with no independent source.
Why it ranked
No. 15 of 46 with a score of 0.59. The panel rated it highest on technical depth (0.94) and new information (0.90), lowest on significance (0.63). Older work: a freshness penalty of 0.100. Placed No. 15, No. 29 by score alone: the placement terms moved other stories past it.
- AI judgeJudge 2claude-haiku-5-5
- Detailed, well-evidenced postmortem of a multi-layer Kubernetes/GKE networking failure with packet captures and a kernel fix; high technical depth, practical lessons for infra builders, mostly new to the week's coverage.
- AI judgeJudge 3gpt-5.6-luna
- A detailed, evidence-rich incident analysis connects PSC packet handling, NAT port reuse, Envoy half-closes, TCP state-machine behavior, and a Linux kernel fix. It is highly actionable for Kubernetes and networking practitioners, though based on anonymized internal case data.
Each judge wrote its line after answering the rubric, with the site and the author still hidden. Every answer behind this is in the table under Score.
Score
0.59total under rubric v2.3
- Significance0.63
- Reader interest0.85
- Audience breadth0.63
- Practical relevance0.68
- Substantiated0.77
- Technical depth0.94
- New information0.90
- Counts against: Promotional0.12
- Counts against: Overclaim0.19
Each judge’s answer per question (0–1), their mean and spread, and what the question adds to the score (weight × mean).
| Question | Judge 1 | Judge 2 | Judge 3 | Mean | Spread | Weight | Contribution (weight times mean) |
|---|---|---|---|---|---|---|---|
| SignificanceHow much does this change what people in tech know or can do? | 0.60 | 0.50 | 0.78 | 0.63 | 0.28 | +0.24 | +0.150 |
| Reader interestWould a well-informed senior engineer or tech founder want to read this today, and likely pass it on? | 0.78 | 0.85 | 0.93 | 0.85 | 0.15 | +0.26 | +0.222 |
| Audience breadthHow many people who build technology does it concern? | 0.57 | 0.60 | 0.72 | 0.63 | 0.15 | +0.13 | +0.082 |
| Practical relevanceHow useful is it to someone who builds software or tech products? | 0.65 | 0.60 | 0.78 | 0.68 | 0.18 | +0.22 | +0.149 |
| SubstantiatedHow well is it backed? Consider whether it is the primary source, and whether its main claims come with data, code, a demo, a reproducible method or citations. | 0.74 | 0.80 | 0.78 | 0.77 | 0.06 | +0.05 | +0.039 |
| Technical depthHow technically substantive is it? | 1.00 | 0.90 | 0.93 | 0.94 | 0.10 | +0.05 | +0.047 |
| New informationDoes it contain information absent from the stories covered in the last 7 days (list provided)? | 0.89 | 0.85 | 0.97 | 0.90 | 0.12 | +0.05 | +0.045 |
| PromotionalIs it mainly marketing with little substance (a press release, ad or sponsored post with no real information)? A launch post that carries real news is not promotional. | 0.04 | 0.15 | 0.18 | 0.12 | 0.14 | −0.19 | −0.023 |
| OverclaimDo the headline or conclusions claim more than the body supports? | 0.16 | 0.20 | 0.20 | 0.19 | 0.04 | −0.11 | −0.021 |
| Sum of contributions | +0.689 | ||||||
| Freshness penalty | −0.100 | ||||||
| Claim-check penalty | 0.000 | ||||||
| Total, as published | 0.589 | ||||||
Spread is the highest minus the lowest judge answer. marks a spread above 0.40: the judges disagree, and the story is marked contested. The freshness penalty is for older work: this one was first published 30 Oct 2026.
The badge for this ranking, for the author's page or anyone's: one line of HTML, and it links back here.
<a href="https://rubric.hrih.si/editions/2026-11-09/15/"><img src="https://rubric.hrih.si/badge/2026-11-09/15.svg" alt="Ranked #15 on Rubric, 9 Nov 2026" width="238" height="28"></a>Claim check
Claim check: 8 no source given. Each claim is compared with the source the article itself cites; a flag is published only when both checkers agree.
No source givenNumber
Around 800 developers were blocked from deploying to production for weeks, and GKE clusters across europe-west ground to a halt.
The article gives no link or citation for it.
The article says “around 800 developers had been blocked from deploying to production for weeks on end, and GKE clusters across europe-west had ground to a halt”
The source says No source to compare with.
No source givenNumber
The conntrack table on the control plane nodes was at roughly 50% capacity, out of a default maximum of 1.3M entries, ruling out exhaustion.
The article gives no link or citation for it.
The article says “The conntrack table was at roughly 50% capacity — of a default maximum of 1.3M entries”
The source says No source to compare with.
No source givenDate
The Andromeda counter "Proxy protocol drop: invalid ACK packet" had been flat for months and spiked sharply in February 2026, matching a USPS version rollout.
The article gives no link or citation for it.
The article says “A counter called "Proxy protocol drop: invalid ACK packet" had been flat for months — and then, in February 2026, it spiked sharply upward”
The source says No source to compare with.
No source givenCausal conclusion
The USPS change dropped challenge ACKs as invalid ACKs, which broke a flow GKE's TCP stack depended on to clear stale state; it did not introduce challenge ACKs.
The article gives no link or citation for it.
The article says “The USPS fix hadn't introduced challenge ACKs — they had always been there. It had introduced the dropping of them”
The source says No source to compare with.
No source givenCausal conclusion
tcp_fin_timeout was set to 60 seconds on the KCP VM but never started because Envoy's half-close via shutdown(SHUT_WR) kept the socket from being orphaned.
The article gives no link or citation for it.
The article says “As long as Envoy held the socket with SHUT_WR, the kernel did not consider it orphaned. tcp_fin_timeout never started”
The source says No source to compare with.
No source givenVersion
A bug in Linux kernel 6.1 made tcp_ack() return zero for old-but-acceptable ACKs, causing the FIN to be silently discarded.
The article gives no link or citation for it.
The article says “in kernel 6.1, there was a code path where tcp_ack() returned zero — indicating "this packet was not fully processed”
The source says No source to compare with.
No source givenVersion
The kernel bug was fixed in kernel 6.9.7, while GKE's KCP VMs ran COS builds based on 6.1 and 6.6.
The article gives no link or citation for it.
The article says “The bug had existed since kernel 6.1 and was already fixed in kernel 6.9.7 — but GKE's KCP VMs were running COS builds based on 6.1 and 6.6”
The source says No source to compare with.
No source givenVersion
The upstream patch "net: tcp: accept old ack during closing" by Eric Dumazet was merged into Linux 6.9.7, backported to 6.6.y and 6.1.y, and ships in GKE 1.33.5 and later.
The article gives no link or citation for it.
The article says “Upstream patch ("net: tcp: accept old ack during closing"), authored by Eric Dumazet. Merged into Linux 6.9.7. Backported to 6.6.y and 6.1.y LTS branches”
The source says No source to compare with.
Also covered by
No other coverage was clustered with this story.
Elsewhere
Elsewhere: HN 2 pts, not on front page
Recorded for context only, checked 9 Nov 2026, 05:46 UTC. It played no part in the ranking.