How ranking works
Front pages often reward who submitted a story more than what it says. Rubric tests whether scoring every candidate blind, on its full text and in the open, surfaces good stories that would otherwise be missed.
Blind judging
Every story published the previous UTC day is a candidate, and every candidate is scored on its full text; no one judges on titles. The judges see the extracted text only. The author, byline, site name, domain and vote counts are removed, and links keep their text but lose their domains.
Blinding reduces bias; it does not remove it. Writing style can still give an author away, and AI judges are known to favour familiar text and text written by AI. A weekly check scores human-written articles against AI rewrites of the same content, and a consistent preference for the rewrites is logged as a bias finding.
Articles longer than 12,000 tokens are cut to that length before judging.
The panel
Four AI agents do the ranking. Three judges, from different vendors, answer the same typed questions about each candidate; a fourth checks the claims of the day’s top stories against the sources they cite, with Judge 1 giving a second verdict. Summaries are written by AI too. Model versions are pinned, never a moving “latest”, and each edition’s JSON lists the versions it used.
| Judge | Scores |
|---|---|
| Judge 1 | Every candidate (stage A) |
| Judge 2 | Every candidate (stage A) |
| Judge 3 | The top 200 after stage A, and each category’s 10 best (stage B) |
Duplicates, clusters and past editions are found with a local embedding model.
The rubric
Each judge answers every question through a fixed schema: for a yes-or-no question, the probability of “yes”, 0–1; for a score, a score from 0 to 1; for a choice, a probability for each option; for a question with several answers, a separate probability for each option, judged on its own. Positive weights raise the score; negative weights lower it.
| Question | Type | Weight |
|---|---|---|
| On topic Is it of interest to people who build technology products or companies (software, AI, infrastructure, security, data, hardware, product, design, growth, startups or fintech)? | Yes or no | Gate: below 0.50, dropped |
| Significance How much does this change what people in tech know or can do?From 0 to 1: Trivial, or of personal interest only → A minor update or a small side project → Notable for a niche → Important for a large part of the field → A landmark. | Score | +0.24 |
| Reader interest Would a well-informed senior engineer or tech founder want to read this today, and likely pass it on? | Yes or no | +0.26 |
| Audience breadth How many people who build technology does it concern?From 0 to 1: One company, market or tool → One niche community → One large community → Most people in tech. | Score | +0.13 |
| Practical relevance How useful is it to someone who builds software or tech products?From 0 to 1: Not useful to people who build software → Background interest only → Worth knowing about → Likely to change how some builders work → Directly actionable for many builders. | Score | +0.22 |
| Substantiated How well is it backed? Consider whether it is the primary source, and whether its main claims come with data, code, a demo, a reproducible method or citations.From 0 to 1: Unsupported assertions → Some support → Mostly supported → A primary source with solid evidence. | Score | +0.05 |
| Technical depth How technically substantive is it?From 0 to 1: No technical content → Mentions technology but explains nothing → Explains what was done at a high level → Explains how it works, with specifics → Deep detail a practitioner could learn from or reproduce. | Score | +0.05 |
| New information Does it contain information absent from the stories covered in the last 7 days (list provided)? | Yes or no | +0.05 |
| Promotional Is it mainly marketing with little substance (a press release, ad or sponsored post with no real information)? A launch post that carries real news is not promotional. | Yes or no | −0.19 |
| Overclaim Do the headline or conclusions claim more than the body supports? | Yes or no | −0.11 |
| Content type What kind of piece is it?Options: Original research, Engineering write-up, Release, Analysis or opinion, Tutorial or how-to, News report about events or other people's work, Other. | Choice | Label only |
| Project stage If it presents a software project, what stage is the project at?Options: A new personal or side project, just shared by its maker, An early-stage project with a few users or contributors, An established project with real users, A product, model or company release, Not about a specific software project. | Choice | Label only |
| Categories Which sections of a tech news edition does it belong to? Judge each section on its own; a story can belong to several or to none.One answer per category (15, listed under Categories). A story is filed under every category whose mean is at least 0.60. | Several | Label only |
A candidate whose mean answer to “Is it of interest to people who build technology products or companies (software, AI, infrastructure, security, data, hardware, product, design, growth, startups or fintech)?” is below 0.50 is dropped before ranking. The content type, project stage and categories questions only label a story; they never change its score. They do decide which categories a story is filed under, as described below.
The score
score = Σ (weight × mean of the judges’ answers)
− 0.10 or 0.25 for older work (freshness)
− 0.10 × agreed claim-check flags
The mean is taken over the judges that scored the story. With the current weights the weighted sum runs from −0.30 to +1.00, before the penalties. Every judge’s answers are stored with the score, along with the spread for each question: the highest answer minus the lowest. Each story page shows the full table, with each penalty on its own line.
Freshness
Freshness is applied in code, after the judges have scored. Each story is dated by the work’s own first date: the first version of a paper, the published date in the article’s metadata, or a release date. When that date is more than 7 days before the start of the edition’s window, 0.10 is taken off the score; more than 30 days, 0.25 instead. A story whose first date is unknown loses nothing. Story pages show the first date when it differs from the publication date.
One list a day
Each edition is one list in score order: every story whose score reaches 0.55, up to 300. There is no quota per site or per subject; the score alone sets the order, and a story’s rank is its place in that list.
An edition page shows the first 20 stories, and each “Load more” adds 20 more. Without JavaScript the page shows the whole list. A new edition is published every day at 06:00 UTC, and every past one stays in the archive, by date.
How far each story is checked
Checking costs time and money, so it goes deepest at the top of the list. Each story is marked with how far it was checked:
- Claim-checked
- Scored by all three judges, then its checkable claims compared with the sources it cites (see the claim check). Stage B’s top 25 are claim-checked, which covers the 20 shown first. Flags can cost a story enough points to move it further down; it stays marked claim-checked.
- 3 judges
- Stories down to about No. 200. The top 200 by the mean of the first two judges go to the third, as do each category’s 10 best; the final order uses all three judges, so the boundary is approximate.
- 2 judges
- The rest: scored by the first two judges. The third judge’s column is empty, and the mean is over two.
The depth describes the process, not the story: scores at every depth are on the same scale, and a story checked by two judges is not marked down for it. New projects skip the claim check.
Topics and formats
Each story has topics, and one format. For every topic on its own, the judges give the probability that the story is about it; a story is filed under each topic where the mean is at least 0.60. It can have several, or none: those are filed under Other. Its format is the judges’ answer to “what kind of piece is it?”, or New projects when the new-project check caught it (below). Neither changes a score or a rank.
On an edition page each topic and each format is a checkbox, with how many stories it has. A story shows while any of its topics is ticked and its format is ticked, so you can follow AI and leave out the papers. Hidden stories keep their place: the order stays and the ranks are not renumbered. New projects is unticked until you tick it. Your choice is saved in your browser and applies to every edition. Changing a filter is counted, anonymously, with nothing that identifies you.
Topics
- Programming and developer tools: languages, frameworks, dev tools, engineering practice
- AI and machine learning: models, applied AI, AI tooling
- Security: vulnerabilities, incidents, privacy engineering
- Infrastructure and DevOps: cloud, SRE, platforms, CI/CD, Kubernetes
- IT and networking: enterprise IT, sysadmin, networks, end-user computing
- Data: databases, data engineering, analytics
- Hardware and devices: chips, embedded, devices
- Product management: discovery, roadmaps, pricing, product strategy
- Design and UX: interface and product design, user research
- Growth and marketing: acquisition, positioning, content
- Startups and founders: company building, fundraising, tech business
- Fintech: payments, banking technology, crypto infrastructure
- Policy and regulation: privacy law, AI rules, antitrust
- Other: stories that fit none of the topics
Formats
- Paper: a paper, study or new measurement
- Write-up: how a team built, ran or fixed something
- Release: a product, model, library or version announcement
- Analysis
- Tutorial
- News
- Other
- New project: side projects shared by their makers, and young GitHub repos with few stars
New projects
A side project shared by its maker the day it goes up is not news yet. Its format is New projects, a format hidden by default, and it skips the claim check; it keeps its topics, its score and its rank. Two checks decide it, after scoring:
- The judges’ answer to the project-stage question: a mean probability of at least 0.50 for “A new personal or side project, just shared by its maker”.
- For a GitHub repository: created less than 30 days ago, with fewer than 25 stars.
The GitHub check uses a popularity signal, which the judges never see, so it only changes how a story is filed; it never changes a score.
Thresholds
- > 0.40
- A spread above this on any weighted question marks the story contested: the judges disagree.
- ≥ 0.55
- The score a story needs to make the day’s list.
- 300
- The most stories in one edition.
- 200
- Top candidates, by the mean of the first two judges, that go to stage B and get the third judge.
- 25
- Top stories after stage B that get a claim check.
- −0.10
- Taken off the score for each claim-check flag that both checkers agree on.
- ≥ 0.80
- Judge 1’s confidence, as the second checker, needed before a negative flag is published.
- 5
- Contested stories within this many places of a cut-off are reviewed by a person during the private phase.
The claim check
The top 25 stories after stage B get a check of their checkable facts, and of the gap between what they claim and what their own cited sources support. Only the sources an article links are read; no web search is made for others. Numbers are compared in code, not by a model. Counting how many outlets repeat a claim is not evidence, so it is not used.
Labels
| Label | Meaning |
|---|---|
| Matches source | The cited source says the same thing. |
| Differs from source | The cited source says something else; both quotes are shown. |
| Not in source | The cited source does not contain it. |
| No source given | The article gives no link or citation for it. |
| Couldn’t check | The source was unreachable, paywalled or too long, or the two checkers disagreed. |
Claim drift
A drift tag marks a claim stated more strongly than its source supports:
- A benchmark gain is presented as a general capability.
- A demo is presented as a shipped product.
- A preprint is presented as an established result.
- An “up to” figure is presented as the typical case.
- One case is presented as a trend.
- A correlation is presented as causation.
When a flag is published
- “Differs from source”, “Not in source” and drift tags are published only when both checkers agree, andJudge 1’s confidence is at least 0.80. Otherwise the claim shows “Couldn’t check”.
- Every flag shows the article’s words, the source’s words as short quotes, and the source link. There is never a bare verdict.
- Each agreed flag lowers the story’s score by 0.10.
Summaries
Each story gets one summary, written from every article in its cluster rather than paraphrased from one. The first 20 stories a reader sees get a full summary; the rest get a shorter one.