We benchmark LLMs at PostHog on practical tasks with real data

PostHog describes how it benchmarks its LLM agents on real tasks, including its ReviewHog code review agent and Scouts, its scheduled research agent. It covers three evaluation approaches of rising complexity: checking against bugs from a single PR, a 33-incident golden set built from real past incidents, and tailored rubrics…