Field notes · 11 March 2026
Reading lag before it becomes an outage
Lag alone rarely tells the full story. Here is how we separate healthy catch-up from the early shape of a consumer stall.
When a streaming consumer falls behind, teams often stare at a single lag chart and wait for a threshold page. That habit misses the quieter signals that appear hours earlier.
Healthy catch-up looks different from a stall. Catch-up shows a rising then falling lag curve after a known producer burst, with stable error rates and steady consumer CPU. A stall shows lag that climbs while processing rate flatlines, often accompanied by retry growth or partition skew.
We ask owners to watch three companions beside lag: messages processed per second, retry count, and the age of the oldest uncommitted offset. When lag rises but processed rate stays flat, the issue is almost never “need more partitions yesterday”—it is a blocked handler, a poison record, or a downstream dependency that stopped answering.
In Busan engagements we have seen on-call teams recover faster when they treat lag as a symptom narrative rather than a single number. Write down what changed in the hour before the climb: a deploy, a new producer, a schema change, or a batch job. That note often shortens the next investigation more than another dashboard panel.
If you only add one habit this month, pair every lag alert with a processing-rate check before paging the wider group. You will reduce false urgency and catch genuine stalls earlier.