v1.3.42.3.1¶
The tag that ships v1.3.42.3 (deployed on prod on 2026-09-29 19:26 UTC, verified, never tagged) plus one fix found on the prod 7d capture during that verification. Read v1.3.42.3.md for the readers; this note covers the fix only.
What ships¶
Hist.Percentile (backend/internal/db/rollup_read.go) reports the 5,000 ms edge when the nearest rank falls in the open bucket above 5,000 ms. Until v1.3.42.3 it returned the merged maximum of the window, which on prod is the lifetime of a long-lived connection: an /api/websocket row of 457,244,622 ms (5.3 days), /socket.io/ rows of 80 min, camera-proxy streams of 35 s. Ten of the 169 hours in the 7d window (2026-09-25 and 26) had more than 1 % of their requests above 5 s, so ten p99 points sat between 373,940 and 457,244,622 ms and the card's axis scaled to 60,000,000. The rollup's exact per-group dur_p99_ms carries the same maximum for a websocket-only group, so it was not a way out.
The dashboard card tooltip shows "5,000 ms or more" for that value, the logs stats card prints "5,000 or more" for a histogram p95 at the edge, both histogram hints explain the open bucket, and api.md says the maximum is never reported.
Tests: the existing merge test expects 5,000 (not 9,000) for the live row above 5 s and checks the maximum is still tracked; TestHistOpenBucketEdge pins p95 in a closed bucket next to p99 in the open one, every rank in the open bucket, and a rollup-only row with the prod maximum of 457,244,622 ms, all reporting 5,000.
Verification¶
Before the deploy, the ten prod hours recomputed with the new function through the Go driver (modernc, the real schema) over a .backup copy of the prod DB (deleted afterwards): p99 5,000 in all ten, p95 5,000, p50 2,500; the merged maxima the old code reported (312,894,063 / 1,198,743 / 5,970,760 / 1,323,614 / 1,293,538 / 1,679,103 / 457,244,622 / 557,083 / 373,940 / 853,986 ms) are still tracked and never leave the histogram. go test ./..., go vet, gofmt, npm run build, mkdocs build --strict, ASCII grep and check-no-personal-data.sh clean.
Results (deployed 2026-09-29 20:21 UTC)¶
make deploy-prod from main (d338860) at 20:19:38 UTC, recreate at 20:21:11, healthy 10 s later, no rollback. Boot log: read pool enabled, boot purge at 20:23:12 (2,326 removed), boot backfill of the cut hour (79 ms), vacuum schedule with last run 2026-09-01 and next 2026-10-01 04:00 UTC (the record written by the 42.3 boot; no catch-up run, as designed).
EFFECT on GET /api/dashboard/traffic?range=7d (cold, rollup path, 169 points): p99 maximum 5,000, zero points above 5,000 (ten on 42.3), 132 points at the 5,000 edge, p95 maximum 5,000, p50 maximum 2,500.
| smoke | result |
|---|---|
rollup-readers.sh | PASS: 7 d stats equal the rows (420,968 requests) in 202 ms; timeseries equal; 1h on rows; 7d and 30d on the rollup, histogram, no bridge fields |
dashboard-latency.sh (MAX_SECONDS=1) | PASS: health 0.8 ms cold, security overview 69 ms cold |
| captures | /tmp/argos-captures-pending/prod/dashboard-after-{24h,7d}.png retaken at 20:23 UTC (load 3.1 -> 3.2); the 7d response-time card reads on a 0 to 5,000 axis again |
Known issues carried¶
New, and the first item of 43.x: long-lived connections (websocket, socket.io, camera streams) carry their lifetime as duration_ms and weigh on every duration statistic, not only the open bucket: averages, the exact p95 and p99 of the 1h and 6h ranges, dur_sum_ms and dur_max_ms in the rollup. After this fix 132 of the 169 weekly p99 points on prod sit at the 5,000 edge: the weekly p99 today measures websockets, not latency (operator, 2026-09-29). Candidate: exclude them by path or above a duration ceiling at fill time, with its own PHASE 0 before any other 43.x work. Everything else as in v1.3.42.3.