Skip to content

v1.3.40.3 -- Strip batch 200, cap as a safety net (tagged release of the .0 line)

argosVersion 1.3.40.2 -> 1.3.40.3. This is the tag for the v1.3.40.0 line (ingest filter, retention per source, raw window); v1.3.40.0, .1 and .2 carry no tag. The incident that took three patches is recorded here.

The incident, 2026-09-26 (UTC)

When What Panel impact
10:31 v1.3.40.0 deployed none; ingest filter live at once
10:33 first purge: raw strip: NOT NULL constraint failed: log_entries.raw (the strip wrote NULL into a NOT NULL DEFAULT '' column; its test table had been invented, strike 12) raw strip not running; age and cap purges of that run done; every later run would have failed
10:45 v1.3.40.1 deployed (writes ''; dbtest with the real migrations) healthy in 10 s
10:47-10:56 first real strip over ~400k rows: each batch re-selected the not-yet-empty rows from the run's start watermark (persisted only at the end), quadratic re-scan of 1.4 KB rows on the single connection /api/hosts p50 2 ms, p90 13.2 s, p99 51.2 s, max 59.4 s for about 10 minutes; 330,000 rows emptied; WAL 2.1 -> 16.3 MB
10:56 mitigation without waiting: logs.retention.raw_hours = 720 (cutoff past the oldest row) + docker restart argos-prod-panel back to 1-3 ms in 15 s; watermark not persisted (v1.3.40.1 wrote it at the end)
11:08 v1.3.40.2 deployed: (timestamp, id) cursor, 1,000-row batches, watermark per batch (strike 13: the p99 gate had run without prod density) healthy in 10 s
11:08:58 kill-switch false positive: two boot-noise samples (397 / 307 ms) before the strip started, raw_hours auto-set to 720 none
11:17:43-11:19:32 forced purge with the watcher armed at the POST: 80,653 rows emptied in 109 s, 400 batches; watermark persisted (2026-09-25T11:17:43Z); 0 access rows older than 25 h with raw; WAL 4.0-5.8 MB /api/hosts p50 178 ms, p90 239 ms, p99 465 ms, max 689 ms; kill-switch fired on the first window (241 ms) and set raw_hours = 720; the in-flight run does not reload the setting and finished
11:2x regression smokes PASS (dashboard-latency strict, waf-sources-agree 1 %, panel-boot); logs-pipeline.sh p99 1,303 ms during a purge without strip the COUNT(*) of a table sitting on its 500,000 cap (cap_counted=true), the same 1.2-1.3 s measured before v1.3.40.x; logs.max_entries = 1,000,000 applied on prod

Total panel degradation: about 10 minutes at 13-59 s per request (10:47-10:56) and 109 s at 180-690 ms (11:17-11:19).

What ships

  • db.RawStripBatch 1,000 -> 200: about 40 ms held per batch, then the 100 ms pause; 48k rows (6 h of access traffic) in about 35 s of light contention.
  • logs.DefaultMaxEntries 500,000 -> 1,000,000; the same value set on prod. The cap is a safety net above what retention keeps, so the purge's MAX(id)-MIN(id)+1 bound stays under it and no COUNT(*) runs.
  • logs-pipeline.sh: the purge-path p99 and the strip are reported as separate checks naming which path ran.

Gate and the item-4 input

After deploy: raw_hours = 24, kill-switch watcher armed at the POST /api/logs/purge (not before, to skip boot noise), 50 ms p99 per 30 s window. Whatever the result, the hold per 200-row batch is the input to item 4 of the v1.3.40 plan: if the gate still fails with 200, the batch is not lowered further and the read pool moves from 40.3 to 40.1 with its pre-flight on the dense demo seed.

After on prod (deploy 11:29:32Z, healthy in 10 s)

  • raw_hours back to 24, max_entries 1,000,000 (cap_bound 500,334, cap_counted=false: no COUNT(*) any more). Forced purge with the watcher armed at the POST: 666 rows emptied in 1.86 s (4 batches), watermark 2026-09-25T11:34:09Z persisted, WAL 4.0 MB; the kill-switch did not fire (the purge ended before the first 30 s window closed). /api/hosts during it: 4 samples, max 1,137 ms, one of them on top of a pinned refresh.
  • The 50 ms gate fails for a cause that is not the strip. With no purge running, 120 s at 20 ms cadence: p99 4.8 ms, max 542 ms, 15 of 3,667 samples over 50 ms, every one within 1 s of the pinned warm-set refresh (every 24 s, traffic 24 h about 0.8 s on the single connection). Any purge longer than a few seconds overlaps one. Item 4 of the plan takes this as its input: the batch stays at 200 and the read pool moves from 40.3 to 40.1 (planning doc, section 4.1).
  • logs-pipeline.sh: families stored since start 0 / 0, 1,088 rows excluded today, strip PENDING (panel younger than 25 h; 0 rows older than 25 h carry raw), purge-path p99 FAIL as above (214 rows stripped in 0.55 s, p99 204 ms, reported on the strip path). dashboard-latency.sh strict PASS, waf-sources-agree.sh 1 % PASS, panel-boot.sh PASS. Access rows now 478,811 with 95 MB of raw (was 661 MB); the DB file stays 1.72 GB until the monthly VACUUM reclaims the pages.
  • 48 h reading (rows/day, DB size) on 2026-09-28, in the CHANGELOG.