Skip to content

v1.3.40.2 -- Raw strip by cursor (hotfix)

Second hotfix on v1.3.40.0. argosVersion 1.3.40.1 -> 1.3.40.2.

What happened on the v1.3.40.1 deploy

The first purge (+120 s) started emptying raw on the ~400k access rows older than 24 h. Each batch ran ... WHERE source = ? AND timestamp >= watermark AND timestamp < cutoff AND raw <> '' ORDER BY timestamp LIMIT 5000; the watermark only moved at the end of the run, so batch k scanned the k-1 batches already emptied before finding its rows: quadratic row visits of 1.4 KB rows, on the single connection, with a 100 ms pause between batches. Measured on prod while it ran (10:47-10:56 UTC, /api/hosts sampled every 100 ms): p50 2 ms, p90 13.2 s, p99 51.2 s, max 59.4 s; 330,000 rows emptied (2026-09-19 to 2026-09-24 08:40) in 10 minutes; WAL 2.1 MB -> 16.3 MB, no checkpoint until the connection was free.

Mitigation applied at 10:56 UTC without waiting: PUT /api/settings/logs.retention.raw_hours = 720 (the cutoff moves past the oldest row, nothing to strip) and docker restart argos-prod-panel; /api/hosts back to 1-3 ms in 15 s. The watermark was not persisted (v1.3.40.1 wrote it at the end), so the 330k already-empty rows are re-visited once by the new cursor (a covering index walk plus no-op updates skipped by raw <> '').

What ships

  • db.stripRawByCursor: one covering SELECT id, timestamp per batch of RawStripBatch (1,000) ids ordered by (timestamp, id) from the cursor, one UPDATE ... WHERE raw <> '' AND id IN (...), the watermark handed to PurgePolicy.OnRawProgress after every batch; the retention loop persists it there, so a restart resumes at the last completed batch. A cancelled context returns the progress made.
  • logs.retention.raw_hours restored to 24 on prod as part of this deploy (the 720 was the mitigation, not a decision).

Strike 13 and the dense demo seed

The gate that should have caught this (/api/hosts p99 during a purge) ran without prod density: 64 rows on the demo, nothing to strip on prod. Strike 13 of the pattern, "EFFECT gate without prod density", distinct from strike 12 (invented schema). The rule in CLAUDE.md: no release with a new long write deploys before running it against the dense demo seed (500k log_entries, 1.4 KB raw, prod's per-source and per-day distribution), which scripts/demo does not generate yet: known issue with priority in 40.x before the 40.1 rollup backfill.

Gate

After deploy on prod: the first purge strips the ~70k rows that remain (2026-09-24 08:40 to now-24 h) plus a no-op pass over the 330k already empty; /api/hosts sampled every 100 ms through the whole run must stay p99 <= 50 ms; the purge log line reports raw_stripped and the watermark; WAL size during and after. Then logs-pipeline.sh (last purge done, filtered families <= 100, p99 during a purge) and the three regression smokes.

After on prod: pending deploy.