Skip to content

v1.3.40.4 -- Strip cursor walks from the cursor (strike 14)

argosVersion 1.3.40.3 -> 1.3.40.4. One query changes. It corrects the root cause of strike 13, which v1.3.40.2 and v1.3.40.3 had treated by lowering the batch size.

What was wrong

stripRawByCursor (v1.3.40.2) selected the next batch with

WHERE source = ? AND timestamp < ?
  AND (timestamp > ? OR (timestamp = ? AND id > ?))
ORDER BY timestamp ASC, id ASC LIMIT 200

With the lower bound inside an OR, the SQLite planner in the panel (modernc, SQLite 3.46) used only timestamp < cutoff on idx_log_entries_source_ts and walked the index from the oldest row until 200 rows passed the filter. Every batch re-walked the rows the previous batches had already emptied: 100 to 150 ms of CPU per batch at 400k rows, growing as the cursor advanced, on the single connection. That is the "40 ms held per batch" of v1.3.40.3 on prod (p99 241 ms at 1,000 rows) and the 392 s strip on the dense demo.

The probes that had looked fine ran in the sqlite3 CLI (3.45), which picks both bounds for the same text. The plan the panel runs is the plan the driver produces.

What ships

  • db.StripCursorSQL: both bounds as plain range terms, timestamp >= cur AND timestamp < cutoff AND NOT (timestamp = cur AND id <= id). Plan: source=? AND timestamp>? AND timestamp<?; the SELECT takes 0.4 ms instead of 100+ ms.
  • TestStripCursorPlan runs EXPLAIN QUERY PLAN through the driver on the real schema (dbtest) and fails when the range term lacks either bound. The v1.3.40.2 OR form stays in the test as the negative case, so the assertion is known to discriminate.
  • scripts/smoke/strip-cursor.sh: strips about 6 h of access rows on the live panel, gates the panel CPU during the strip (max <= 10 %) and that rows were stripped; reports /api/hosts p50 / p99 / max.
  • Strike 14 recorded (pattern file, CLAUDE.md): a planner probe in the sqlite3 CLI is not a probe of the panel's plan.

Measured on the dense demo (500k rows, 430k stripped)

v1.3.40.3 v1.3.40.4
strip duration 392 s 242 s
panel CPU during the strip 50-60 % 2-5 %
cursor SELECT per batch 100-150 ms 0.4 ms
/api/hosts p99 (single connection) 433 ms 626 ms in 3 windows, 9-17 ms in the rest

The /api/hosts tail on the single connection is the checkpoint fsync under IO pressure and the session lookup on the writer handle, both addressed by the read pool (v1.3.41.0); this release does not claim the 50 ms gate.

Gate on prod

Deployed 2026-09-26 14:07 UTC with make deploy-prod (no rc build: one query with a plan test, BuildKit cache cold); healthy in 10 s. The boot purge at +2 min stripped 6,690 rows (removed 6,783).

strip-cursor.sh, raw_hours 24 -> 18 for the run and restored:

v1.3.40.3 on prod (11:17 UTC) v1.3.40.4 on prod (14:10 UTC)
rows stripped 80,653 in 109 s (740 rows/s) 18,003 in 9.5 s (1,900 rows/s)
panel CPU during the strip not sampled (55 % on the dense demo) median 4.6 %, max 4.8 % (gate <= 10 %) PASS
/api/hosts during the strip p50 178 ms, p99 465 ms, max 689 ms p50 1.5 ms, p99 5.8 ms, max 32.6 ms (reported, not gated)

The /api/hosts figure is on the single connection: with 10 ms of writer time per 200-row batch and no IO pressure at that hour the sampler slipped between batches. The read pool (v1.3.41.0) is what makes that number hold under a checkpoint fsync; this release does not claim it. Regression: dashboard-latency.sh strict PASS (health cold 0.9 ms cache hit, overview 186 ms), panel-boot.sh PASS (boot-to-listen 97 ms, first purge 127 s after listen).

Known issues carried

Unchanged from v1.3.40.3: Until shown as "expired" on imported decisions; blocked_requests_24h mixing; dashboard/cache_test timing flake; the 48 h DB-size reading.