v1.3.40.4 -- Strip cursor walks from the cursor (strike 14)¶
argosVersion 1.3.40.3 -> 1.3.40.4. One query changes. It corrects the root cause of strike 13, which v1.3.40.2 and v1.3.40.3 had treated by lowering the batch size.
What was wrong¶
stripRawByCursor (v1.3.40.2) selected the next batch with
WHERE source = ? AND timestamp < ?
AND (timestamp > ? OR (timestamp = ? AND id > ?))
ORDER BY timestamp ASC, id ASC LIMIT 200
With the lower bound inside an OR, the SQLite planner in the panel (modernc, SQLite 3.46) used only timestamp < cutoff on idx_log_entries_source_ts and walked the index from the oldest row until 200 rows passed the filter. Every batch re-walked the rows the previous batches had already emptied: 100 to 150 ms of CPU per batch at 400k rows, growing as the cursor advanced, on the single connection. That is the "40 ms held per batch" of v1.3.40.3 on prod (p99 241 ms at 1,000 rows) and the 392 s strip on the dense demo.
The probes that had looked fine ran in the sqlite3 CLI (3.45), which picks both bounds for the same text. The plan the panel runs is the plan the driver produces.
What ships¶
db.StripCursorSQL: both bounds as plain range terms,timestamp >= cur AND timestamp < cutoff AND NOT (timestamp = cur AND id <= id). Plan:source=? AND timestamp>? AND timestamp<?; the SELECT takes 0.4 ms instead of 100+ ms.TestStripCursorPlanrunsEXPLAIN QUERY PLANthrough the driver on the real schema (dbtest) and fails when the range term lacks either bound. The v1.3.40.2 OR form stays in the test as the negative case, so the assertion is known to discriminate.scripts/smoke/strip-cursor.sh: strips about 6 h of access rows on the live panel, gates the panel CPU during the strip (max <= 10 %) and that rows were stripped; reports/api/hostsp50 / p99 / max.- Strike 14 recorded (pattern file, CLAUDE.md): a planner probe in the
sqlite3CLI is not a probe of the panel's plan.
Measured on the dense demo (500k rows, 430k stripped)¶
| v1.3.40.3 | v1.3.40.4 | |
|---|---|---|
| strip duration | 392 s | 242 s |
| panel CPU during the strip | 50-60 % | 2-5 % |
| cursor SELECT per batch | 100-150 ms | 0.4 ms |
/api/hosts p99 (single connection) | 433 ms | 626 ms in 3 windows, 9-17 ms in the rest |
The /api/hosts tail on the single connection is the checkpoint fsync under IO pressure and the session lookup on the writer handle, both addressed by the read pool (v1.3.41.0); this release does not claim the 50 ms gate.
Gate on prod¶
Deployed 2026-09-26 14:07 UTC with make deploy-prod (no rc build: one query with a plan test, BuildKit cache cold); healthy in 10 s. The boot purge at +2 min stripped 6,690 rows (removed 6,783).
strip-cursor.sh, raw_hours 24 -> 18 for the run and restored:
| v1.3.40.3 on prod (11:17 UTC) | v1.3.40.4 on prod (14:10 UTC) | |
|---|---|---|
| rows stripped | 80,653 in 109 s (740 rows/s) | 18,003 in 9.5 s (1,900 rows/s) |
| panel CPU during the strip | not sampled (55 % on the dense demo) | median 4.6 %, max 4.8 % (gate <= 10 %) PASS |
/api/hosts during the strip | p50 178 ms, p99 465 ms, max 689 ms | p50 1.5 ms, p99 5.8 ms, max 32.6 ms (reported, not gated) |
The /api/hosts figure is on the single connection: with 10 ms of writer time per 200-row batch and no IO pressure at that hour the sampler slipped between batches. The read pool (v1.3.41.0) is what makes that number hold under a checkpoint fsync; this release does not claim it. Regression: dashboard-latency.sh strict PASS (health cold 0.9 ms cache hit, overview 186 ms), panel-boot.sh PASS (boot-to-listen 97 ms, first purge 127 s after listen).
Known issues carried¶
Unchanged from v1.3.40.3: Until shown as "expired" on imported decisions; blocked_requests_24h mixing; dashboard/cache_test timing flake; the 48 h DB-size reading.