v1.3.41.1 -- CSV export by keyset, streamed, paced¶

argosVersion 1.3.41.0 -> 1.3.41.1. The v1.3.41.0 gate 3 (three back-to-back 24 h CSV exports with /api/hosts p99 <= 50 ms) failed on prod for a CPU reason: the export held the panel at 99-102 % of its 1-CPU cgroup and every request that landed in that window stalled. This release changes how the export walks and paces the match set; ARGOS_CPU_LIMIT stays at 1.

What ships¶

• db.ListLogEntriesAfter: the export walks the match set by a (timestamp, id) keyset instead of LIMIT/OFFSET. Each OFFSET page re-read every earlier row, so a 66k-row export visited about 1.1 M index entries; the keyset cursor uses both bounds as range terms (strike 14 rule) and TestListLogEntriesAfterPlan pins the plan. TestListLogEntriesAfterWalksEveryRowOnce covers rows that share a timestamp and the filter.
• ExportLogsCSV flushes the CSV writer and the HTTP response after every page (the client streams, the encoder holds one page) and pauses exportPagePause (20 ms) after each page of exportPageRows (500).
• The pause is the yield that works. runtime.Gosched() was tried first: with the keyset alone the export dropped from 5.5 s to 1.5 s and the kernel throttled the container in 14 of 16 scheduler periods (155 ms) during that export; gate 3 still failed (p99 88 ms with the pool on, 240 ms off). A sleep between pages keeps the duty cycle low: 0 of 36 periods throttled during a 3.5 s export at 1,000 rows per page, and the 500-row page halves the burst a request can land in.

Measured on the dense demo (500k rows, 24 h export of 66k access rows, no IO pressure)¶

/api/hosts sampled every 100 ms across three exports; panel CPU from docker stats.

Reading the table:

• The export's own share is gone: with the pool off, where every read still queues on the writer, p95 fell from 249 ms to 24 ms and p99 from 395 ms to 38 ms. The two 300 ms samples left in the off column sit together at one instant and are the pinned dashboard refresh holding the writer, which is the pool's job, not the export's.
• With the pool on, run a passes with p99 9.8 ms. Run b fails on a cluster of three samples (68-88 ms) inside its slow export 2 (6.4 s) while docker stats read 101 %: the pinned 24 h refresh (a 1-2 s query at full CPU on the dense demo, 0.8 s on prod) landed on top of the export and the sum crossed the 1-CPU quota. The export alone no longer throttles the container.
• Wall time: 5.3 s, the same as v1.3.41.0's, because the pause gives back what the keyset saved; the difference is that the encoder now uses about a third of the quota instead of all of it.

Gate on prod¶

Deployed 2026-09-26 15:58 UTC (make deploy-prod 67 s, healthy in 10 s, read pool enabled). After-smokes with the threshold unchanged at 50 ms:

What is left in gate 3, with the numbers that separate it from the export: three more exports right after, with every sample timed, gave p99 7.4 ms, max 23.5 ms, 0 of 180 over 50 ms, and 8 throttled scheduler periods in 15 s (the export alone stays under the quota). The 8 slow samples of the 9-export run (63-75 ms, 1.75 % of the samples) are the moments when the pinned 24 h refresh (about 0.8 s at full CPU on prod, every 24 s: about two ticks in the 46 s the nine exports took) landed on top of the export's third of the quota and the container was throttled (nr_throttled +50 during the run, in steps that match the ticks). Known issue, cause identified, closed in v1.3.42.0 when the dashboard reads the hourly rollup and the refresh takes milliseconds. Compared with v1.3.41.0 on prod: gate 3 p99 884 ms -> 63 ms, p95 unchanged at about 7 ms.

Known issues carried¶

The pinned 24 h refresh is the remaining CPU burst that can coincide with an export on a 1-CPU cgroup; it goes away when the dashboard reads the hourly rollup (v1.3.42, migration 034). Carried from v1.3.41.0: Until shown as "expired" on imported decisions; blocked_requests_24h mixing; dashboard/cache_test timing flake; the 48 h DB-size reading.