v1.3.42.0 -- Hourly rollup: migration 034, fill job, drift check¶

argosVersion 1.3.41.1 -> 1.3.42.0. First schema change since v1.3.33. This release only builds and verifies the rollup; no reader changes: the UI keeps reading raw rows. v1.3.42.1 moves the long ranges onto it once prod has 48 h of drift 0.

What ships¶

• Migration 034 (034_log_hourly.up.sql / .down.sql): tables log_hourly (per closed UTC hour, source, host and status class: requests, bytes out, duration sum / max, exact p50 / p95 / p99, an 8-bucket duration histogram, forbidden (403), rate_limited (429) and errors (level = error) counters) and log_hourly_paths (the 50 busiest paths per host per hour), with idx_log_hourly_source_hour and idx_log_hourly_paths_hour. Nothing on log_entries changes; the four load-bearing indexes referenced with INDEXED BY are untouched. The down migration drops both tables; TestRollbackLastMigration peels 034 first.
• Fill job (internal/logs/rollup.go, db.FillRollupHour): runs inside the retention goroutine, so it never overlaps the purge. The boot backfill follows the boot purge (2 min after start), the hourly fill runs at HH:02, the 6 h tick does purge, rollup retention and the drift check in that order. Each hour is one transaction (clear the hour, grouped pass over its rows through idx_log_entries_timestamp, exact percentiles, top paths) with a 100 ms pause between hours. Recovery: every run starts at MAX(hour) (re-filling the newest hour, so rows that landed after its first fill are counted) or at the oldest log_entries hour when the table is empty, and stops before the hour in progress; a lost tick is repaired by the next one or by the next boot. The log line rollup: fill done carries hours, groups, paths, total time and the slowest hour; hours over 3 s are logged one by one.
• Drift check: the 6 h tick compares the newest closed hour at least 3 h old with the raw rows (COUNT(*), SUM(size_bytes), per-class counts; tolerance 0) and persists the verdict (logs.rollup.drift, 0 or the numbers) and its time; a divergence logs rollup drift with both sides.
• Retention: logs.rollup_days (default 90; editable in Settings and PUT /api/settings/logs.rollup_days), purged by the 6 h tick.
• GET /api/logs/pipeline gains rollup (days, last_hour, last_run_at, drift, drift_checked_at); Settings shows it read-only next to the new days field.
• scripts/smoke/rollup-agree.sh: job lag <= 2 h, raw vs rollup for a closed hour at tolerance 0 (rows, bytes, per-class), ticker verdict.

Measured on the dense demo (PHASE 0 probe, same SQL)¶

168 h backfill inside the panel cgroup under dd IO pressure: 2 m 15 s (per hour p50 636 ms, p95 1.23 s, one IO-stall hour of 23.6 s), /api/hosts p99 7.2 ms, panel CPU median 11 %, 36 of 1,362 periods throttled, WAL peak 4.5 MB, 9,073 + 66,260 rows (17 MB for 7 days). Divergence on a closed hour: 3,031 rows and 57,931,839 bytes on both sides. Planning doc section 9 has the tables.

Final image on the demo (argos-prod-argos:1.3.42.0, 500k rows, no IO pressure): migration 034 applied at boot; boot purge at +2 min, then rollup: fill done for 169 hours in 39.0 s (slowest hour 347 ms), 9,073 groups and 66,260 path rows; /api/hosts during the purge and the backfill p50 1.4 ms, p99 5.7 ms, max 29.8 ms, 0 of 1,172 samples over 50 ms; rollup-agree.sh PASS: lag 0 h, hour 13:00 UTC 3,049 rows and 55,240,994 bytes on both sides, five status classes equal. The 6 h drift verdict is first written by the ticker on prod.

Deploy procedure (read before deploying)¶

1. POST /api/backups (VACUUM INTO) and note the filename; wait for it to appear in GET /api/backups.
2. make sync-prod-dry, then make deploy-prod; the 42.0 boot applies migration 034 (two tables, two indexes, sub-second), and 2 min later runs the boot purge followed by the backfill (1 to 3 min on prod behind the 100 ms pause; reads stay on the pool).
3. Rollback logic: none needed for readers (unchanged). If the panel is not healthy in 60 s: image rollback below.

Way back to v1.3.41.1¶

• Repin argos-prod-argos:1.3.41.1 in ~/argos-prod/docker-compose.override.yml and make deploy-prod. v1.3.41.1 ignores log_hourly and log_hourly_paths; the DB needs nothing.
• Optional cleanup after the repin (never before, the 42.0 boot would recreate them): docker compose exec argos /argos migrate rollback from ~/argos-prod, which runs 034_log_hourly.down.sql.
• Last resort: docker compose exec argos /argos restore --file /data/backups/<name>.tar.gz --yes and docker compose restart argos from ~/argos-prod (the flag is applied at the next boot before the DB opens).

Gate on prod¶

After the deploy: the boot log shows applied migration with version 034_log_hourly, then rollup: fill done with hours and total; rollup-agree.sh PASS (tolerance 0); read-pool.sh gates 0-3 (CAP_PURGE_ROWS=0) during or right after the backfill; dashboard-latency.sh strict, panel-boot.sh. Then 48 h of the 6 h drift check at 0 (the ticker logs rollup drift check ... drift=0; the operator reads the log) before v1.3.42.1 starts.

Results (deployed 2026-09-26 16:51 UTC)¶

Pre-deploy: disk 87 % (3.6 GB free) and 1.8 GB of backups (15 files); POST /api/backups created id 88 in about 5 s; disk 87 % after (3.5 GB free), no backup retention needed. make deploy-prod 90 s, panel healthy in 10 s; boot log in order: read pool enabled, applied migration 034_log_hourly, http listening; boot purge at +2 min (2,828 rows removed); then:

rollup: fill done reason=boot backfill hours=173 groups=3591 paths=17360
  total=55.775s slowest_hour=2026-09-25T03:00:00Z slowest=432ms
  from=2026-09-19T11:00:00Z to=2026-09-26T15:00:00Z

173 hours in 55.8 s on prod (about 320 ms per hour with the 100 ms pause), 3,591 groups and 17,360 path rows. /api/hosts sampled through the purge and the backfill: p50 1.5 ms, p99 7.7 ms, max 243 ms, 2 of 1,297 samples over 50 ms.

GET /api/logs/pipeline rollup: days 90, last_hour 2026-09-26T15:00:00Z, last_run_at 2026-09-26T16:54:25Z, drift not yet checked. Next: 48 h of rollup drift check ... drift=0 in the log (every 6 h) before v1.3.42.1.

Known issues carried¶

The v1.3.41.1 export gate tail (pinned refresh on top of the export on a 1-CPU cgroup) stays until v1.3.42.1 moves the refresh onto the rollup. Carried: Until shown as "expired" on imported decisions; blocked_requests_24h mixing; dashboard/cache_test timing flake; the 48 h DB-size reading.