v1.3.38.3 -- Boot path, certs list, deterministic test, one TLS probe¶

Fourth release of the v1.3.38 series ("dashboard paints in under a second"; scope in docs/operations/ux-performance-review.md). Four items in the order the operator set. No schema change. argosVersion 1.3.38.2 -> 1.3.38.3.

1. Boot path¶

PHASE 0 came from the 1.3.38.2 container's own boot log rather than a restart of prod:

Two facts changed the plan: the panel container runs with cpus: 1.0, so its boot cannot produce host load above 1, and the 9.7 load spike of a deploy starts ~30 s before the container exists (7.3 at 20:06:32 for a 20:07:00 start), i.e. in the build's final phase and the recreate. The boot fix is therefore about the DB connection, not CPU:

• logs.BootPurgeDelay = 2 min: the first purge runs two minutes after StartRetention (listen follows it within a second), then every 6 h as before.
• db.PurgeOld deletes in batches of PurgeBatchSize (5,000) with PurgeBatchPause (100 ms) between them, for both the age DELETE and the cap DELETE, at boot and on every tick and for POST /api/logs/purge. The longest single statement is now one batch.

Expected after: boot-to-listen ~0.3 s; first purge logged ~2 min after listen. scripts/smoke/panel-boot.sh asserts both from the boot log without restarting anything.

The deploy-tail spike is measured per process during this release's deploy (top -b every 5 s); see "Deploy" below.

2. /api/certs¶

PHASE 0 on a .backup copy of prod (deleted after): every one of the 21,681 caddy_error rows had host_id NULL and host_domain empty, because fillError only read an identifier field that this Caddy's error lines never carry (93 % are health_checker.active; the rest have the domain under request.host). Options measured:

Shipped B + C:

• ListCerts returns rows without last_renewal_event.
• GET /api/certs/{id}/last-event -> {host_id, domain, event | null, matched_by: "host_domain" | "message" | "", window: "720h"}. Lookup 1: source='caddy_error' AND timestamp >= now-30d AND host_domain = ? (A2, index range on (source, timestamp)). Lookup 2 (only when 1 is empty): same bounded slice with a substring match on message, for rows that predate C.
• fillError: identifier -> identifiers[0] -> request.host (port stripped, lowercased). Old rows keep an empty host_domain and leave with retention.
• Certificates table: the "Last event" column stays; each row has a show button -> loading... -> the event link, none (30d), or failed, retry. No page redesign.

3. Deterministic mutex test¶

fakeLAPI gained gate / entered channels: with gate set, every AddRangeDecisions sends on entered and then blocks until gate is closed (or the context ends). TestSubmit_serialisesViaMutex submits two jobs, waits on entered, asserts that at that instant exactly one job is running and the other pending (Submit does not promise which of two back-to-back jobs wins the mutex; it promises one at a time), closes the gate, and waits for both to complete. The 250 ms addDelay and the 400 ms sampling window are gone from this test (addDelay stays for other tests). jobs.go untouched.

4. One TLS probe¶

internal/certprobe: Probe (the SNI dial, formerly api.probeCert) and Cache (formerly api.CertProbeCache), with the same five tests. internal/api, internal/server, main.go and internal/notifications use it; CertAndDetectCron takes a Probes *certprobe.Cache instead of CaddyTLSDial and its 20-line copy of the dial is deleted. main.go builds one cache before the cron and hands the same instance to the HTTP handlers.

Smoke gate¶

Measured EFFECT of the batched purge, the number that matters more than boot: /api/hosts polled every 0.5 s from +112 s to +140 s after start, spanning the deferred purge at +120.5 s (1,258 rows removed, one batch):

Before this release a large purge held the single connection for the whole DELETE and /api/hosts was seen waiting up to 24.7 s behind a long statement (PHASE 0 probe). Caveat: this purge fitted in one batch (1,258 < 5,000), so the 100 ms pause between batches was not exercised; the 0.2-0.3 s that remains is the SELECT COUNT(*) over 500k rows before the cap step plus the one DELETE. A cheaper count (or skipping it when the age step removed nothing) is a candidate for a later hygiene pass.

/api/certs after: 1.4-1.8 ms per call (three calls), from 0.93-1.6 s.

Gates before commit: gofmt -l clean, go vet ./..., go test ./..., npm run build, check-no-personal-data.sh, mkdocs build --strict.

Deploy¶

Same sequence as 1.3.38.2 plus per-process sampling (top -b -n1 every 5 s, top three processes by CPU) from build start to +2 min after the panel started. Deploy 20:32:16Z -> 20:34:19Z (123 s); image sha256:fce3f844... -> sha256:7226b4bb..., /api/system/version 1.3.38.3/bfd5831, healthy at +15 s, only the panel container recreated (Caddy and CrowdSec kept their 2026-09-13 start time).

Load attribution (peak 5.6, versus 9.7 on 1.3.38.2 and 10.9 on the cold-cache 1.3.38.1 deploy):

So the remaining spike is the frontend stage: GOMAXPROCS=1 capped esbuild but node itself (rollup) ran on both cores at ~150 % for ~15 s; the Go stage stayed at one core as designed, and the panel's boot is invisible (its container is capped at cpus: 1.0 and now listens in 0.14 s). Capping node (for example --max-old-space-size does not help; a taskset/cpuset is not available under BuildKit) is not worth a release: the peak is under 6 and lasts 15 s. Panel at rest at +140 s: 0.08 % CPU, 48.6 MB.

Files changed¶

• backend/internal/db/logs.go (+ logs_purge_test.go), backend/internal/logs/retention.go -- batched, deferred purge.
• backend/internal/api/certs.go, backend/internal/server/server.go, backend/internal/logs/ingestor.go (+ ingestor_fillerror_test.go), frontend/src/api/client.ts, frontend/src/components/ActiveCertsPanel.tsx -- certs.
• backend/internal/security/country/{expander,jobs}_test.go -- test.
• backend/internal/certprobe/ (new), backend/internal/api/ (probe files removed), backend/internal/notifications/cron.go, backend/cmd/argos/main.go, backend/internal/api/{dashboard, handlers}.go -- shared probe.
• scripts/smoke/panel-boot.sh, scripts/smoke/certs-latency.sh.
• CHANGELOG.md, docs/release-notes/v1.3.38.3.md, mkdocs.yml, docs/operations/verification-report.md, frontend/package.json.