v1.3.38.2 -- Dashboard cache: stale-while-revalidate, warm-up, shared TLS probes¶

Third release of the v1.3.38 series ("dashboard paints in under a second"; scope in docs/operations/ux-performance-review.md, section 7). Four items, in the order the operator set:

1. gofmt of the pre-existing unformatted Go files, as a separate commit that precedes everything else (26 files, not the 11 the v1.3.38.1 note guessed from a three-package scan).
2. Dashboard response cache rewritten: stale-while-revalidate, single-flight, boot warm-up, background refresh, and the age of the served value in the response.
3. One shared, cached TLS probe pass for /api/certs, the Dashboard overview card and the Dashboard health card, invalidated on host and certificate changes.
4. Build resource cap for make deploy-prod on the 2-vCPU host.

No schema change. Bans, reconcilers, AppSec and auth untouched. argosVersion 1.3.38.1 -> 1.3.38.2.

2. Dashboard cache¶

internal/dashboard/cache.go used to be a TTL map: a hit inside 30 s returned the value, anything else recomputed synchronously inside the request, nothing was warmed, and two tabs opening at once recomputed twice. The new cache keeps the same 30 s TTL but changes what happens around it:

Pinned set, chosen from the measured cold compute cost of each view on prod on 2026-09-25 (operator rule: sum under 3 s, i.e. 10 % of the 30 s interval; any single view over 1 s is not pinned):

Two of the four could not be caught cold on the first round because the operator's open tab keeps them warm; the numbers above are the rounds where the value was a real compute (a hit is ~1 ms).

The warm-up runs in a goroutine started after ListenAndServe, so it can never delay /healthz or the first request; it shares the single SQLite connection like every other background job. Loads run under a 60 s context detached from the request that triggered them.

Every /api/dashboard/* response now carries generated_at in the body and X-Argos-Generated-At / X-Argos-Cache headers. The Dashboard's "updated Ns ago" counter is driven by the overview's generated_at instead of the time the response arrived, so a value served from cache 25 s after it was computed reads "updated 25s ago", not "0s".

Unit tests (internal/dashboard/cache_test.go, run with -race): miss then hit, stale serve + background refresh, beyond-max-stale synchronous reload, single-flight on miss (8 concurrent callers, one load) and on stale refresh, failed refresh keeps the old value, miss error propagates, pinned warm-up + periodic refresh, recently-used vs idle unpinned keys.

3. Shared TLS probe pass¶

Before, three code paths each dialled Caddy once per enabled tls_mode=auto host on every cache miss: /api/certs (sequential, 1.07-1.15 s per call on the operator's 19 hosts, no cache at all), the overview card's expiring-soon count and the health card's cert list (each 19 parallel dials under a 10 s timeout). One dashboard open plus one Certificates visit was 57 TLS handshakes against Caddy.

internal/api/certprobe.go runs one parallel pass for the whole domain set and memoises it for 5 min; the three callers read the same map. Single-flight, and a pass is rerun when the TTL elapsed, the domain set changed, or Invalidate was called. Invalidation sites, enumerated from every caller of the probe:

Not on the cache: the daily notifications cert cron (notifications/cron.go) keeps its own sequential probe; it runs once a day and lives in another package. Left as is for this release and listed under Known issues as a v1.3.38.3 hygiene item (two copies of one dial diverge).

/api/certs now reports last_checked_at as the time of the probe pass instead of the time of the request, which is the honest value when the pass is shared.

Unit tests (internal/api/certprobe_test.go): one pass shared within TTL regardless of domain order, re-probe on domain-set change and on Invalidate, TTL expiry, single-flight (6 callers, 2 dials for 2 domains), nil-safe Invalidate.

4. Build resource cap¶

The v1.3.38.1 deploy (cold build cache after the operator's docker builder prune) drove the 2-vCPU LXC to load 10.9 with 57 MB free while the panel kept serving. Pre-implementation check on the host: Docker 28.3.3 with BuildKit / buildx 0.26.1 exposes no --cpu-quota, --cpu-period or --cpuset-cpus on docker build (docker build --help lists only --shm-size among resource flags), so a client-side cap cannot exist. The cap therefore lives in the Dockerfile stages, driven from the Makefile:

• Go stage: ARG GO_BUILD_PARALLELISM=1, go build -p ${GO_BUILD_PARALLELISM} with GOMAXPROCS set to the same value.
• Frontend stage: GOMAXPROCS=1 (esbuild is a Go binary spawned by vite; checked in node_modules: the only esbuild env knobs are ESBUILD_BINARY_PATH, ESBUILD_MAX_BUFFER and ESBUILD_WORKER_THREADS, the last one for the sync API vite does not use) and UV_THREADPOOL_SIZE=1 for libuv.
• Makefile: BUILD_PARALLELISM ?= 1, passed as the build arg; make deploy-prod BUILD_PARALLELISM=2 on a bigger builder.

EFFECT to verify at deploy time: peak load during make deploy-prod versus the 10.9 of v1.3.38.1 (same cold-cache condition is not reproducible without another prune, which the operator has ruled out; the comparison is against the v1.3.38.0 warm-cache deploy, peak 6.1).

Smoke gate¶

scripts/smoke/dashboard-latency.sh, now with EXPECT_CACHE=hit: after 40 s idle each cold call to /api/dashboard/health must return in <= 1 s AND carry X-Argos-Cache: hit (the warm set kept it fresh; before this release the same call was a synchronous recompute). /api/security/overview keeps its own 30 s cache and is checked on latency only.

Measured EFFECT on prod after the deploy (panel started 20:07:00Z, nobody opened the panel):

• +45 s: container healthy.
• +60 s: the four default views answer in 0.6-0.8 ms with X-Argos-Cache: hit and X-Argos-Generated-At 20:07:33/34Z, i.e. the warm set's second tick; 3 min later still hit, generated 20:10:34Z. No dashboard cache: refresh failed warn in the logs.
• /api/certs: every row's last_checked_at is 20:07:02.77Z, the probe pass the warm-up ran two seconds after start, not the request time. But the call still takes 1.06 s: the shared probe pass removed the 19 dials, and what is left is enrichWithLastEvent, one LOWER(message) LIKE '%domain%' over all caddy_error rows per host, sequential (0.9 s per host with no match on the PHASE 0 copy). The review attributed /api/certs to both causes; the dials were the smaller one. Bounding that query (source + timestamp range, or host_domain match) is the first item of v1.3.38.3.
• +120 s at rest: panel 1.9 % CPU, 58 MB RSS (docker stats), the cost of refreshing the four pinned views every 30 s.
• Smoke with MAX_SECONDS=1 EXPECT_CACHE=hit: PASS (dashboard/health 0.0007 s hit, security/overview 0.0016 s).

Build with BUILD_PARALLELISM=1: 109 s, load 2.6 -> 3.4 -> 3.8 -> 4.0 -> 4.7 during the compile (v1.3.38.1 cold-cache: 4.5 -> 5.4 -> 8.2 over the same window), then 9.7 in the last sample, which covers image export + --force-recreate + panel start (boot-time log retention pass over 500k rows, reconcile, warm-up). The compile cap did what it could; the end-of-deploy spike is not the compiler and needs its own measurement (start the panel without a build) in the v1.3.38.3 PHASE 0. Minimum free RAM during the build: 31 MB, with 2.7 GB available.

/api/dashboard/overview could not be measured cold on 1.3.38.1: the operator's open Dashboard tab refreshes it every 30 s, which is exactly the situation the warm set now guarantees for everyone.

Files changed¶

• backend/internal/dashboard/cache.go (+ cache_test.go) -- SWR cache.
• backend/internal/dashboard/types.go -- generated_at.
• backend/internal/api/dashboard.go -- loaders, pins, headers, WarmDashboard, probe callers on the shared pass.
• backend/internal/api/certprobe.go (+ certprobe_test.go) -- shared probe pass.
• backend/internal/api/certs.go, hosts.go, handlers.go, backend/internal/server/server.go, backend/cmd/argos/main.go -- wiring and invalidation.
• frontend/src/api/client.ts, frontend/src/pages/Dashboard.tsx -- age from generated_at.
• backend/Dockerfile, Makefile -- build cap.
• scripts/smoke/dashboard-latency.sh -- EXPECT_CACHE.
• CHANGELOG.md, docs/release-notes/v1.3.38.2.md, mkdocs.yml, docs/operations/verification-report.md, frontend/package.json.

Upgrade¶

cd ~/argos-edge && git pull
make deploy-prod            # BUILD_PARALLELISM defaults to 1
# EFFECT: docker inspect argos-prod-panel --format '{{.Image}}' differs from before
ARGOS_SESSION_TOKEN=<cookie> MAX_SECONDS=1 EXPECT_CACHE=hit scripts/smoke/dashboard-latency.sh