UX + perceived-performance review (PHASE 0)¶

Read-only review of the panel as deployed on the operator's prod LXC. Goal: make what already exists coherent, fast to open and easy to find. No features. This page is the PHASE 0 deliverable; nothing has been changed in code, compose files or running stacks. PHASE 1 scope is proposed at the end and waits for operator approval.

Review date: 2026-09-25 Panel binary measured: 1.3.35 (commit d6a32b3), running in argos-prod-panel Repo HEAD: 66e414c (v1.3.37) Method: LXC inventory, source map (frontend + backend), HTTP GET timing against the live prod panel, EXPLAIN QUERY PLAN and single-query timing against a sqlite3 .backup copy of the prod DB (copy deleted afterwards), headless-browser paint timing per route.

Status: sections 0-8 are the PHASE 0 record as measured on 1.3.35 and are kept as written. Section 9 is the live status after v1.3.38.0-v1.3.38.5 (2026-09-26): what closed, with the before/after measured on prod, what is open and when.

0. Executive summary¶

1. Environment inventory (LXC)¶

Host: 2 vCPU, 2048 MB RAM (swap 512 MB, 169 MB in use), root disk 28 GB at 91 % used, 2.5 GB free. Load average ~2.5 on 2 cores at rest. The panel is limited to 512 MB, Caddy 256 MB, CrowdSec 256 MB (CrowdSec sits at 206-228 MB of its 256 MB cap).

Directories:

Running containers (all belong to the prod stack):

Volumes in use by prod: argos_prod_* (9). Orphans: argos_* without prefix (7: argos_caddy_*, argos_crowdsec_*, argos_panel_*, argos_shared_setup) and network argos_net. Demo: argos_demo_* (9), no network, no containers.

1.1 Anomalies (STOP+REPORT before any state-changing step)¶

1. Compose project name is inverted with respect to the working agreement. docker-compose.yml declares name: argos-edge, and the prod override does not rename the project, so the live prod stack is compose project argos-edge (config files under ~/argos-prod). The project named argos-prod is a dead leftover: one exited argos-crowdsec-init started from ~/argos-edge/docker-compose.yml. Consequence: a docker compose -p argos-prod ... command targets the orphan, and -p argos-edge targets prod. Every operator note and script that assumes -p argos-prod is wrong today. Nothing was run against either project during this review.
2. Image pin vs running image. The prod override pins image: argos-prod-argos:1.3.35.4, the running container is argos-prod-argos:1.3.35 (binary reports 1.3.35, built 2026-04-27). Images 1.3.35.2, 1.3.35.3, 1.3.35.4 exist locally but were never rolled into the container (container is 12 days old, i.e. restarted with the old image on the last host reboot). v1.3.35.2-.4 are demo/tooling patches, so no prod behaviour is missing, but this is the strike-11 shape: the compose file says one thing, docker ps another. The deploy-rebuild.sh smoke would have caught it.
3. Disk at 91 %. A .backup copy of the 1.7 GB DB took the root FS to 97 %; the copy was removed. Nightly backups are ~128 MB each (14 kept, ~1.75 GB). Any PHASE 1 work that needs a DB copy or the demo stack must budget disk first.
4. RAM headroom is ~1.4 GB "available" but almost all of it is page cache holding the DB. Starting the demo stack (3 containers, ~400 MB) evicts that cache and slows prod reads; the cold numbers below would get worse, not better.

Orphans (1) and the pin mismatch (2) are operator decisions; nothing here requires them to be fixed before PHASE 1, but (1) must be settled before any smoke that uses docker compose -p.

2. Navigation map and frictions¶

2.1 Current map¶

Twelve flat entries in a hamburger drawer at every viewport (frontend/src/components/Layout.tsx:21-34): Dashboard, Hosts, Target Groups, Security, Threats, AppSec, Notifications, Certificates, Logs, Backup, System, Settings. Header pills link to /system (version, LAN mode) and /appsec.

Pages reachable only through in-page links: /hosts/:id/rules, /hosts/:id/security, /security/hosts (Security overview), /target-groups/:id. Tabbed pages keep the active tab in component state only (Security 5 tabs, Notifications 4, Backup 3, Certificates 2): a reload or a shared link always lands on the first tab.

Global behaviours that shape every navigation:

• ProtectedRoute re-fetches /api/auth/me and blanks the page to a "loading..." screen on every route change (ProtectedRoute.tsx:22-48, remounted per route in App.tsx:88-94).
• SelfBlockBanner polls /api/security/check-self every 60 s on every page; each poll is one LAPI round-trip per session IP (0.13-0.19 s measured).
• Layout polls /api/appsec/status every 30 s (a live HTTP probe to crowdsec:7423).
• No URL state anywhere (useSearchParams unused), no document.title, no breadcrumbs except Rules.

2.2 "Where does each thing live"¶

2.3 Concrete frictions ("I want X, I have to go A -> B -> C")¶

1. "Is this IP banned, why, and what did it do?" Threats (search, case-sensitive) or Security > Banned IPs (search, case-insensitive) for the decision; then Logs, paste the IP into q (which LIKEs the 1.4 KB raw column of every row in range) for its requests; Dashboard "top attacking IPs" links to /logs?q=<ip> but Logs drops q, so you land on the unfiltered log. Three pages, three inputs, one broken link.
2. "Turn the WAF to block for host X." Hosts -> shield icon -> HostSecurity -> Save -> you are dropped on /security Banned IPs tab. Or Security -> "Hosts" link -> overview -> Configure -> Save -> same drop.
3. "Ban a country." Settings (last section of the longest page). Progress polls 1 s for up to 10 min. Not discoverable from Security or Threats.
4. "Is the WAF doing anything?" Dashboard says "No WAF events in range" (dead source). Security overview says "Blocked 24h: 0". AppSec page says 172 blocked. Threats shows bans. The operator has to know which one is real.
5. "What did the panel change yesterday?" Security > Activity tab (not a top-level entry), or Logs with source=audit; the Dashboard card counts them but does not link.
6. "Open the panel and see if things are fine." Dashboard after >30 s idle: 5 requests, one of which (health) blocks 21-25 s, during which every other request in the panel and the log ingestor queue behind it (section 3.3). Each section shows "loading..." until its own call returns; there is no cached last-known state.
7. "Share a view with future me." No URL carries tab, range, filter or search state; a bookmark or a reload always resets to defaults.

3. Load time per page (measured against prod)¶

3.1 API endpoint timing¶

Method: curl with the operator session cookie from the LXC itself (no network hop). "cold" = first call after >30 s idle (every 30 s cache expired) with the DB mostly in the OS page cache; "first" = the very first call of the session while the DB copy was being made (cold disk + contention), shown to bound the worst case an operator sees after a quiet night. "warm" = immediate repeat.

Ranges for the dashboard on the same cold cache: traffic 1h 0.05 s, 6h 0.22 s, 24h 0.88 s, 7d 119.5 s; security 1h 0.004 s, 6h 0.01 s, 24h 0.02 s, 7d 1.24 s.

3.2 Per-page composition¶

From the frontend map (calls on mount, poll interval) combined with the timings above. "Cold open" is the sum an operator waits for the slowest card after >30 s idle.

3.3 The single-connection effect (EFFECT verified)¶

internal/db/db.go:61 sets SetMaxOpenConns(1); docs/ architecture/storage.md documents it as the single-writer choice. In practice it also serialises every read: WAL's concurrent readers are never used.

Probe: 40 x GET /api/hosts every 250 ms while GET /api/dashboard/health recomputed cold.

The same applies to the log ingestor flush (every 2 s), session touch, drift detector writes and the notification worker: all wait behind any of the slow reads above. A 7d range selection therefore stalls ingestion for two minutes.

3.4 Database shape (copy of prod)¶

Why every "visit each row" query costs ~10 s: rows average ~2.5 KB on disk because of raw, so reading timestamp,status for 480k rows means touching 1.2 GB of pages. A covering-index count over the same window takes 0.027 s.

3.5 Query plans and as-shipped vs proposed shapes (copy of prod)¶

4. Search and filter inventory, and a unified model¶

4.1 Inventory¶

Backend naming for the same things: q (logs, security) vs search (threats); total (security decisions) vs total_count (logs) vs nothing (deliveries); range (dashboard) vs window (appsec) vs from/to (logs, deliveries); limit defaults 100/100/200/20/5 with maxes 1000/1000/1000/200/50; removal outcomes unbanned/deleted/removed.

UI naming: "Banned IPs" and "Unban" (Security) vs "Active decisions" and a "whitelist" button that removes a ban (Threats) vs "Active bans" (Dashboard). "Blocked" means status IN (403,429) on the Dashboard overview, status=403 in the security chart, waf_audit CRITICAL/ERROR rows in Security overview, and alert with decision or block mode on the AppSec page. "Alerts" means CrowdSec alerts (AppSec), notification deliveries (recent-alerts), and WAF counts (alerts_critical_24h, which is numerically the same as blocked_24h_total).

4.2 Proposed unified model¶

One FilterBar component and one query vocabulary, used by every page that lists or aggregates time-bound data:

Behaviour rules for the component: 300 ms debounce on text, AbortController on every refetch, state mirrored in the URL (useSearchParams) so links, reloads and bookmarks reproduce the view, and every page that shows an IP, a domain, a scenario or a CRS rule id renders it as a link that opens the same FilterBar pre-filled on the target page. "Paste an IP anywhere" then resolves to the same place: the decisions view with ip=, and a "Logs" link with ip= and range=24h.

Global search: a header input that classifies the pasted value client-side (IP/CIDR, domain, CRS id, text) and routes to the right page with the filter set. No new aggregate endpoint is required for the first version; the existing lists already accept the fields. A cross-entity GET /api/search is a later option if the routing version proves insufficient.

Terminology to standardise (UI + JSON): decision is the CrowdSec object; the UI calls a type=ban decision a ban and the action "Unban"; blocked request is any 403/429 at the edge; WAF alert is a CrowdSec AppSec alert; event is a log row; notification is a delivery. alerts_critical_24h goes away or gets a real definition.

5. Prioritised proposal¶

Effort: S (< 1 day), M (1-3 days), L (> 3 days). Risk is regression risk to security behaviour (bans, reconcilers, AppSec, auth), which none of these items touch by design.

Item 0 is a bug, not a design choice, and goes first. Items 1-5 and 7 make the dashboard paint in under a second from cache and remove the two-minute stalls without touching the schema or any security path. Item 6 is the one architectural change and is the only item that needs an explicit decision before implementation. Items 12-14 are product decisions and are listed so they are not forgotten, not proposed for the first release.

6. What NOT to touch, and why¶

• Reverse-sentinel files (/data/shared/argos-*.json, argos-*.txt, argos-managed-profiles.yaml) and their readers/writers: they are the only channel between the panel (as nobody) and setup-appsec.sh (as root). Any cache or filter refactor must keep mtime-based invalidation intact.
• Async-job pattern (country_expansion_jobs, JobRunner, boot-time recovery): a UI move of the country-bans section is fine; the submit/poll contract and the 1 s job polling stay.
• CAPI alert shape for range bans (1 alert, N decisions) and the LAPI client emit paths (AddRangeDecisions, revoke by origin=): strikes 4, 8 and 9 live here.
• Migrations: schema is frozen since 033. Items 1-13 need no migration. If 14 is approved it is a single additive migration with the rollback test extended; nothing rewrites log_entries.
• Drift detector and country reconciler tickers, bouncer polling, AppSec mode switching, ForwardAuth cache, session and TOTP flows: not part of a UI/performance release; any change there is a separate release with its own smoke.
• Bootstrap Caddyfile and compose files: the only compose change worth considering is out of scope here (project naming in 1.1) and is the operator's call.
• SetMaxOpenConns(1) for the writer: keep it. Item 6 adds a reader pool beside it; it does not remove the single writer.

7. Proposed scope for the first release (v1.3.38, "dashboard paints in < 1 s")¶

Backend (no schema):

1. RecentErrors bounded + UNION ALL (item 1).
2. Security overview grouped query (item 2).
3. Dashboard cache with stale-while-revalidate, single-flight, boot warm-up and background refresh (item 3).
4. Shared 5 min cert-probe cache (item 4).
5. 7d traffic/stats via covering-index counts; q on raw bounded (item 5).

Frontend:

1. Fix the onLoaded refetch loop (item 0). Smoke: open / in headless Chromium for 60 s after the page settles and assert fewer than 10 /api/dashboard/overview requests.
2. One /auth/me per session; no blank screen on route change; dashboard last-known data from sessionStorage + skeletons (item 7).
3. Threats pagination (item 9): /api/threats/decisions returns {decisions, total, limit, offset}; the page requests 100 at a time. Smoke: response < 100 KB and page settles < 2 s with the CAPI blocklist enrolled.

Smoke (EFFECT, prod density):

• scripts/smoke/dashboard-latency.sh: after 40 s idle, GET /api/dashboard/health, /overview, /security/overview each < 1 s; traffic?range=7d and logs/stats?from=<7d> < 5 s; concurrent /api/hosts p99 < 50 ms during a forced cache miss. Run against ~/argos-prod before tag.
• Capture: dashboard-overview.png / dashboard-security.png before/after via scripts/capture/run.sh.

Explicitly out of this release: item 6 (reader pool) unless the operator approves the architecture change, and everything from item 8 onwards (v1.3.39 filters, v1.3.40 navigation).

8. Browser paint timing (headless Chromium, 1440x1080, from the LXC)¶

Method: Playwright Chromium (the capture toolchain's own browser) logs in once, then navigates route by route with all server caches warm. shell = header rendered; settled = no "loading..." text left and network idle (cut off at 120 s); API calls = requests to /api/* seen until settled. A second pass with 35 s idle before each route (cold caches) was started and aborted: the first pass alone pushed the 2-core LXC to a load average of 11 while the Threats page rendered, so the cold pass was not worth the risk to prod. Cold server costs are already in section 3.1.

The 0.7 s floor on trivially cheap pages is the cost of ProtectedRoute re-fetching /auth/me and blanking on every navigation plus the three layout probes; item 7 removes it.

Threats belongs with item 9 (server pagination) but deserves its own line in the first release: with the community blocklist enrolled, the page is unusable as shipped and the 15 s poll keeps a 7.2 MB download plus a full re-render going in the background.

8.1 Observations on the host during the probe¶

Load average went from ~2.5 to 11.5 (2 vCPU) while Chromium rendered /threats; the panel container stayed at 0.3 % CPU and ~240 MB. The LXC memory limit was raised from 2,048 MB to 4,048 MB during the review (observed at 20:46 UTC); the numbers in section 1 describe the 2 GB state at the start.

9. Status after v1.3.38.x (2026-09-26)¶

Six releases, v1.3.38.0 to v1.3.38.5, all deployed with make deploy-prod and verified on prod (verify-deploy, deploy-rebuild.sh). No schema change in any of them; the schema is still frozen at migration 033. Numbers below are the tag messages and release notes, all measured on prod unless marked "demo" (browser runs stay off prod by operator rule).

9.1 Resolved in v1.3.38.x¶

Cron and probe items from section 2.1 / 3.3 are in this table: the boot retention cron (v1.3.38.3) and the per-page TLS probe passes (v1.3.38.2) are closed; the check-self 60 s poll and the appsec/status 30 s probe stay as measured (0.13-0.19 s each) and are listed under open.

9.2 Open¶

9.3 Prod resources after five releases (baseline for v1.3.40)¶

Measured 2026-09-26 10:28 UTC+2 on the prod LXC with the panel at rest (no tab open), panel binary 1.3.38.5 started 07:51:03Z.

What v1.3.40 has to move: the DB size (raw is ~1.4 KB of every access row; item 13), the 7 d cost model (rollup; item 14), the CrowdSec container's memory headroom (26 MB under its cap, not a panel item but the next thing to fail on this host), and the reader pool decision (item 6).