Skip to content

Pre-public functional verification

This page is the v1.3.36.8 verification gate -- a single-source inventory that maps every shipped feature of argos-edge to a smoke script (or documents why no smoke exists). Run before making the repo public; re-run on any future release that might regress a covered surface.

Summary

Pre-v1.3.32 smoke scripts 9
Verification gap fillers (v1.3.32) 4
Post-v1.3.32 smoke scripts (v1.3.33-v1.3.36.x) 5
Total smoke scripts 18
EFFECT-verified PASS against prod stack (panel binary v1.3.35) 16
Gated on operator-mediated input (creds / TOTP) 1 (auth-flow)
Legacy regression test (intentionally tests broken path) 1 (country-block)
Blockers preventing public release 0

The four post-v1.3.32 additions reflect features shipped or deploy-pipeline incidents addressed since the original verification gate was drawn:

  • country-reconciler.sh (v1.3.33) — 5min ticker EFFECT for expansion-divergence detection
  • lapi-flush-cap.sh (v1.3.33) — alert-shape verification (one alert with N decisions, mirroring CAPI/community-blocklist shape) after the eight-strike CAPI cascade-flush incident
  • deploy-rebuild.sh (v1.3.34.3) — make deploy-prod actually rebuilds the panel image (closes eleventh-strike silent-no-op gap)
  • demo-environment.sh (v1.3.35) — ~/argos-demo parallel stack self-smoke (separate volumes/network from ~/argos-prod)
  • capture-automation.sh (v1.3.36.x) — Playwright capture spec self-smoke (storageState wiring, safeClick blocklist, per-surface selector regression-guards). 14 phases of static checks.

Smoke matrix

Each row: feature, smoke script, last EFFECT verified.

Feature Smoke script Status Verifies
Sync-prod operator tooling sync-prod.sh ✅ PASS 5/5 self-gates against tmpdirs (refuse invalid paths, no-op when in sync, drift propagates, operator files protected, excludes work)
LAPI SQLite WAL (v1.3.28) lapi-wal.sh ✅ PASS PRAGMA journal_mode=wal; startup warning absent in current container's logs; .db-wal sidecar present
Scenario descriptions (v1.3.30) scenario-descriptions.sh ✅ PASS Slimmed index produced (115KB); 54/54 scenarios have description; CVE-2017-9841 carries expected text; graceful degrade with file removed; mtime-driven recovery
Scenarios management (v1.3.25) scenarios-toggle.sh ✅ PASS PATCH disable -> sentinel -> setup-appsec.sh -> cscli scenarios list confirms removed -> re-enable -> cscli confirms back
AppSec tuning (v1.3.25) appsec-tuning.sh ✅ PASS PATCH inbound 12 -> sentinel -> reload -> argos-tuning.yaml carries new threshold -> restore round-trip
Drift detection (v1.3.27) drift-detection.sh ✅ PASS 12 phases: scenarios + tuning surfaces both flip drift_detected=true on PATCH+wait65s and clear on setup-appsec.sh+wait65s
True detect mode (v1.3.29) true-detect-mode.sh ✅ PASS PUT true_detect_mode=true -> profiles.yaml splice -> crowdsec restart -> synthetic LAPI alert with target_fqdn=test_host produces 0 decisions; toggle off produces 1 decision (default_ip_remediation baseline)
Country expansion async (v1.3.31) country-expansion-async.sh ✅ PASS 8 phases: BR async expand 11/11 chunks 5009 ranges in <60s; failure path with crowdsec stopped produces state=failed + error_message; recovery to healthy within 30s
Country expansion legacy (v1.3.21) country-block.sh ⊘ SKIP-LEGACY Tests upstream-broken cscli scope=Country path the bouncer plugin doesn't natively handle (seven-strike #2). Replaced by country-expansion-async.sh. Refuses to run with placeholder defaults by design
Authentication lifecycle (NEW) auth-flow.sh ⏸ DEFERRED Login -> session cookie -> /me -> logout -> 401. Requires operator credentials (ARGOS_USERNAME + ARGOS_PASSWORD); cannot run unattended in CI. Detects TOTP-pending response and exits PASS-PARTIAL
Host CRUD + Caddy reconcile (NEW) host-crud.sh ✅ PASS 7 phases: POST host -> GET echo -> toggle flips enabled -> PUT updates auth_required -> DELETE -> 404 -> caddy admin status reachable (proxy for "reconciler healthy")
Whitelist round-trip (NEW) whitelist-roundtrip.sh ✅ PASS 8 phases: POST whitelist -> GET contains -> sentinel updated -> setup-appsec.sh -> argos-whitelist.yaml has the IP -> DELETE -> sentinel + yaml clean
Banned IPs round-trip (NEW) banned-ips-roundtrip.sh ✅ PASS 5 phases: cscli add -> panel /security/decisions lists with origin=cscli -> panel DELETE -> cscli confirms gone (15s cache TTL on Client.ListDecisions accounted for)
Country expansion reconciler (v1.3.33) country-reconciler.sh ✅ PASS 5min ticker compares panel-tracked CIDR count against actual LAPI Range decisions for the country; flips state='drifted' when divergent; clears on next reconcile after expansion completes
LAPI alert-shape cap (v1.3.33) lapi-flush-cap.sh ✅ PASS Mirror CAPI/community-blocklist shape: 1 alert with N decisions inside decisions[] (NOT N alerts with 1 decision each). NG +1 chunk + IR +3 chunks under 5000-item flush.max_items default; no cascade flush observed
Deploy-pipeline rebuild (v1.3.34.3) deploy-rebuild.sh ✅ PASS make deploy-prod actually rebuilds the panel image (post-fix for the eleventh-strike build: !reset + image-pin silent no-op that let v1.3.34.1+v1.3.34.2 ship without deploying). Verifies image hash changes after a known source edit
Demo environment isolation (v1.3.35) demo-environment.sh ✅ PASS ~/argos-demo parallel stack self-smoke — separate compose project, volumes, and docker bridge from ~/argos-prod; ensures demo-stack mods can never bleed into operator's prod
Playwright capture spec (v1.3.36.x) capture-automation.sh ✅ PASS (14/14 phases) Static checks: run.sh refuses without .env, .env gitignored, viewport 1440x1080, storageState wiring, safeClick blocklist (13/13), waitForSettled helper, openModal modal-visibility wait, host-row trigger selector, safeClickTab tab nav, DNS-01 selector, threats-decisions selector + screenshot helper

Coverage gaps documented

Feature Why no automated smoke Mitigation
Recovery CLI subcommands (reset-password, disable-2fa, migrate, restore) CLI invocation against the panel binary needs a separate test process + container exec; meaningful test would require seeding a known user state and asserting post-conditions. Not blocking; CLI is operator-only and exercised manually during incident recovery Documented invocation in docs/operations/troubleshooting.md (existing); each subcommand has a --help that the operator validates before running it for real
Self-block detection / banner v2 Requires the operator's actual public IP to be banned in CrowdSec to surface the banner; cannot synthesise without breaking the operator's own connectivity to the panel. The underlying API endpoint (GET /api/security/check-self) is exercised via the auth-flow.sh smoke (the panel returns the data; the banner is pure UI) Manual: operator follows the documented "self-block recovery" flow at first onboarding to validate the banner renders correctly
Activity / audit log queries Read-only endpoint with no behavioural side effect; an empty response is indistinguishable from a working query against a fresh DB. Smoke would mostly assert "200 OK + JSON-shape" which adds little signal Read-only; if the endpoint breaks the only impact is the Activity tab renders empty. No incident risk
Dashboard widget stats Aggregated counters (banned count, whitelist count, etc.) computed live from the same endpoints other smokes already exercise; if those work the dashboard math works Implicit via banned-ips + whitelist + scenarios smokes
TOTP / 2FA enrollment + verification flow Requires interactive operator (TOTP secret + a real authenticator app); not scriptable Manual: documented in docs/features/auth-local.md
OIDC SSO end-to-end Depends on an external IdP (Keycloak / Authentik / etc.); operator's choice of IdP varies per deployment Per-IdP smoke would belong in the IdP's own test surface, not argos-edge
Backup + restore round-trip Existing argos backup CLI + argos restore are exercised manually during incident recovery; building a smoke would require tearing down the panel mid-test which conflicts with running other smokes alongside Documented manual path; make sync-prod covers operator-tooling sweeps
Reverse-proxy live healthcheck propagation Caddy's healthcheck pings backend targets; would need to spin up a stub backend that answers 200 vs 503 to assert; out of scope for a single-stack homelab smoke Implicit via host-crud (Caddy admin reachable post-reconcile)
Notifications (Slack / push / email) External delivery side-effects; smoking these would spam real channels. Each provider has its own test surface Per-provider configurability test exists in panel UI ("Send test notification" button)

Recommendation: ready for public

All 16 in-scope smokes PASS against the v1.3.35 panel binary. The 1 deferred (auth) is an operator-credential concern, not a code defect; the underlying handlers are exercised indirectly by every other session-bearing smoke. The 1 legacy-skip (country-block) tests an upstream-known-broken path that v1.3.21 worked around.

The eleven-strike upstream-behaviour pattern (documented in CLAUDE.md and the per-strike memory file) is now reflected in the smoke matrix: every external-protocol surface that caused an incident has a dedicated EFFECT-verifying smoke (LAPI WAL, scenarios source-of-truth, AppSec tuning, drift detection, true-detect-mode, country expansion async, country reconciler, alert-shape cap, deploy-pipeline rebuild).

Zero blockers. The repo is functionally ready for public release. The pre-public audit (docs/operations/pre-public-audit.md, v1.3.37) covers the non-functional gates (sanitization, doc currency, GitHub governance files).

How to re-run

SESSION=$(docker run --rm -v argos_prod_data:/data alpine sh -c \
    "apk add --no-cache sqlite >/dev/null 2>&1
     sqlite3 /data/argos.db \"SELECT token FROM sessions
       WHERE expires_at > datetime('now')
       ORDER BY id DESC LIMIT 1;\"")

# Cheap (no-auth, ~5-30s each):
./scripts/smoke/sync-prod.sh
CROWDSEC_CONTAINER=argos-prod-crowdsec ./scripts/smoke/lapi-wal.sh

# Auth-needing happy paths (~15-90s each):
ARGOS_SESSION_TOKEN="${SESSION}" \
PANEL_BASE_URL=http://localhost:9180 \
CROWDSEC_CONTAINER=argos-prod-crowdsec \
  ./scripts/smoke/scenario-descriptions.sh

# ... repeat the env block for the other smokes ...

# Long (4-min drift detector + 90s country async):
ARGOS_SESSION_TOKEN="${SESSION}" \
PANEL_BASE_URL=http://localhost:9180 \
CROWDSEC_CONTAINER=argos-prod-crowdsec \
  ./scripts/smoke/drift-detection.sh

ARGOS_SESSION_TOKEN="${SESSION}" \
PANEL_BASE_URL=http://localhost:9180 \
CROWDSEC_CONTAINER=argos-prod-crowdsec \
COMPOSE_DIR=$HOME/argos-prod \
SKIP_FAILURE_PATH=1 \
  ./scripts/smoke/country-expansion-async.sh

# Operator-credentials smoke (manual):
ARGOS_USERNAME=admin ARGOS_PASSWORD='...' \
  ./scripts/smoke/auth-flow.sh

What this report does NOT prove

  • Frontend visual rendering. Smokes exercise the API; the React UI is verified by the operator's browser pass.
  • Load behaviour at scale. Smokes are one-shot single-host exercises; sustained load testing is out of scope.
  • Cross-version migration path. Smokes run against the current schema; incremental upgrade from older argos-edge versions is documented in each release's notes and would warrant a dedicated migration smoke if released as a product (not yet a homelab need).