Skip to content

v1.3.8 — AppSec log-spam fixes + defensive client-IP propagation

Bug-fix release. Closes a long-standing source of CrowdSec log noise that made genuine WAF events hard to spot, and makes the client-IP plumbing explicit so the WAF-inline feature works correctly behind a future CDN / cloud LB.

Bugs fixed

1. Panel AppSec probes were generating log spam every 30 seconds

Symptom (reproduced in prod): every 30 s, CrowdSec logged

level=error msg="missing 'X-Crowdsec-Appsec-Ip' header"
       module=acquisition.appsec name=argos-appsec-detect type=appsec

The cadence held even with no traffic flowing through Caddy -- clearly not a real-traffic bug. Trace narrowed the source to the panel container's two AppSec liveness probes:

  • backend/internal/appsec/healthcheck.go -- the appsec_unavailable notification cron.
  • backend/internal/appsec/hub.go -- the Status-page rule-count probe.

Both used to dial http://crowdsec:7423 with only the bouncer API key. CrowdSec's AppSec listener validates four envelope headers (X-Crowdsec-Appsec-Ip / -Uri / -Verb / -Host) before rule evaluation; missing any of them aborts the request and logs an error per probe.

Fix: both probes now send a synthetic-but-well-formed AppSec envelope:

req.Header.Set("X-Crowdsec-Appsec-Ip", "127.0.0.1")
req.Header.Set("X-Crowdsec-Appsec-Uri", "/.well-known/argos-appsec-healthcheck")
req.Header.Set("X-Crowdsec-Appsec-Verb", "GET")
req.Header.Set("X-Crowdsec-Appsec-Host", "argos-panel.local")

CrowdSec accepts the request, runs it through the rule engine (no rule matches the well-known path / loopback IP combo), and replies allow cleanly with zero log output. Liveness behaviour is unchanged -- the probe still succeeds on any HTTP response.

2. Caddy trusted_proxies was implicit; client IP could be lost behind a future CDN

The Bug A filing flagged this as the suspected root cause. Trace showed it was NOT the actual cause of the prod symptom (Caddy's determineTrustedProxy falls back to RemoteAddr and populates caddyhttp.ClientIPVarKey correctly under the current single-hop deployment shape -- verified client_ip in Caddy's access logs).

But the broader concern is real: the moment a CDN / cloud LB / ingress-controller sits in front of Caddy, the implicit fallback would surface the LB's IP rather than the real client. The caddy-crowdsec-bouncer plugin reads ClientIPVarKey to build the X-Crowdsec-Appsec-Ip header it forwards to AppSec. Wrong IP -> wrong allowlist behaviour, wrong rate-limit keying, wrong audit logging.

Fix: the Caddy config emitter (backend/internal/caddycfg/caddycfg.go) now sets trusted_proxies and client_ip_headers on the main server:

{
  "trusted_proxies": {
    "source": "static",
    "ranges": [
      "127.0.0.1/8", "::1/128",
      "10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16",
      "fc00::/7"
    ]
  },
  "client_ip_headers": ["X-Forwarded-For"]
}

Defaults cover RFC1918, Docker bridges, IPv4/IPv6 loopback, and ULA. A public-cloud-LB deployment can extend the ranges list in a future release; today's single-host LXC + Docker shape works unchanged.

Bug B — known cosmetic issue (kept as-is)

docker compose logs crowdsec shows ~190 conflicting id <N> for rule ! warnings on every boot. Cause: argos installs two AppSec acquisitions (argos-appsec on :7422 for block mode, argos-appsec-detect on :7423 for detect mode) so the bouncer can flip mode by changing appsec_url at runtime without a CrowdSec restart -- a UX win. Both acquisitions reference the same rule collections; the second-loaded one logs a conflict warning per rule.

Functional impact: none. First-loaded copy stays effective; both listeners route requests against that rule pool.

Why deferred: collapsing to a single acquisition either regresses the mode-toggle UX (CrowdSec reload on every change, ~1 s window) or requires operator intervention to re-run setup-appsec.sh. Neither is worth shipping for a cosmetic boot warning. A future release may revisit if upstream CrowdSec gains a shared-rule-pool mode.

Operator workaround documented in Troubleshooting -> Boot warnings: conflicting id.

Investigated, not addressed in v1.3.8

The Bug A filing additionally claimed total_hits permanece en 0 indefinidamente after sending obvious attack payloads. Reproduced. Trace results:

  1. Real traffic IS reaching AppSec. client_ip: 172.18.0.1 in Caddy's access log; direct wget to :7422 / :7423 from inside the caddy container returns {"action":"allow"} with valid headers.
  2. Rules ARE loaded. cscli appsec-rules list shows 188 inband + 2 outofband (vpatch-CVE-, generic-, base-config).
  3. Rules don't emit alerts in the detect-mode config. The vendor crowdsec/crs appsec-config declares an explicit on_match: - filter: IsOutBand == true; apply: SendAlert() directive; the argos appsec-detect config and the vendor appsec-default do not. Without SendAlert(), rule matches are evaluated and the verdict is returned to the bouncer, but no LAPI alert is created -- so cscli alerts list and the panel's total_hits both stay at 0.

The fix is a one-line change to crowdsec/appsec-configs/argos-appsec-detect.yaml adding the on_match: SendAlert() directive. It needs upstream-config review (does adding it to inband break block-mode behaviour? does it overwhelm LAPI under DDoS?) and is therefore deferred to v1.3.9 with proper testing rather than rushed into this release. The misleading "missing IP header" log spam that drove the original bug filing is fixed here regardless.

Tests

  • appsec.TestPingSendsAppSecEnvelopeHeaders -- verifies the four envelope headers are sent on healthcheck probes.
  • Existing 6 healthcheck/hub tests still pass.
  • caddycfg tests -- existing JSON-shape assertions unchanged (the new trusted_proxies / client_ip_headers fields are additive).

Files changed

  • backend/internal/appsec/healthcheck.go -- envelope headers on probe.
  • backend/internal/appsec/hub.go -- envelope headers on Status-page probe.
  • backend/internal/appsec/healthcheck_test.go -- new test covering the envelope.
  • backend/internal/caddycfg/caddycfg.go -- trustedProxies type, defaultTrustedProxies() builder, server initialised with both fields.
  • docs/operations/troubleshooting.md -- two new entries ("missing 'X-Crowdsec-Appsec-Ip' header every 30 s" and "Boot warnings: conflicting id ...").

Upgrade

Drop-in:

cd argos-edge
git pull
docker compose build
docker compose up -d

The CrowdSec log noise stops within seconds of the panel coming back up. The Caddy trusted_proxies field is additive; existing deployments behave identically until they're put behind a real proxy.