Skip to content

v1.3.42.2: the community-blocklist pull no longer floods the event queue (2026-09-28)

Patch release between the rollup (42.0), the FilterBar (42.1) and the rollup readers, which take the v1.3.42.3 slot from here on (numbering decided by the operator on 2026-09-28; both planning docs say so). One backend file, one test, one smoke, no schema, no bouncer change.

What was wrong

The 48 h reading of 2026-09-28 (v1.3.40 planning doc, section 11) found 219,607 WARN notifications: event queue full, dropping lines in the panel log in 40 h, dropped_total 230,813, type threat_ip_banned, in 20 bursts of 10-12k every 2 h. The bursts are the CrowdSec community-blocklist pull: it rewrites its ~15,000 decisions with new IDs, the monitor (internal/crowdsec/monitor.go) diffs decision IDs against the previous 15 s poll and emitted one threat_ip_banned event per new ID, so every pull was 15,000 "new bans" into the 1,000-slot event queue. Pre-existing since the monitor was written; the v1.3.42.1 restart put it at the top of a fresh log. No notification was lost (the notification_rules table is empty on prod); the cost was 21 MB of journal per day and the CPU of building and dropping 15k events every 2 h.

What ships

  • notifyPerDecision(origin): decisions with origin CAPI or lists (case-insensitive) no longer get a per-decision threat_ip_banned event. Local origins (crowdsec, cscli, manual, the panel's own argos-country-*) still notify one by one. The threat_intel_updated summary is unchanged and keeps the added / removed / total counts of every pull.
  • TestMonitorSkipsBlocklistPerDecisionEvents: a simulated pull of 15,000 CAPI decisions plus 10 lists decisions with new IDs and one new local decision produces exactly one threat_ip_banned (the local one), one summary with added_count 15,011, and zero drops from the queue. TestNotifyPerDecision pins the origins.
  • scripts/smoke/event-storm.sh, EFFECT: over the last 4 h (two pulls at the 2 h cadence, asserted from the crowdsec log) the panel log carries zero event queue full lines; the panel's log volume in MB per day is reported next to the before figure.
  • docs/features/notifications.md: the two event rows say what fires them now.

Host change, outside the repo (operator's decision, 2026-09-28, reversible): /etc/systemd/journald.conf.d/50-argos-maxuse.conf with SystemMaxUse=500M and systemctl restart systemd-journald on the LXC. Journal 2.7 GB -> 491 MB, disk 87 % -> 78 % (3.6 GB -> 5.9 GB free); the containers kept logging through the restart (crowdsec 61 lines in the next minute).

Verified before the deploy

  • go build ./..., go vet, gofmt, go test ./... (28 packages ok), ASCII grep on the touched .go files, check-no-personal-data.sh.
  • event-storm.sh self-run on prod while 42.1 was still running: FAIL as expected, 21,395 queue-full lines in the 4 h window with 2 pulls, 19.1 MB per day of panel log; that run is the "before" of the after-smoke.
  • Not run on the demo: the demo stack has no CAPI enrolment, so the real pull only exists on prod; the unit test simulates it.

Deploy

make deploy-prod with the operator's OK; rollback = repin 1.3.42.1 and make deploy-prod. The restart resets the 6 h drift check cadence (the ticker starts at the boot purge, +2 min), so the next two verdicts land at deploy + 2 min + 6 h and + 12 h; the 42.3 gate wants verdicts 7 and 8 at 0, which is why the deploy time matters (after 11:56 UTC keeps verdict 7 on the old cadence).

After-smokes: event-storm.sh at deploy + 4 h with BEFORE_MB_PER_DAY=21, panel-boot.sh, read-pool.sh gates 0-2, the boot log (rollup: fill done for the hours the restart missed), and the verdicts as they come.

Results (deployed 2026-09-28 13:56 UTC)

make deploy-prod from main at 13:54:53 UTC (sync, image build, recreate), panel 1.3.42.2 healthy 37 s after the recreate, no rollback; boot log read pool enabled, boot purge at 13:58:31 UTC (6,074 removed, 3,419 stripped), boot backfill of the one hour the restart cut (12:00 UTC, 29 ms). The drift cadence restarted with that purge: verdict 8 is due about 19:58 UTC.

smoke result
panel-boot.sh PASS: boot-to-listen 0.12 s, first purge 123 s after listen
dashboard-latency.sh (MAX_SECONDS=1) PASS: health 1.1 ms cold, security overview 48 ms cold
read-pool.sh gates 0-2 PASS: idle p99 4.8 ms (n=1,092), strip of 11,896 rows in 6.7 s with /api/hosts p99 4.5 ms
read-pool.sh gate 3 (information, closes in v1.3.42.3) p99 31.0 ms, max 55.2 ms over 3 exports of 45,654 rows: the pinned-refresh tail as carried
event-storm.sh, first pull PASS: the community-blocklist pull at 14:59:47 UTC (15,000 added, 14,908 deleted) produced 0 event queue full lines; the panel wrote 2 log lines in the hour around it (0.01 MB per day against 21 before) and no WARN or ERROR line since the deploy (17 lines in the first hour, 5,500 per hour before). Tagged on this pull (operator's rule).
event-storm.sh, two pulls (closes the window) PASS at 18:00 UTC: second pull at 16:59:47 UTC (15,000 added, 14,929 deleted), 0 event queue full lines across the 2 pulls in the 4 h window since 14:00 UTC; the panel wrote 5 log lines (1,346 bytes, 0.01 MB per day against 21 before), none in the ten minutes around the second pull, no WARN or ERROR since the deploy.

Known issues carried

  • AppSec fails open under a scanner burst even at 1 CPU: 16 context deadline exceeded in the two minutes 08:10-08:11 UTC on 2026-09-28 while one external IP sent 353 requests to one host (v1.3.40 planning doc, 6.1). Data point for the bouncer stream pre-flight (v1.3.42.4): in stream mode the LAPI stops answering one lookup per proxied request (about 59,000 a day) and that CPU is AppSec's.
  • The monthly VACUUM (maybeVacuum: day 1, hour 04 UTC) is checked at boot and then on a 24 h ticker from the boot time, so it only fires when a tick lands inside hour 04 UTC, that is, when the panel (re)started between 04:00 and 04:59 UTC. With the current boot times it does not run on 2026-10-01. The passive sampler for that run is prepared in scripts/ops/vacuum-sample.sh (not installed); the schedule itself is the operator's decision.
  • From v1.3.42.1: the v1.3.41.1 export gate tail (until v1.3.42.3), Until shown as "expired" on imported decisions, blocked_requests_24h mixing, dashboard/cache_test timing flake. The 48 h DB-size reading is closed (v1.3.40 planning doc, section 10).