Skip to content

v1.3.40.0 -- Ingest filter, retention per source, raw window

First release of v1.3.40 (PHASE 0: docs/planning/v1.3.40-log-pipeline-phase0.md). No schema change; nothing in the blocking path. argosVersion 1.3.39.1 -> 1.3.40.0.

What ships

  • Ingest filter (logs.IngestFilter): rules from three settings (logs.ingest.drop_loggers, drop_user_agents, drop_paths), defaults for the reverse-proxy health checker's routine lines and the uptime monitors. Placed between the notification watcher and the writer: the watcher keeps seeing every line. Audit and WAF rows are never filtered. Per-rule, per-day counters, persisted once a minute (logs.ingest.dropped), shown on the Logs page and in Settings. Writing a logs.ingest.* setting reloads the filter.
  • Retention policy (logs.RetentionPolicy, db.PurgeWithPolicy): days per source, raw = '' on access rows older than raw_hours behind a watermark (logs.retention.raw_watermark), the cap checked through MAX(id)-MIN(id)+1 with COUNT(*) only past the bound. The purge log line reports removed, raw_stripped, cap_bound and cap_counted.
  • Honest degradation where raw is gone: raw_stripped / raw_note on the entry, notes[] on a list with q, the drawer and the search box show them.
  • Settings > Logs: days per source, raw hours, cap, the three drop lists, today's excluded counters and the size estimate from GET /api/logs/pipeline.

Readers of raw, inventoried

Reader Effect of the 24 h raw window
Entry drawer (GET /api/logs/{id}) raw_stripped + note; columns shown
Free-text q (GET /api/logs) note when the range reaches older rows; path, user agent, message searched everywhere
CSV export never carried raw; unchanged
Notification watcher (extractHostPort(e.Raw ...)) ingest time only; raw always present there
WAF parser ingest time only
SSE live tail fresh rows; raw present
Backups smaller file

Measured

Before on prod (1.3.39.1), logs-pipeline.sh: 501,475 rows, 1,720,086,528 B, 661.5 MB of access raw; last full day 67,130 access + 3,378 error rows; 450 monitor rows and 64 health-checker rows stored in the first 20 min after a restart; POST /api/logs/purge 1.51 s with /api/hosts p99 1,246 ms (one request queued behind the purge on the single connection).

Expected after (from the PHASE 0 numbers): about 48,000 rows/day stored, raw only on the newest 24 h (about 70 MB), table around 250-300 MB after the first full 7-day cycle. Real numbers at 48 h in the after-smoke note.

After on prod: pending deploy.

Smoke gate

scripts/smoke/logs-pipeline.sh: filtered families stored since the panel start <= 100; raw empty on access rows older than raw_hours + 1 h (PENDING until the panel is 25 h old); rows and DB size printed for the 48 h comparison; p99 of /api/hosts during a purge <= 50 ms.

Gates before commit: gofmt -l clean, go vet ./..., go test ./..., npm run build, npm run lint, check-no-personal-data.sh, mkdocs build --strict.

Files changed

  • backend/internal/logs/pipeline.go (new, + pipeline_test.go), ingestor.go, retention.go.
  • backend/internal/db/logs.go (+ logs_purge_policy_test.go).
  • backend/internal/api/logs.go, settings.go, handlers.go, backend/internal/server/server.go, backend/cmd/argos/main.go.
  • frontend/src/api/client.ts, pages/Logs.tsx, pages/Settings.tsx.
  • scripts/smoke/logs-pipeline.sh, docs/reference/api.md, docs/features/logs-browser.md, docs/features/settings.md, docs/architecture/storage.md, CHANGELOG.md, docs/release-notes/v1.3.40.0.md, mkdocs.yml, docs/operations/verification-report.md, frontend/package.json.