Remote Log Shipper (rwarden push)

The rwarden push command is the core data shipper for distributed Routewarden deployments. It tails your reverse proxy containers or log files, batches events in an in-memory ring buffer, encrypts them using X25519 ECDH + AES-256-GCM, and streams them to the central dashboard.

---

1. Zero-Flag Execution

Once registered, you can start streaming by simply running:

rwarden pushbash

# Zero-flag execution (reads ~/.routewarden/agent.json and auto-discovers containers)rwarden push# Run in background (detached daemon mode)rwarden push -d# orrwarden push --detach

●READONLY│UTF-8│BASH

What Happens Automatically:

1. Profile Loading: Reads ~/.routewarden/agent.json to load your dashboard URL, agent authentication token, and server public key.
2. Auto-Discovery: Automatically inspects the Docker daemon via /var/run/docker.sock and tails all running containers labeled routewarden=true or routewarden.logs=true.
3. Heartbeat Startup: Launches a background routine sending health heartbeats every 30 seconds to the central dashboard.
4. Encrypted Batching: Collects log events into an in-memory ring buffer (up to 2,000 events) and flushes encrypted batches every 1 second or whenever 50 events accumulate.

Startup Output

Shipper Startup Outputplaintext

🚀 Routewarden Remote Shipper Starting...─────────────────────────────────────────────────────────────  Dashboard URL:   http://198.51.100.10:7890  Agent Name:      edge-frankfurt-01  Encryption:      🔒 End-to-End Encrypted (X25519 + AES-256-GCM)  Batch Settings:  Size=50, Interval=1s  Offline Queue:   Max=10000, Timeout=5m0s  Auto-Discovery:  Watching Docker containers with label 'routewarden=true'─────────────────────────────────────────────────────────────[15:04:05] 🟢 Connected to Routewarden dashboard (http://198.51.100.10:7890)Press Ctrl+C to stop.🐳 [Docker] Attached to container: edge-traefik (4b7e89a1c2d3)

●READONLY│UTF-8│PLAINTEXT

---

2. Modes of Operation

Mode A: Auto-Discovery (Default)

Tails all gateway containers dynamically as they start and stop:

# Auto-discover all containers labeled routewarden=truerwarden push --docker# Or with custom discovery labelrwarden push --label "security.monitor=true"

●READONLY│UTF-8│BASH

Mode B: Tail Specific Container(s)

Tail one or more target containers by name or ID using repeatable -c or comma-separated names:

rwarden push --containerbash

# Tail specific container by namerwarden push --container traefik# Or using shorthand alias (-c)rwarden push -c edge-gateway# Tail multiple specific containers (repeatable or comma-separated)rwarden push -c traefik -c api-gatewayrwarden push -c "traefik,api-gateway,auth-service"

●READONLY│UTF-8│BASH

Mode C: Monitor Multiple Log Files

Stream events from host log files using repeatable flags (-f), comma-separated paths, glob patterns (*.log), or positional arguments:

rwarden push --filebash

# Stream events from a single access log filerwarden push --file /var/log/nginx/access.log# Stream multiple files using repeatable flags (-f / --file)rwarden push -f /var/log/nginx/access.log -f /var/log/caddy/access.log# Comma-separated pathsrwarden push --file /var/log/nginx/access.log,/var/log/traefik/access.log# Glob pattern wildcard expansion (all matching files)rwarden push --file "/var/log/nginx/*.log"# Positional file argumentsrwarden push /var/log/nginx/access.log /var/log/auth.log

●READONLY│UTF-8│BASH

rwarden push uses non-blocking tailing (tail -F), ensuring that log rotations (logrotate) are handled seamlessly without losing telemetry across multiple log files.

Mode D: Concurrent Hybrid Streaming (Files + Docker)

Stream multiple log files and Docker containers simultaneously. Both files and Docker streams feed into the unified in-memory ring buffer for encrypted shipping:

# Monitor multiple log files AND auto-discover Docker containers concurrentlyrwarden push --docker -f /var/log/nginx/access.log -f /var/log/syslog# Tail specific container(s) AND multiple log files simultaneouslyrwarden push -c custom-gateway,api-proxy -f /var/log/auth.log# Full hybrid: labeled Docker auto-discovery + specific container + wildcard filesrwarden push --docker -c auth-service -f "/var/log/nginx/*.log"

●READONLY│UTF-8│BASH

---

3. In-Memory Ring Buffer & Offline Queueing

Unlike traditional shippers that write heavy WAL journals to disk, rwarden push uses zero-disk, memory-buffered streaming:

• Collector Ring Buffer: Ingests up to 2,000 events from log streams and Docker pipes before backpressure throttling (~2MB RAM).
• Offline Event Queue (--queue-size): In-memory buffer holding up to 10,000 events (configurable) when the dashboard is unreachable, restarting, or disconnected.
• Edge Storage Protection: Prevents millions of daily disk write cycles, preserving SSD life on edge hardware, Raspberry Pis, and low-cost VPS instances.
• Stale Event Protection (--disconnect-timeout): If the dashboard remains disconnected longer than 5 minutes (default), stale events are gradually pruned with throttled warnings to protect node memory from unbounded growth.

---

4. Tuning Batching & Flush Intervals

You can tune batching parameters to match high-traffic or low-latency environments:

Batching Flagsbash

# High-volume cluster: larger batch sizerwarden push --batch-size 200 --flush-interval 2s# Low-latency testing: flush immediatelyrwarden push --batch-size 10 --flush-interval 500ms

●READONLY│UTF-8│BASH

Parameter	Default	Description
--batch-size	50	Maximum number of events to buffer before triggering an immediate flush.
--flush-interval	1s	Maximum time to wait before flushing partial batches to the server.
--queue-size	10000	Maximum events to queue in memory while disconnected from dashboard.
--disconnect-timeout	5m	Duration disconnected from dashboard before dropping stale events to protect memory.
--ping-interval	10s	Interval between connection health pings when connected.
--tail	50	Number of existing lines to replay when attaching to a container or file.
--status, -s	""	Forward HTTP requests matching status codes or ranges (e.g. '400-599, 301', '4xx, 5xx').
--include-failed, -F	false	Legacy alias: forward all failed HTTP requests (status >= 400, equivalent to --status 400-599).
--all-traffic	false	Ship all HTTP access traffic including successful 200 OK responses.
--all-logs	false	Ship all raw container lines unconditionally without Routewarden filtering.

---

5. Telemetry Filtering & Shipping Modes

By default, rwarden push optimizes edge bandwidth and central dashboard storage by shipping only security events:

• Explicit Routewarden plugin actions (BLOCK, TARPIT, HONEYPOT, threat rule matches).
• Security status codes (403 Forbidden, 429 Too Many Requests).
• Successful 200 OK access logs are automatically filtered out.

You can customize telemetry filtering using flags, environment variables, or agent.json:

Filtering Flagsbash

# Default: push only security events (BLOCK, TARPIT, HONEYPOT, 403, 429)rwarden push --docker# 1. Independent: Ship requests matching explicit status codes or ranges (e.g. 400-599, 301, 4xx, 5xx)rwarden push --status "400-599, 301"# Shorthand:rwarden push -s 400-599,301# 2. Independent: Ship failed HTTP requests (status >= 400, legacy alias)rwarden push --include-failed# 3. Independent: Ship all HTTP access traffic including successful 200 OK responsesrwarden push --all-traffic# 4. Layer 4 (UDP & TCP) Traffic Logs: Stream tcp-warden security events or all sessionsrwarden push -f /var/log/routewarden/tcp-warden.jsonlrwarden push -c tcp-warden --all-traffic# 5. Raw unfiltered logs (bypass all filters)rwarden push --all-logs

●READONLY│UTF-8│BASH

Available Modes:

1. Security Events Only (Default):

   • Forwards threat mitigations and security rejections (verdict, action, rule, matched_pattern, or status 403/429).
   • Drops benign access traffic, keeping the dashboard focused on actionable incidents while reducing database size.

2. Explicit Status Code Ranges (--status / -s):

   • Forwards reverse proxy HTTP requests matching specific status codes, ranges, or wildcard families (e.g. --status "400-599, 301", -s 4xx,5xx, -s 404,500-504).
   • Works across Caddy (logger: "http.log.access"), Traefik (JSON / Combined), and NGINX (JSON / Combined access logs).
   • Unblocked requests with status $\ge 400$ are categorized as FAILED on the central dashboard (with rule HTTP <status>), while requests $< 400$ (such as 301 or 302 redirects) are categorized as ALLOW.
   • Also configurable via environment variables RW_STATUS="400-599, 301" or "status_codes": "400-599, 301" in agent.json.
   • (Legacy Alias): --include-failed / -F (or RW_INCLUDE_FAILED=true) is fully supported as shorthand for --status 400-599.

3. Full HTTP Access Traffic (--all-traffic):

   • Forwards all reverse proxy access logs including clean 200 OK traffic.
   • Useful when centralizing full access auditing into the dashboard.
   • Also configurable via RW_ALL_TRAFFIC=true or "all_traffic": true in agent.json.

4. Raw Unfiltered Logs (--all-logs):

   • Disables all filters and forwards every raw stdout/stderr line emitted by the monitored container or file.
   • Configurable via RW_ALL_LOGS=true or "all_logs": true in agent.json.

Flag Independence & Resolution Precedence

Can --status, --include-failed, and --all-traffic be used independently? Yes. Each flag controls an independent telemetry tier and can be specified alone:

• rwarden push --status "400-599, 301": Matches explicit status ranges and security events.
• rwarden push --include-failed: Matches all failed requests ($4xx, 5xx$) and security events.
• rwarden push --all-traffic: Matches all HTTP requests ($2xx, 3xx, 4xx, 5xx$) and security events.

When multiple options are present simultaneously (via flags, environment variables, or agent.json), the shipper evaluates them using the following precedence hierarchy:

Priority	Setting	Resolution Behavior
1 (Highest)	Security Signals	Events with explicit verdicts (BLOCK, TARPIT, HONEYPOT), actions (blocked, banned), rules, or status 403/429 always pass, regardless of other flags.
2	--all-logs	Completely bypasses the filter engine and streams all raw lines.
3	--all-traffic	Forwards all HTTP traffic. Overrides both --status and --include-failed.
4	--status (-s)	Matches specific codes/ranges. Overrides --include-failed if both are specified.
5	--include-failed (-F)	Matches all HTTP status $\ge 400$. Acts as fallback when --status is not set.
6 (Default)	None	Only explicit security events are forwarded.

---

Handling Layer 4 (UDP & TCP) Traffic Logs

In addition to Layer 7 HTTP reverse proxies, RouteWarden includes a native Layer 4 proxy engine (tcp-warden / rwarden tcp).

UDP Traffic Inspection & Logging

When running services with transport: udp or transport: both in tcp-warden.yaml, every inbound datagram evaluates through security stages:

1. IP Banlist: Active bans trigger immediate drops (banned: <reason>).
2. CrowdSec LAPI: Bouncer checks trigger datagram drops (crowdsec_ban: <scenario>).
3. CIDR Network Deny: Subnet matches drop datagrams (ip_denied).
4. GeoIP Filtering: Country blocks drop packets (country_blocked: <code>).
5. Rate Limiting: Datagram bursts exceeding connections_per_minute are dropped (rate_limit_exceeded).
6. MaxSessions Guard: Active sessions exceeding udp.max_sessions are dropped (udp_max_sessions_exceeded).
7. Protocol UDPInspector: Plugins (e.g. DNS or QUIC) inspecting datagram payloads can issue UDPVerdictDrop or UDPVerdictReject.

Decisions are written as structured JSON lines to /var/log/routewarden/tcp-warden.jsonl (or container stdout):

{
  "type": "security_event",
  "timestamp": "2026-10-24T06:00:00Z",
  "level": "warn",
  "plugin": "tcp-warden",
  "service": "dns_service",
  "protocol": "udp",
  "client_ip": "198.51.100.22",
  "action": "blocked",
  "reason": "rate_limit_exceeded",
  "bytes_in": 78,
  "bytes_out": 0
}

Shipping UDP Logs with rwarden push

• Security Events (Default): All blocked or banned UDP events carry "plugin": "tcp-warden" or "action": "blocked", so they are automatically detected as security events and forwarded to the central dashboard without any special flags:

  # Stream UDP security events from log file
  rwarden push -f /var/log/routewarden/tcp-warden.jsonl
  
  # Or stream from a Docker container
  rwarden push -c tcp-warden

• Forwarding All UDP Datagram Sessions: To capture successful/allowed UDP traffic (action: "allowed", forwarded datagram metrics), enable --all-traffic:

  rwarden push -f /var/log/routewarden/tcp-warden.jsonl --all-traffic

• Piping from systemd / journald:

  journalctl -u tcp-warden -f -o cat | rwarden push -f -

---

6. Resilience, Active Health Supervisor & Auto-Reconnect

rwarden push includes robust network failure recovery and zero-event-loss failover:

1. Active Connection Health Supervisor (/api/v1/ping):
   • When connected, sends health pings every --ping-interval (10s default) to /api/v1/ping. This refreshes the agent's last-seen status on the dashboard and verifies end-to-end reachability.
   • When disconnected, automatically switches to fast-polling mode (2-second interval) to detect when the dashboard returns online immediately after reboots.
2. Zero-Drop Offline Queueing & Backlog Drain:
   • If the dashboard server restarts or connectivity drops, events are seamlessly held in the offline queue buffer.
   • Upon reconnection, rwarden push detects the online state, logs 🚀 Dashboard reconnected! Flushing <N> queued offline event(s)..., and instantly drains the backlog in batches to the dashboard without dropping telemetry.
3. Encryption Key Rotation Auto-Recovery:
   • If the dashboard restarts and rotates or regenerates its encryption keypair, the client automatically receives the new public key, re-encrypts the current payload, updates its cached server key, and retries the batch immediately.
4. Exponential Backoff: If the central dashboard is temporarily unreachable, sendBatchWithRetry retries in-flight requests with backoff before queueing them offline.
5. Graceful Shutdown: Listens for SIGINT (Ctrl+C) and SIGTERM, flushes remaining buffered events, and terminates cleanly.

---

7. One-Off Push (Without Saved Config)

For ephemeral test environments or CI/CD pipelines, you can push directly by supplying flags without running rwarden register:

Direct Push Flagsbash

# Push directly without pre-saving configrwarden push   --url http://198.51.100.10:7890   --key rw_sec_8f9a2b4c1e0d3f7a...   --docker -f /var/log/nginx/access.log

●READONLY│UTF-8│BASH

---

8. Background Daemon Execution (-d, status, stop)

Run the shipper as an unprivileged, detached background daemon without needing systemd or nohup:

Daemon Managementbash

# Launch shipper in detached background moderwarden push -d# Inspect status of running shipper daemonrwarden push status# orrwarden status push# Stop running background shipper daemonrwarden push stop# orrwarden stop push

●READONLY│UTF-8│BASH

How Background Mode Operates:

• Process ID Tracking: The daemon writes its process ID to ~/.routewarden/push.pid.
• Detached Log Redirection: All diagnostic outputs, gateway attachments, and connection logs are safely redirected to ~/.routewarden/push.log.
• Status & Health: Running rwarden push status (or rwarden status push) reads the PID file and confirms that the background process is alive and active.
• Graceful Teardown: Running rwarden push stop (or rwarden stop push) delivers a graceful termination signal, flushes remaining telemetry in memory, and cleans up the PID file.

---

9. Ultra-Lean Micro-Container (ghcr.io/routewarden/shipper:latest)

For Docker and containerized edge hosts, Routewarden publishes a dedicated, stripped-down shipper container image:

services:  routewarden-shipper:    image: ghcr.io/routewarden/shipper:latest    container_name: routewarden-shipper    restart: unless-stopped    environment:      - RW_DASHBOARD_URL=https://soc.example.com      - RW_AGENT_KEY=rw_sec_your_agent_key_here      - TZ=Asia/Kolkata    volumes:      # Auto-discover Docker containers with label routewarden=true      - /var/run/docker.sock:/var/run/docker.sock:ro

●READONLY│UTF-8│YAML

Zero-Touch Environment Variables (Recommended)

When deploying via Docker or Docker Compose, you do not need to run rwarden register on the host or mount configuration files. Pass RW_DASHBOARD_URL and RW_AGENT_KEY directly as environment variables:

Environment Variable	Shorthand Alias	Description	Example
RW_DASHBOARD_URL	RW_URL	Central Routewarden dashboard address	https://soc.example.com or http://198.51.100.10:7890
RW_AGENT_KEY	RW_KEY	24-byte secret agent token created via rwarden keys create	rw_sec_a8f9c2d1e4b7...
RW_STATUS	RW_STATUS_CODES	Forward HTTP calls matching status codes or ranges across Caddy, Traefik, and NGINX	400-599, 301 or 4xx, 5xx
RW_INCLUDE_FAILED	—	Legacy alias: ship all failed HTTP calls (status >= 400)	true or false
RW_ALL_TRAFFIC	—	Ship all HTTP access traffic including 200 OK responses	true or false
RW_ALL_LOGS	—	Ship all raw container lines without Routewarden filtering	true or false
TZ	—	IANA Time Zone identifier for log timestamp normalization	Asia/Kolkata, UTC, America/New_York

TIP

Dynamic Public Key Auto-Discovery: When RW_DASHBOARD_URL and RW_AGENT_KEY are provided, rwarden-shipper automatically queries /api/v1/pubkey from the central dashboard on startup to derive the ephemeral shared secret (X25519 ECDH + AES-256-GCM). Full asymmetric end-to-end encryption is established immediately without any manual key exchanges or mounted config volumes.

Deployment Options:

1. Zero-Touch Environment Variables: Ideal for Docker Compose and automated cluster rollouts. Set RW_DASHBOARD_URL and RW_AGENT_KEY in environment:, mount /var/run/docker.sock (or host log directories), and the shipper connects automatically.
2. Mounted Host Profile: If the node has already been enrolled via rwarden register, mount the host directory -v ~/.routewarden:/root/.routewarden:ro to reuse the saved agent identity.

Container Highlights:

• Binary Size: Only ~6.2 MB (stripped of all SQLite, embedded Web UI assets, HTML/JS/CSS, and MaxMind GeoIP code).
• Multiple Ingestion Modes: Stream multiple host access log files (-f /var/log/nginx/access.log -f /var/log/caddy/access.log), auto-discover labeled Docker containers (--docker), or run both concurrently in hybrid mode.
• Zero Configuration: Defaults to rwarden push --docker out-of-the-box when run without command flags.
• Memory Footprint: Consumes under ~10 MB RAM on low-cost VPS instances and IoT edge gateways.