Observability

SCRUBR emits OpenTelemetry (traces, metrics, logs) over OTLP/gRPC, exposes a Prometheus scrape endpoint, and writes structured logs. Everything is off by default — you get only human-readable stderr logs until you opt in via the telemetry config block.

Exporters are best-effort: a down or absent collector never blocks or fails a request — signals are buffered and dropped on the floor rather than backpressuring the data path.

flowchart LR
    subgraph SCRUBR
        R["request path"]
        ADM["admin listener<br/>:9464"]
    end
    R -->|"stderr · text/json"| L["local logs"]
    R -->|"traces · metrics · logs<br/>OTLP/gRPC · best-effort"| COL[["OTEL Collector<br/>:4317"]]
    ADM -->|"GET /metrics · scrape"| PROM[["Prometheus"]]
    COL --> BK[["observability backend"]]

The entire OTLP + Prometheus stack sits behind a default-on otel Cargo feature. cargo build --no-default-features drops OTLP and Prometheus entirely, keeping only local stderr logging and file-log rotation.

The telemetry config block

All fields live under telemetry: and default to off/empty.

A full-featured block:

telemetry:
  service_name: scrubr
  log_format: json
  resource_attributes:
    deployment.environment: prod
    service.version: "1.0.0"
  otlp:
    endpoint: https://otlp.example.com:4317
    headers:
      authorization: "Bearer <token>"   # SaaS backends: Honeycomb, Grafana Cloud, …
    timeout_secs: 10
  traces:
    enabled: true
    sample_ratio: 0.1
  metrics:
    otlp: true
    interval_secs: 15
    prometheus:
      enabled: true
      listen: "0.0.0.0:9464"
  logs:
    otlp: true

Metrics

• scrubr.request.duration times request receipt → first response byte; scrubr.upstream.duration is the upstream round-trip.
• Every Prometheus series also carries otel_scope_name="scrubr".

Scrape the admin listener:

curl -s http://127.0.0.1:9464/metrics

# HELP scrubr_requests_total proxied requests
# TYPE scrubr_requests_total counter
scrubr_requests_total{otel_scope_name="scrubr",route="openai",tenant="acme",mode="enforce",status="200"} 1274
# HELP scrubr_request_duration_seconds request receipt to first response byte
# TYPE scrubr_request_duration_seconds histogram
scrubr_request_duration_seconds_bucket{otel_scope_name="scrubr",route="openai",mode="enforce",le="0.25"} 1180
scrubr_request_duration_seconds_sum{otel_scope_name="scrubr",route="openai",mode="enforce"} 118.42
scrubr_request_duration_seconds_count{otel_scope_name="scrubr",route="openai",mode="enforce"} 1274

Admin endpoints

When metrics.prometheus.enabled is set, SCRUBR starts a small admin HTTP listener (metrics.prometheus.listen, default 127.0.0.1:9464) separate from the proxy port, so scraping never rides the data path:

/healthz and /readyz are also served on the main proxy port, so load balancers and Kubernetes probes can reach them without exposing the admin listener.

Traces

With traces.enabled, SCRUBR emits one span per proxied request, named scrubr.request, with attributes:

Spans are exported via OTLP/gRPC to otlp.endpoint. Head sampling is parent-based on traces.sample_ratio (1.0 keeps everything; 0.1 keeps ~10% of root traces while honoring an incoming sampling decision).

Logs

Structured stderr logs are always on — text or json per log_format — and never contain secret values. Set the level filter with RUST_LOG (e.g. RUST_LOG=scrubr=info).

When logs.otlp is set (and otlp.endpoint is configured), the same logs are also exported via the OTEL logs signal over OTLP, alongside stderr.

Exporting transaction inputs and outputs (audit)

The transactions block records the provider-facing request + response bodies to a local file (created 0600). With transactions.export.otlp_logs: true, each transaction is also emitted as an OTEL log record (event name scrubr.transaction) for centralized audit and retention, with attributes: request_id, route, tenant, method, path, status, mode, detected.

Whether the bodies themselves are attached is governed by transactions.export.include_bodies:

transactions:
  enabled: true
  path: /var/log/scrubr/transactions.jsonl
  export:
    otlp_logs: true              # also emit each transaction as an OTEL log record
    include_bodies: enforce-only # never | enforce-only | always

Requires telemetry.otlp.endpoint and telemetry.logs.otlp.

Dry-run exports the original, unmasked payload. In dry-run mode nothing is masked, so a transaction body is the caller's original content. With include_bodies: always those plaintext secrets are shipped to your OTLP backend. Keep the default enforce-only (or never) unless the sink is trusted and in-network; SCRUBR warns loudly at startup whenever always is configured.

Log rotation ("chunking")

Both file logs — audit and transactions — support in-process rotation via a rotate sub-block:

Rolled segments are named {stem}-{unixsecs}-{seq}.{ext} (e.g. transactions-1719960000-0.jsonl) — chronologically sortable — and created 0600.

audit:
  enabled: true
  path: /var/log/scrubr/audit.jsonl
  rotate:
    max_bytes: 104857600   # 100 MiB
    daily: true
    max_files: 0           # keep all — history stays verifiable

transactions:
  enabled: true
  path: /var/log/scrubr/transactions.jsonl
  rotate:
    max_bytes: 104857600   # 100 MiB
    max_files: 10          # size × max_files bounds disk

Because rotation is in-process, there is no lost-file-descriptor race like the one an external logrotate rename introduces. (External logrotate still works if you prefer it — use copytruncate.)

The audit log's tamper-evident hash chain carries across segments:

scrubr audit-verify /var/log/scrubr/audit.jsonl

verifies the whole chain — all rolled segments in order, then the active file — and deleting an entire segment is detected. Keep max_files: 0 (keep-all) for the audit log so the full history stays verifiable; for the transaction log, max_bytes together with max_files bounds disk usage.

Environment overrides

Graceful shutdown

On SIGINT or SIGTERM, SCRUBR drains in-flight requests and then flushes and shuts down all exporters, so the final spans, metrics, and logs are delivered before exit.

Quick start with an OTEL Collector

Run a collector locally and point SCRUBR at it. First, a minimal collector that accepts OTLP on :4317 and prints everything it receives:

# otel-collector.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
exporters:
  debug:
    verbosity: detailed
service:
  pipelines:
    traces:  { receivers: [otlp], exporters: [debug] }
    metrics: { receivers: [otlp], exporters: [debug] }
    logs:    { receivers: [otlp], exporters: [debug] }

# docker-compose.yml
services:
  otel-collector:
    image: otel/opentelemetry-collector:latest
    command: ["--config=/etc/otelcol/config.yaml"]
    volumes:
      - ./otel-collector.yaml:/etc/otelcol/config.yaml:ro
    ports:
      - "4317:4317"   # OTLP/gRPC

Then enable traces, metrics (OTLP + Prometheus), and logs in SCRUBR:

telemetry:
  service_name: scrubr
  log_format: json
  otlp:
    endpoint: http://localhost:4317   # otel-collector:4317 if SCRUBR shares the compose network
  traces:
    enabled: true
    sample_ratio: 1.0
  metrics:
    otlp: true
    prometheus:
      enabled: true
      listen: "0.0.0.0:9464"
  logs:
    otlp: true

Bring it up with docker compose up, send traffic through the proxy, and watch spans, metrics, and logs land in the collector's output; scrape SCRUBR at http://localhost:9464/metrics in parallel. A ready-to-run example lives under examples/.

See also

• CONFIGURATION.md — reference for the telemetry, audit, and transactions blocks.
• DEPLOYMENT.md → Kubernetes — health probes and metrics in a cluster.