Skip to content

Monitoring

Four sources of operational truth, in rough order of usefulness.

1. The Discord channel itself

The single best operational signal. The bot posts:

  • Booker successes: ✅ Booked recon #N to e-Fin entry #X
  • HITL prompts (one per gated row)
  • Errors that surface to the operator (e.g. ❌ e-Fin rejected client creation: ...)

If the Discord channel is quiet for >24h, either nothing happened or the bot crashed. Either way, look at logs.

2. Logfire / LGTM stack

All three containers send OTLP traces + logs to the shared LGTM stack at prod-lgtm-otelcol:4318/v1/traces.

Access:

  • Grafana — https://grafana.657656.xyz (TFA-gated). Loki for logs, Tempo for traces, Mimir for metrics.
  • Direct logs — docker logs prod-accounting-bot, docker logs prod-accounting-webhooks, docker logs prod-accounting-booker.

Notable log lines worth searching for:

  • enablebanking poll: sessions=N inserted=M skipped=K (every 6h, webhooks container)
  • re-match: recon #N -> doc #M (score=0.59) (matcher promoting an unmatched row)
  • booker: recon #N vat_decision rate='X' confidence=Y rules=[ids] (every successful book)
  • booker: recon #N -- registered purchase_invoice #X (every successful book)

3. Sentry

Captures unhandled exceptions across all three containers. Each release tagged with the image SHA so we can group by deploy. Most issues that warrant a Sentry alert also surface to Discord as a ❌ message; Sentry is the durable record.

4. Turso queries

When something gets confusing, look at the data directly:

-- How many recons are sitting unmatched?
SELECT COUNT(*) FROM reconciliations
WHERE status = 'unmatched';

-- How many bookings are pending HITL?
SELECT COUNT(*) FROM agent_checkpoints
WHERE status = 'pending' AND kind IN
  ('booker_approve', 'client_create_approval');

-- Bookings stuck in the gap between 'booked' and registered?
SELECT id, source_reference, occurred_on, amount, document_id
FROM reconciliations
WHERE status = 'booked' AND efin_entry_id IS NULL
ORDER BY decided_at DESC;

The last query is the canonical "what's the booker working on?" view. Empty = no work. Same row appearing for >1h = booker is stuck on something; check Sentry / logs.

What's NOT monitored

  • e-Fin's own uptime. RIK runs it; we trust it.
  • Enable Banking's uptime. Polls catch transient failures via the retry layer; persistent failures surface as failed_sessions > 0 in the poll summary.
  • Cloudflare R2. Sentry surfaces upload/download errors.

Alert routing

Sentry → Discord (a dedicated alerts channel separate from the agent's operator channel) via Sentry's Discord integration. No PagerDuty, no on-call rotation — this is a single-operator system. If something breaks at 3am it can wait until morning.