09 — Observability & SLOs
What we measure and why
The system has three kinds of signal, and they answer different questions:
| Signal | Question | Source of truth |
|---|---|---|
| Service health | Is it up and fast? | Logs, metrics [PLANNED] |
| Business truth | Did we bill correctly, did revenue flow? | Usage ledger, chain events |
| Agent quality | Are the signals any good? | Public scoreboard |
The third is the unusual one: agent output quality is measured publicly and adversarially, rather than by self-report.
Service level objectives
| Service | SLI | Target |
|---|---|---|
| Gateway (non-streaming) | p95 overhead added over upstream | < 150 ms |
| Gateway | Availability (excluding upstream faults) | 99.5% |
| Gateway | 5xx rate | < 0.5% |
| Entitlement resolution | p99 (cache hit) | < 10 ms |
Platform API /scoreboard |
p95 | < 500 ms |
| Agent chat | p95 time to first token | < 3 s |
| Signal pipeline | Completion rate | > 95% |
| Entitlement indexer | Chain lag | < 2 min |
Gateway overhead is measured excluding upstream time deliberately — it is the only part we control, and conflating it with model latency hides regressions.
Instrumentation
Today: structured logging across services; the usage ledger doubles as the billing audit trail; CI enforces tests and lint per repo.
[PLANNED] — in priority order:
- Prometheus metrics on the gateway — the single most valuable addition,
since it sits on the revenue path:
-
hunter_requests_total{model,tier,auth_method,status}-hunter_request_duration_seconds{model,phase}(gateway vs upstream) -hunter_credits_consumed_total{wallet_bucket,model}-hunter_quota_rejections_total{tier}-hunter_x402_payments_total{result}—settle_failedis the one to alert on, since it means service was rendered unpaid -hunter_entitlement_cache{hits,misses,failures} - Agent pipeline metrics: run duration per node, revision counts, approval rate, tool error rates by MCP server.
- Indexer metrics: blocks behind head, events processed, upsert failures.
- Request tracing with a correlation id propagated terminal → gateway → agent → MCP.
Alerting
| Alert | Condition | Why it pages |
|---|---|---|
| Gateway 5xx spike | Threshold set in the alerting config | Revenue path broken |
| All upstreams unhealthy | No healthy vLLM backend | Total inference outage |
| Settlement failures | Threshold set in the alerting config | Free service being rendered |
| Indexer lag | Threshold set in the alerting config | Paying users can't access their tier |
| Quota rejection spike | Threshold set in the alerting config | Either an attack or a pricing bug |
| Buyback keeper failure | Scheduled clip missed | Public commitment not honoured |
| Contract pause | Paused event |
Someone hit the emergency stop |
Business alerts (indexer lag, buyback misses) matter as much as technical ones here: a user who locked tokens and cannot use them is an outage, even though every service reports healthy.
Dashboards
- Revenue flywheel (public, built; pre-launch until contracts deploy) —
drpxbt.xyz/flywheel: burned, locked, staked, % off market, read live from Base. - Signal track record (public, API built) — accuracy overall and per asset; [PLANNED] terminal UI.
- Operations [PLANNED] — gateway throughput/latency/errors by model and tier, cluster GPU utilisation, queue depths.
- Business [PLANNED] — active tiers, active keys, credits consumed vs allowance, revenue routed on-chain, tier conversion.
Auditability
Three independent records let anyone reconstruct what happened:
- Usage ledger — append-only, per-request, the basis for any billing dispute.
- Chain events — every lock, subscription, distribution and burn, publicly verifiable and not editable by us.
- Signal record — calls recorded at publication time, before outcomes are known, so the accuracy number cannot be curated after the fact.