Skip to content

M35 solution: Operations Support, going deeper

Five small, independent mini-labs, one per topic that rounds out operations support. Each extends a module you already built. Offline, deterministic, dependency-free, no API key.

Files

File Extends What it shows
structured_logging.py M20 logs as queryable JSON records that share a request_id; query() finds every failed retrieval, correlate() pulls one request's whole story
dashboard.py M20 + M31 the four golden signals (latency, traffic, errors, saturation) + SLO burn from a window of requests, with breach flags
online_eval.py M26 + M30 sample live traffic, score it reference-free, and detect quality drift the fixed offline gate never anticipated
rate_limit.py M25 + M29 a token-bucket rate limit, a per-tenant quota, and a concurrency limit (the three capacity levers)
improvement.py M30 + M31 + M26 the reliability flywheel: incidents become regression guards, and repeats get prevented over time
demo.py — runs all five back to back
../starters/extend.py — your turn: a composite alert that only pages when ≥2 signals breach at once

Run it

python demo.py            # all five
# or each on its own:
python structured_logging.py
python dashboard.py
python online_eval.py
python rate_limit.py
python improvement.py
No key, no .env; pure operations logic.

Verified (offline, deterministic)

  • structured_logging: correlate("req-B") returns the failing request's three lines (request → tool error → 503); query(event="tool", tool="retrieve", error=True) finds exactly the one failed retrieval.
  • dashboard: the sample window flags latency p95, error rate, saturation, and SLO burn as BREACH.
  • online_eval: the healthy half scores avg 1.0 (no drift); the full stream scores avg 0.6 → drift detected from a 10-of-50 sample, while the offline gate would have missed it.
  • rate_limit: 8 requests at t=0 → 5 allowed + 3×429; one at t=3 allowed (refill); quota 3/day → 4th over-quota; concurrency 2 → 3rd rejected.
  • improvement: over four weeks, new incidents fall (3 → 1 → 0 → 0) while repeats are prevented and the guard count plateaus at 4, the flywheel.