Problem
Two Slack channels received a mixed stream of production alerts — ~78% AWS CloudWatch (RDS replica lag, ALB unhealthy targets, EC2 status checks, Lambda throttles), ~18% Prometheus AlertManager (host CPU/OOM/disk), ~4% Grafana/Kubernetes — all posted by the same Slack bot, so nothing could be routed by sender. Every alert meant a manual investigation across CloudWatch, RDS logs, Grafana, and Loki.
The old process
Read the alert. Guess which system it came from. Open CloudWatch, or Grafana, or the RDS console. Pull the metric at the alert minute. Cross-reference logs. Decide whether it already self-resolved. Every alert was a fresh archaeology dig, and the knowledge lived only in the heads of whoever had seen that alarm before.
What I built
Mention @agent in any alert thread and a dispatcher fetches the thread context (including the structured alert payload hidden in message attachments), spawns a headless agent run guided by the channel's investigation runbook, and posts back a structured RCA: ROOT CAUSE / EVIDENCE / SEVERITY / RECURRING / NEXT STEPS.
How the investigation works
Content-signature routing
Step zero classifies the alert by payload signature — is now *ALARM* means CloudWatch, [FIRING:] means Prometheus, a grafana_folder label means Kubernetes — then routes to a resource specialist runbook (RDS, ALB, EC2, Lambda, Prometheus-host, or the K8s router).
The Performance Insights lesson
Replica-lag culprits are often invisible to slow-query logs: thousands of individually fast writes, each under the slow-query threshold, that in aggregate drive sustained write IOPS. So the RDS runbook always queries Performance Insights at the exact spike minute, grouped by user and by SQL — and applies a timing-alignment rule, verbatim:
“The slow-query culprit must still be running AT THE MINUTE lag spiked. If a batch finished 3+ minutes before the spike, it's a red herring.”— from my runbooks
Kubernetes without cluster access
The K8s side investigates entirely through the Grafana HTTP API datasource proxy — PromQL against the cluster's Prometheus, LogQL against Loki — so the agent needs no VPN, no kubeconfig, no direct cluster access. It distinguishes traffic spikes from memory leaks (CPU + memory both up = load; memory alone = leak), checks whether the alert already self-resolved before investigating, and embeds shareable Grafana Explore deep-links in every RCA so anyone can click through to the raw evidence.
Stale-alarm intelligence
Some alarms are artifacts — e.g. burst-balance alarms left over from a disk-type migration, on volumes that no longer emit the metric. The agent says “delete this alarm” instead of fabricating a cause.
Severity is topology-aware
The runbooks encode which replicas are on the production-critical path and which serve internal pipelines, so the agent doesn't over-escalate.
The engineering story that closes the sale — the tested router
The routing logic is mirrored into a pure-stdlib Python module with 13 real production alert payloads as regression fixtures — including the genuinely nasty ones: HTML-escaped comparison operators, British spellings, and one permanently misspelled alarm name. A replay harness asserts every fixture routes correctly; a bad runbook edit fails the test suite instead of surfacing as a wrong RCA in production.
On its first run it caught a real latent bug — a regex that truncated ALB target-group ARNs at the first slash, which would have silently broken every ALB metric lookup.
“Prompt engineering becomes real engineering when it has fixtures.”— the thesis
$ python -m routing.replay --fixtures tests/fixtures/*.json [ok] rds-replica-lag.json → rds_specialist [ok] alb-unhealthy-html.json → alb_specialist (HTML-escaped '>=') [ok] prom-host-oom.json → prometheus_host [ok] grafana-k8s-memory.json → k8s_router ... 13 passed, 0 failed
Production posture
- Separate, deeper read-only IAM policy — adds RDS log download, Performance Insights, and CloudWatch Logs Insights, still zero write actions.
- The agent is forbidden from posting to Slack itself — the dispatcher owns posting, so no duplicate or wrong-channel messages.
- Concurrency capped at two simultaneous investigations.
- A 5-minute staleness guard against CloudWatch alarm-history propagation lag.
Outcome
Typical human investigation time before: [[METRIC — CONFIRM]]. After: a 2–4 minute agent turnaround, with evidence attached.