Problem
Automated cost alerts told the team that a service breached its daily threshold, never why. Each alert cost an engineer 30–60 minutes in Cost Explorer — clicking through grouping dimensions and cross-referencing CloudWatch — before anyone could say what actually changed.
The old process
Open Cost Explorer. Group by service, then usage type, then operation, then instance type, then purchase type. Notice the number moved. Switch to CloudWatch. Try to line up the spike with a metric. Forty-five minutes later, announce that it was a Savings Plan utilization shift — not runaway spend — and close the thread.
What I built
A Slack-native bot that receives the cost-alert webhook, acknowledges within Slack's 3-second window, then in a background worker pulls — in parallel — 7 days of Cost Explorer data across five grouping dimensions (usage type, operation, instance type, purchase type, linked account) plus the Name cost-allocation tag (the single best signal for naming the culprit resource), plus CloudWatch metrics resolved through a generic service map covering ~18 AWS services. It hands the raw JSON to Claude under a strict analysis contract and posts the RCA as a threaded reply, in 2–4 minutes.
How it works
Key design decisions
No hardcoded service branches
Verbatim from the code header:
“Python collects raw AWS data for any service. Claude performs all analysis. No hardcoded service branches — works for any AWS service.”— from the code header
Supporting a new service costs one line in a metrics map — or nothing.
Diff-based output, enforced
The output contract forbids inventory dumps: one line per finding, drop anything flat or decreasing, formatted like ↑ r7i.xlarge: 144h/day → 161h (Δ+17h, +$5.15, +11.5%). If nothing crosses the statistical bar, the agent says “No single driver — threshold may need raising” in one line instead of padding.
Statistical outlier detection, not naive thresholds
Standard-deviation-based flagging separates a genuine change (this instance type ran 144 hours every day, then suddenly 161) from normal volatility (this type varies 5–24h/day; a jump means nothing).
Domain gotchas encoded
Cost Explorer data for yesterday isn't settled, so the agent analyzes D-2. The 1st-of-month Reserved Instance upfront charge is expected, not an anomaly. Savings-Plan utilization shifts look like cost increases but aren't runaway spend — the agent shows the purchase-type split to prove it:
Prev day: On-Demand 1,308h $125.77 | Savings Plan 165h $17.28 | Spot 48h $1.23 → $144.28 Alert day: On-Demand 1,288h $124.91 | Savings Plan 267h $27.65 | Spot 57h $1.51 → $154.06 Finding: on-demand usage DROPPED. The increase is Savings Plan utilization — not runaway spend.
Token-budget engineering
Every dimension is trimmed to the top 10 keys by alert-day cost to stay under model rate limits; payload token counts are logged; subprocess timeouts were tuned from real incidents (180s → 300s after a ~9,300-token payload timed out; → 500s after a production run came within 10% of the ceiling).
Production posture
- Read-only IAM policy versioned in git.
- HMAC-SHA256 Slack signature verification with a replay window.
- Event dedup so a re-delivered webhook never double-posts.
- Docker Compose behind Caddy TLS on a real domain.
- A 15-minute health watchdog that is silent when healthy and posts the exact fix command when not.
- Daily git-bundle backups, verified by restoring them.
Outcome
Investigation time went from ~[[X]] minutes of engineer attention to 2–4 minutes of unattended run plus about a minute of reading.