Here's the short version: we were paying a large observability vendor a lot of money, using a fraction of what we paid for, and we switched to a leaner platform. The bill dropped by about 75%, and on a six-figure contract that is real money, not a rounding error.

The why behind it is more interesting and helpful for you.

The Ferraris of Observability

Datadog, New Relic, and the rest are the Ferraris of the observability world. They're arguably great. Solid dashboards, every feature you could dream up, endless roadmaps, an integration for everything. The problem is that most of the team was never driving the Ferrari. We were using a fraction of what we paid for, at enterprise per-seat prices.

What I Actually Want: Query-First Observability

"Agentic observability" is the buzzword of the month, so I'll avoid it. What I mean is simpler: I want to query my telemetry through an API or an MCP server and get an answer.

When something breaks at 2 AM, I don't want to click through six dashboards. I want to ask a question in plain language, through CLI and have it pull the logs, metrics, and traces it needs to tell me what's on fire. As AI has become genuinely useful, this is often how I troubleshoot now.

A platform whose data I can reach programmatically is worth more to me than one with a hundred features I'll never open.

Alerts Belong in Code

The other hard requirement: everything should be manageable as infrastructure as code, especially the alerts.

If you've ever had to recreate a few hundred hand-clicked alerts during a migration, you know it's a nightmare. Alerts defined in code are reviewable, reproducible, and possibly even portable to the next platform because there's always a next platform.

Just as important: when alerts live in code, anyone can contribute one. A single DevOps or SRE team can't hold the context for every service in the company. The engineers who own a service know what "unhealthy" looks like far better than that central team does. Let them open a PR against the alert definitions and own their own signals.

The Takeaway

Cutting the bill by 75% wasn't a magic trick. We stopped paying for a feature set we didn't use, picked a simpler but equally as efficient platform we could query and script, and put the alerts in version control where the whole team could own them. The savings were the outcome and the real win was a setup that actually fits how we work.

Quick Tips on Reducing That Observability Bill

None of these are silver bullets, but together they add up. Here's a grab bag of what worked for us and what I'd reach for anywhere else.

  • Justify the switch in sprints, not vibes. Before moving platforms, add up the engineering time the migration will actually cost you in sprints, then weigh that against the yearly savings. If the platform pays for itself in a quarter or two, the math sells itself to leadership.
  • Fix your log levels. DEBUG and TRACE have no business running in production by default. Flip them on when you're actively chasing something, then flip them back off. A surprising amount of spend is just verbosity nobody remembers turning on.
  • Hunt down legacy logging. Old logging frameworks and copy-pasted boilerplate tend to shout the same thing thousands of times an hour. Modernizing or trimming that noise was one of our single biggest levers.
  • Right-size your alerting. Every alert that pages someone at night should be actionable. Demote the rest, delete the ones nobody has acted on in months, and stop paying to evaluate signals you already ignore.
  • Delete metrics nobody looks at. Audit your most expensive metrics and check which ones actually show up in a dashboard or an alert. If a metric hasn't been queried in months, it's just a line item.
  • Watch your cardinality. High-cardinality labels are the quiet budget killer. Never tag a metric with something unbounded like a user ID, request ID, or email. One careless label can quietly explode into millions of time series.
  • Sample the boring stuff. You don't need every successful, 200-response request. Keep all of your errors and slow outliers, then sample the happy path down to a small percentage. Tail-based sampling makes this painless because it decides what to keep after the trace finishes.
  • Filter at the collector. Dropping noise at the OpenTelemetry collector, before it ever reaches storage, costs you nothing operationally and can cut ingest dramatically. Health checks and repetitive cron output are usually the first things to go.
  • Right-size retention. Not everything deserves hot storage. Keep recent data fast and searchable, roll older data into cheaper tiers, and send compliance logs straight to object storage.
  • Tier your backups too. Disaster recovery snapshots and long-term backups don't need to sit in expensive hot storage. Let them age into cheaper classes automatically with something like S3 Intelligent-Tiering or Glacier. The same idea applies on Google Cloud with Autoclass and its Coldline or Archive tiers, or on Azure with the Cool, Cold, and Archive blob tiers.
  • Know your rhythms. Figure out what you actually reach for daily, weekly, and monthly. The daily stuff earns premium storage. The monthly stuff can live somewhere slow and cheap.

If you only do one thing, start with logs. They're usually the majority of the bill, and where the easy wins hide.

May you all keep both the bill and the nightly pages to a minimum.