Observability priced per gigabyte means you log least when you need it most

Your observability platform bills by data ingested. Something goes wrong. Error rates spike, retries multiply, every service starts emitting stack traces, trace volume goes up by an order of magnitude.

The incident generates exactly the data you need, and generates a bill proportional to how badly things are going.

Somebody notices the overage. Sampling rates come down. Debug logging gets a “temporarily” in front of it that stays for two years. Retention drops from thirty days to seven. And the next time something breaks, the window where the interesting thing happened has a sampling rate of one percent.

The incentive is precisely inverted

The pricing model asks you to decide what’s worth observing before you know what’s going to break. If you knew that, you wouldn’t need the observability platform.

Metrics get billed by cardinality, so you drop the labels — customer ID, region, instance, version — and those labels are the entire mechanism by which you find out that the problem is confined to one deployment in one region. Traces get sampled at a fixed rate, so the rare slow request is statistically absent from your data, and the rare slow request is the entire question. Logs get filtered at source, so the debug line somebody added specifically for this failure mode was compiled out of the shipping config.

Every one of those is a locally rational cost decision, and together they produce a system that’s instrumented for the failures you already understand.

The vendor isn’t villainous here. Storing and indexing high-cardinality telemetry at scale is genuinely expensive, and the price roughly reflects the cost. The problem isn’t the margin. It’s that per-gigabyte pricing makes volume the billing dimension, and volume correlates with the moments you most need the data.

Where I was wrong about self-hosting

My standing advice was to run it yourself. Prometheus, Loki, Grafana, Tempo, a big disk. No per-gigabyte anything, keep everything, sample nothing.

I still run that stack and I’d still recommend it for a lot of shops. But I was quoting the license cost and ignoring the operational one, and that’s not an honest comparison.

A self-hosted observability stack is a distributed system with its own storage, its own scaling limits, its own upgrade path, and its own failure modes. It needs somebody who knows it. And it has one specific pathology that the hosted version doesn’t: it tends to fail during the incidents it exists to explain. Shared infrastructure, correlated load, and a query engine getting hammered by six engineers at once. I have personally watched a metrics stack fall over from the query load of people trying to diagnose why the metrics stack’s neighbours were unhealthy.

Whatever else you say about a hosted platform, it’s on someone else’s hardware, and it stays up when yours doesn’t. That’s worth real money and I used to hand-wave it.

The shape of a sane answer

Not “self-host everything” and not “buy the platform and eat the bill.” Route by value:

Tail-based sampling, not head-based. Head-based sampling decides at the start of a request, before anything interesting has happened, so it keeps a representative sample of normal traffic — which is the traffic you don’t care about. Tail-based sampling buffers the trace and decides after it completes: keep everything that errored, everything slow, a small percentage of the boring ones. Same volume, dramatically better content. This is the single highest-leverage change available and it’s badly under-adopted.

Metrics dense, logs sparse, traces surgical. Metrics are cheap and aggregate well — that’s the layer that should always be on, at full resolution, with the labels intact. Logs are the expensive layer and should be structured enough that you can be selective. Traces are for when you have a specific question.

Keep raw data cheap and locally, index selectively. Object storage costs almost nothing. Ship everything to a bucket with a long retention, and index only what you query routinely. When an incident needs the raw data from six weeks ago, it’s there — it’s just slow to search, which is fine, because you’re doing archaeology, not paging.

Instrument with OpenTelemetry. Not because the standard is beautiful — it’s a committee protocol and it shows. Because it makes the exporter a config change rather than a rewrite, which is the only thing that gives you leverage in the pricing conversation. The vendor knows whether you can leave.

The tell

If your team has ever, during an active incident, said “we don’t have that data because we turned down sampling,” you already know what the pricing model bought you.

That sentence is the entire cost of the discount, and it doesn’t show up on any invoice.