Editor’s note: The following is a guest post from Chen Goldberg, EVP of product and engineering at CoreWeave.
Put two teams on identical infrastructure — same GPUs, same model, same configuration, but different workloads. The economics they walk away with will look nothing alike.
That surprises most people, but we’ve seen this coming for a while. For a couple of years after the initial hype, the hard part of AI adoption was figuring out whether and where it could add business value. Thankfully, that’s mostly behind us now.
The models are now in full production mode, carrying actual business workloads with real consequences. Production is about continuous operations, not a deployment milestone. So, technology decisions are now about the business value they deliver, with real roadmaps riding on the answer.
Right now, the popular metric is cost-per-token, because it’s easy to talk about and simple to compare across vendors. A published cost-per-token reflects a clean run: healthy hardware, a steady workload, a batch size picked to make the chart look good. However, it tells executives very little about what that same setup does in completely different environments.
Going back to the two teams at the beginning: same model, same hardware, different job, different economics. Decisions get nuanced fast as the bottlenecks move.
Serve short prompts in a large batch, and operations are compute-bound, with GPUs working as fast as they can. Move to long context and the same model becomes memory-bound, where loading the key-value cache sets the pace and the chip spends time waiting for bandwidth instead of computing.
Putting a hard latency ceiling for real-time applications means a system that benchmarks at thousands of tokens per second can only deliver a fraction of that.
The levers have no universal setting. GPU choice, quantization, attention kernels, KV-cache layout, batching, speculative decoding: every one of them trades speed, cost, and sometimes model quality against each other. The tradeoffs depend entirely on the workloads.
That’s what happened to those two teams: nothing about the hardware changed; they were simply running different jobs.
So how can one metric capture all that? The answer is simple: It can’t. Cost-per-token isn’t broken; it’s just answering a smaller question than the one businesses should be asking.
What companies actually pay for
I try to steer conversations away from the invoice and onto the system. The question I tell customers to ask isn’t what a token costs. It’s how much of every GPU-hour the customer pays for is doing useful work.
A few questions can help decision-makers understanding compute pricing more clearly:
- How much compute time turns into productive output? A team that finds a third of its GPU hours lost to stragglers can recover more capacity within its own cluster than switching vendors would ever provide.
- How fast can problems be detected and remediated? At scale, something is always degrading somewhere. The cost isn’t whether problems happen. It’s how long they run undetected when they do. Time-to-detection is an economic variable, even if no pricing page has a column for it.
- What does infrastructure cost when demand isn’t smooth? Production traffic doesn’t look like a benchmark. Inference spikes with users, agents fire at any hour and fine-tuning jobs arrive in bursts. If the platform can’t absorb irregular load, customers end up paying for peak capacity that idles most of the day or drops requests at exactly the moment they matter most. The cost of elasticity never shows up in a per-token price.
Customizing compute
Production AI infrastructure can’t be chosen the way customers pick a SKU.
Benchmarks are the floor, not the ceiling, and specs describe a chip in isolation rather than workloads under constraints. The only reliable way to know is to run your own workload under real conditions and watch what happens.
Things like cost, latency, accuracy, what happens when a component dies, and what happens when the queue backs up. Often, the experiment answers a bigger question too: which model actually fits the job? The teams that make good calls treat evaluation as an experiment.
That’s also why the vendor relationship is changing. The good ones now look less like a menu and more like a joint engineering effort. Sit with the customer, look at the real application, find where the workload will break, and tune for it before anyone signs anything. You learn things in that process that no data sheet contains.
