Let’s say you built an AI that could write haikus about your Q4 earnings call while juggling risk reports and inventory forecasts. Cool, right? But every time it generates those poetic insights, your budget cries a little. Welcome to the economics of inference, where smarter AI doesn’t just mean better answers. It means more expensive ones.
Capital One recently made headlines for going all-in on AI. But behind the scenes of flashy model demos and press releases lies the not-so-sexy reality: inference costs are sneaky, recurring, and potentially ruinous if you’re not paying attention.
Here’s the deal: training a model is expensive, but it’s mostly a one-time hit. Inference (the process of actually using the model) is where the meter keeps running. Every time your AI responds to a prompt, it generates tokens. More tokens = more cost. And if your AI is smart, verbose, and chatty (sound like anyone you know?), those tokens add up fast.
So what’s the play? Capital One and others are now laser-focused on making inference faster, cheaper, and less power-hungry. We’re seeing major gains:
- Inference costs dropped 280x between Nov 2022 and Oct 2024 for GPT-3.5-level performance.
- Energy efficiency is up 40% year over year.
- Open-weight models are now within spitting distance of their closed-source cousins.
What does that mean for the rest of us?
If you’re in charge of AI initiatives, or even just dipping your toes in, you need to think beyond “Can this model solve the problem?” and ask “Can we afford to let it?”
Enter: Goodput
Goodput is the new buzzword, and unlike most corporate jargon, it actually matters. It’s not just about how many tokens your system can churn out (throughput) or how fast it spits out the first one (latency). Goodput measures how many useful tokens you get while still hitting speed and quality targets. It’s the efficiency stat we’ve been waiting for.
Imagine paying a team to write reports. Throughput is how fast they type. Goodput is how many reports you can actually use without rewriting or apologizing to the board.
Smarter = More Expensive (But Also More Valuable)
Models that can reason, solve problems, and plan multi-step actions tend to generate a lot of tokens. They explore different options, evaluate tradeoffs, and output well-thought-out responses. That’s amazing but it’s also expensive.
If you want AI that does more than autocomplete your sentence, you’ll need infrastructure that can handle serious token volume. That’s where things like NVIDIA’s AI factories come in. Fully optimized systems designed to crank out tokens without melting your servers or your wallet.
So What?
If you’re building or buying AI, factor in inference economics early. Don’t get seduced by benchmark scores or training parameters alone. Ask how fast it runs, how many tokens it generates, how it scales and whether your CFO will still like you after the first billing cycle.
Better yet, start tracking goodput. Ask vendors for energy usage per 1,000 tokens. Evaluate speed and output quality. Think like a product owner, not a fanboy.
Because the AI future isn’t just about who has the biggest model. It’s about who can run it sustainably, profitably, and fast enough to matter.
And if that model can still write a mean haiku about earnings season? Even better.