Past the Post

Selling inference over a stateless API is a great business for frontier labs. Here I describe modest reasons which apply even in slow-prnpogress worlds.

1\. Anxiety about commoditization leans on a particular attitude – call it past the post – about how we’ll use models. Maybe there’s a “workflow” you’d like to automate. Contrast past the post with headroom. Some jobs are hard to saturate. Easy examples are adversarial games. Litigators or HFT shops don’t make money getting past the post. Here a redqueenish flavor. But elsewhere it looks more like an open field.

One pleasant idea about investors is that they’re widget-makers, they put out units of clever analysis. Incidentally there are some decisions also and money gets shuffled around. Or say consultants make slide decks. Investors are not widget makers, neither are consultants. There are no widget-makers. Rather, it’s nobody’s job to make widgets; or rather, if it’s yours, you are gonna get run over. There are factories which literally make widgets but the factory is not the firm. The firm’s mandate is to have some grip on the future. If you are making widgets, probably make some more of them and do it better, and do it better than someone else could. And so on.

Notice how Google, Meta, etc. continue making more and more off of ads. They are doing this enormous optimization job. You might guess they’ve squeezed everything they can from their userbase; and the only way to move the needle is finding new people to squeeze. Not so. It is in one sense a neatly bounded problem, but maybe contains a number of messy unbounded problems, i.e. there are surprises left such that at any point even the best solution will involve some mistakes, or at least a degree of myopia waiting to be undone.

And that’s not mentioning e.g. the life sciences. If there is a lot of work with headroom, there will be a big and growing appetite for the very best models.

So say scaling plateaus. Distillation pulls fast-followers along; algorithmic ideas diffuse also. Nobody enjoys the brute advantage from another OOM. Also, nobody has to pay for it. There is still a frontier, if not so far away. This is the category which matters. There will be a big and growing appetite for it; and those consumers who can swallow the most will beat out their peers. If they can just stay ahead, even barely, frontier labs will make money. If you make the best model you will have pricing power. You are in basically a different business from commodity fast-followers, no matter how close they are nipping at your heels. Noticing that you both sell tokens makes a bad analogy.

2\. In the pantheon of chart crimes Leopold’s works will have an honored place. Preschooler – elementary schooler – smart high schooler. We’ve got vague ideas about the nature of intelligence (in humans and elsewhere), also the nature of work. Mostly it’s muddy. And then, narrowly stronger models can be qualitatively way better. For continuous gains in raw capability, expect practical gains are discontinuous. We saw this with neat improvements like tool calling. But also general ones: Opus 4 was subtly smarter, but a step change for practical use. 4.5 again. Say in a world with steadyish progress benchmark scores bunch up under the pareto frontier. If there are more step changes waiting, even narrow advantages could be massively useful.

3\. Noam Brown talks about intelligence/dollar. Can you match a strong model by inference-scaling a weak one? On some jobs, sure, the only difference is efficiency. But intelligences don’t always funge. Crudely imagine a “capacity” to hold a problem in your head.

Put me in a cave and give me a thousand years. Even with arbitrary inference-scaling there are some problems I just couldn’t solve (1) where I can’t recognize the solution (2) where some bottlenecking move doesn’t fit in my mind. Think of Von Neumann's prodigious working memory. There could be other bounds like this. Anyway, stronger models will tend to themselves have better inference scaling properties.

4\. Today’s frontier models are characteristically generalists (granted, some of that generality hard-won from RL patchwork). If we plateau, won’t the strongest model in each domain be a specialist built on top of a nearly-as-good OS base? I think the frontier/commodity gap will be greater than the specialist/generalist gap.

It matters where the gains come from. Finetune a model to not screw up in excel. There are gains for specialists. You get past the post quicker. But you can only be so good at excel. What about finetuning a model to be a better investor? It’s a very broad problem – you want to deeply understand the companies you invest in, have a gears-level view of what they do, the fields they’re in, macro conditions, the people, cultural tides – in the limit you want a picture of the whole world.

A lot of jobs are surprisingly general. Openai stopped training separate Codex models. Models like Cursor’s Composer, which take commodity capabilities (Kimi) and then specialize on top of them, are not competitive in their specialty vs generalists like Claude. Super strong practical reasons, we don’t yet know for sure. (And of course there will be Alphafolds.) But I expect this is basically true, and less a property of models than of the world.

Importantly models are so good because they are so general. Part of what makes Claude a great coding agent is that it has a strong theory of mind, including for your end-users.

5\. Will Brown says models are like Einstein, but it’s his first day on the job. We are supposed to notice: wouldn’t it be better to have a regular Joe who knows his way around? I think we are not nearly excited enough about the Einstein bit.

How absurd to have polymathic frontier models doing regular-Joe tasks. But we can, and should. You can’t say upfront how it’s gonna pay off. But having high-caliber attention on every corner of your business, you may notice things you didn’t know to look for.

We soon return to an old question. How susceptible is the world to intelligence. Sure, there are diminishing returns. When are they important? (Some stronger answers produce views like “superintelligence wouldn’t even be that interesting”). A good bet is we have a ways to go.

Recall some boring but plausible ways labs could turn a profit: verticalize pushing into app layer; paradigm shift e.g. continual learning yields a stickier product; RSI then total dominion over the lightcone. But stay in the nothing-happens world. Here it’s easy enough to notice those same dynamics which play in the more dramatic alternatives. There’s just so much headroom. The world is big and richly textured, so is the set of possible minds. (Another reason why models won’t converge entirely). It is worth noticing that we can make models a whole lot better, even in-paradigm, no inspiration whatsoever. The big labs are today immature organizations. They fail to notice when tens of millions of dollars of inference guess get spent on Doing Crime. Safety aside it’s a failure by their capabilities people. They could be much, much better at training models. I wonder what well-made models look like. At any rate when they arrive we will want them badly.