For Consolidation

All very stylized: the models are important, two or three frontier labs make them. Maybe there are fast followers. It could be good business. It doesn’t matter much. The labs have got secret knowledge and then vast scale. Takeoff or no there is a departure. Ilya shuffles off to his bunker. Sam and Dario and Demis sit with heads of state. We are quick to recognize, this is a world with concentration of power. You get what it says on the tin.

This story is easy to imagine and not wrong either; it easy to imagine so elides structural reasons. In fact I think it is pretty overdetermined. Another story, sometimes popular: upward pressure from open source and a capability ceiling above; modelmakers squished up against the frontier so models will get commoditized, we enjoy intelligence on tap, and it will be just like electricity. I think SOTA models won’t get commoditized. Even if they do, we should expect concentration, and for familiar reasons.

\ You can do it or you can’t – once you can, good enough. Some jobs are like this. Maybe there is a “workflow” you want to automate. Call the idea PAST THE POST. It is a claim about the kind of stuff we’ll use models for. One pleasant idea about investors is that they are widget-makers, they put out units of clever analysis. Incidentally there are some decisions also and money gets shuffled around. Or say consultants make slide decks. Of course investors are not widget makers, neither are consultants. There are no widget-makers. Rather, it’s nobody’s job to make widgets; or rather, if it’s yours, you are gonna get run over. There are factories which literally make widgets but the factory is not the firm. The firm’s mandate is to have some grip on the future. If they are making widgets they will probably make more of them and do it better, and do it better than someone else could. And so on. Contrast PAST THE POST with HEADROOM.

HEADROOM against humans. Humans are really impressive and we continue failing to appreciate it. Taking HEADROOM against humans seriously we can reconcile how impressive today’s models are with the fact that they haven’t upended everything yet. (Without arguing about diffusion.) Then we can see strong in-paradigm progress for a while and get very strong systems that are not AGI. Nothing ever happens: the RL industry patches flaws one by one; at each turn some new deficiency comes into focus, or rather, we learn a new way that humans are pretty cool. The process repeats. We get increasingly useful models. Work is sliced up and some biggish swathes are traversed only by models. Mostly organizations are centaurs, and really shot-through with the centaur arrangement rather than slicing the bottom half off. We shrug and rearticulate the O-ring idea. The degenerate case is “humans are magic” failing to assimilate the concept AGI at all. HEADROOM suggests – at least for many sensibilities – a dangerously appealing world, just enough like ours, new and exciting also, just strange enough that surely it is an eminently reasonable guess. Still I’m pretty sure it is worth devoting more subtle attention to what makes humans so damn good.

HEADROOM against work. Some jobs are hard to saturate, you can keep getting better. Easy examples are adversarial games. Traders don’t make money by getting PAST THE POST. Or notice e.g. how Google, Meta, etc. continue to make more and more money off of ads. They are doing this enormous optimization job. You might guess they’ve squeezed everything they can from their userbase; and the only way to move the needle is finding new people to squeeze. Not so. It is in one sense a neatly bounded problem, but maybe contains a number of messy unbounded problems, i.e. there are surprises left s.t. at any point even the best solution will involve some mistakes, or at least a degree of myopia waiting to be undone.

If you think these two expressions of HEADROOM are very different I would like to nudge you and say they are importantly similar. So far we’ve only had jobs for humans (maybe better, jobs for firms) so the measure is pretty tied up.

The commoditization story leans hard on PAST THE POST. Say we get commoditization – everybody is selling pretty much the same thing – what is true in this world? Progress has stalled, or demand has stalled (e.g. the best models churn out strange new math but nobody is buying it). More or less contingent reasons: maybe scaling in the current paradigm only gets us so far, a new one doesn’t come in time. Instead fresh AI winter (bimodal, this decade or bust). Maybe there is a “natural” capability ceiling. Hard to say why this would be the case. If you want to rescue the intuition: there could be diminishing returns to using intelligence against the stuff we care about. The very aggressive version of this claim is that a superintelligence wouldn’t even be that interesting.

If there is HEADROOM against humans, then there is plenty of room to make better and better models, even below human level And you should expect dramatic effects even if you have the very aggressive view that superintelligence will be boring. If there is HEADROOM against jobs, people will keep buying the best and newest models. There will be a big mass of work done by commoditized models (maybe OS) getting PAST THE POST. You can also anticipate the proliferation of specialist models. There will be small specialists trained to do specific jobs as cheaply as possible; and maybe specialists which push particular regions of the frontier. Generalist models could orchestrate specialists. (There are strong reasons to avoid compound systems of this style; but you should definitely expect them.) Consumer apps will tend to wind up in this bucket. It’s a popular claim that the models are already smart enough for consumers who won’t notice, much less care about, further gains on the capability frontier. This is pretty obviously wrong, and reveals a confused sense of current capabilities, or at least a lack of imagination regarding the apps we’ll make. But eventually it will be right. But jobs with HEADROOM are the ones you should care about. They will present a big and only-growing appetite which, in time, describes where most dollars and tokens go. Following HEADROOM we should preserve the category frontier lab – it’s in the name – they do this characteristic thing which is pushing a frontier.