Will inference move to the edge?

Compute has been consolidating into hyperscale facilities for fifteen to twenty years for two reasons that predate AI. The first is efficiency: Google’s power usage effectiveness is close to 1.1, about a tenth of a watt of overhead for every watt reaching the chips, which smaller operators cannot match. The second is sharing, since hardware bought once and used by many customers is cheaper per customer. Training then adds a hard technical reason to centralize. A model with on the order of a trillion parameters is learned by splitting the data across hundreds of thousands of graphics processing units, each working its own slice, which then periodically stop and exchange the weights they have learned. That exchange is expensive enough that the chips must sit close together, which is why training means thousand-megawatt campuses, and it is also what produces the violent power swings that make training loads awkward for the grid.

Inference has none of that structure. A prompt is handled by a single graphics processing unit, or maybe eight inside one machine, because the model, the data and your prior conversation all live there, and no large group of chips has to coordinate to answer. Lee says that to his knowledge there is no real technical penalty for running inference in a small facility. Two arguments push it outward. The first is latency, and Lee immediately weakens it: before generative AI users expected a web service to answer in about a hundred milliseconds, but chatbots have reconditioned people to wait seconds or tens of seconds. What still demands low latency is cyber-physical AI, meaning robots and autonomous vehicles, where responsiveness underpins safety. The second argument is power. Kann’s thought experiment asks whether it will stay easier to find one gigawatt-scale site in a metro area than ten hundred-megawatt sites; both expect that to flip, which would make distribution the path of least resistance rather than a technical preference.

So why is nobody doing it? Lee’s answer is uncertainty about applications. Until it is clear which AI products drive inference demand, nobody knows the performance requirements, so nobody commits capital to a network of small sites. Meanwhile the giant sites have to be built anyway for training, so inference rides along on capacity that already exists. The existing edge layer is no shortcut either: content distribution networks and points of presence were provisioned for central processing units, with power delivery and cooling sized for far lower density, so converting one is a retrofit rather than a swap. And there is a scenario where the shift never becomes necessary. If training hits diminishing returns, or exhausts the available data, the big facilities free up and inference absorbs the slack, which weakens the case for small sites. The trigger Lee names for the other branch is commercial, not technical: a provider competing on performance rather than on capability has to put chips in the market it wants to win, even with idle ones in Nebraska.

On-device inference is the version that sounds most disruptive and the one Lee expects least. The advantages are real: responsiveness, privacy, and tight integration with a platform such as Apple’s. The trade is capability. A frontier model carries on the order of a trillion parameters; a phone-resident model might carry seven billion, orders of magnitude smaller and correspondingly less capable, which works only if you have identified a handful of tasks you care about. Device compute also changes the binding constraint from power to energy, because what matters is battery life, and adds a thermal ceiling nobody wants in a pocket. Lee puts the device share on the order of 1%. Which leaves the energy conclusion, and it inverts the episode’s opening premise. Kann asks whether spreading inference across many smaller facilities would raise total consumption, since their power usage effectiveness will be worse than a hyperscaler’s, and Lee agrees. Moving inference to the edge does not shrink AI’s energy demand; it relocates that demand into many smaller grid connections and probably increases the total.