Runway: Learning From Watching the World

Welcome to the 1,825 newly Not Boring people who have joined us since our last essay! Join 281,011 smart, curious folks by subscribing here: Hi friends 👋, Happy Tuesday! A couple of weeks ago, I mentioned that I was partnering with Runway this quarter as our presenting sponsor to bring some of my writing to life in the form of video. It took us a while to land on the right partner to form a deeper relationship with — it has to be a company I want to talk about a bunch — but when Runway reached out, it was a clear perfect fit. For one thing, I’ve had a lot of fun playing around with their video models. But for two others: 1. I’m fascinated by World Models (and quoted Runway co-CEO Anastasis Germanidis in the essay Pim DeWitte and I wrote) and 2. Runway has been on a tear with new product releases. I wanted to understand how, so to kick off the partnership, I asked to write an essay with me explaining how video, interfaces, operating systems, and robotics all fit together, why video was the right starting point for World Models, where the world is heading if Runway’s bets are right, and whether or not, as someone who simulates worlds, he believes we’re living in a simulation. Let’s get to it. Runway: Learning From Watching the World A Co-Written Essay with Runway Co-CEO Anastasis Germanidis From time to time, a certain company catches fire. You can tell when it’s happening by the pace and breadth of their product releases. One day, a new capability. Cool. The next day, another. Woah. Next day, another still. Runway, the world model company best known for releasing the first commercial video generation model, is on such a run. Over the past six weeks, it has shipped a real-time interface model, a new interactive world model, an open-weight robotics model, an AI agent for performance marketers, a research preview of an operating system, and it bought a robotics lab in Paris along the way. Meanwhile, the business is growing faster than at any point in its history, adding more than $100 million in annual recurring revenue in the last quarter alone. From the outside, it looks like a company that suddenly figured something out. From the inside, the view is much more interesting. Runway has been pursuing a particular set of bets for eight years, and that near-decade of research and iterative deployment is now yielding a seemingly disparate but actually related set of new models and products. On September 1st, Runway introduced Solaris, the “first Interface World Model,” which allows users to generate interfaces in real-time, through prompts or actions on screen, like this one where a guy takes a selfie and tries on glasses and tweaks his hair and smile with the drag of a mouse. On September 8th, Runway announced it had acquired Kinetix, a Paris-based research lab, which specializes in… robotics? From the video company? Then at their September 30th AI Summit, the team shared Praxis-1, “an open-weight World Action Model that turns our video pre-training expertise into real world control for robots.” The thesis is that while robotics data is limited, video data is practically infinite. So Runway pre-trains on video data and then uses that as a policy model for robotics developers who can train it on their specific embodiments. Standard Bots, whose co-founder Evan Beard and I co-wrote an essay on robotics, is working with Runway to deploy Praxis-1 on its arms. And then the very next day, on October 1st, Runway previewed Project Continuum, “a new operating system research application built around real-time video interfaces.” A video model. An interface model. A robotics model. An operating system. Are you getting it? These are not separate pretrains. They’re different applications, built around the same General World Model backbone, with different post-training on top. Made with Runway All of them are examples of what you can do when you can simulate any pixel. An operating system ends up as pixels on a screen, like video and interfaces do, and like the world does through a robot’s eyes. If you can simulate every pixel at high enough fidelity and speed, then you can make almost anything that shows up to the world as pixels. This is the future Runway is building towards. Runway is making a few different bets, on a research direction, the Bitter Lesson, and a business strategy. It is betting that world models can learn, like humans do, from third-person observation. It is betting that letting models watch a ton of video will lead to models that understand how the world works better than they could by reading about how the world works or from any other human-mediated data source. And it is betting that it can build a massive business in content creation in order to both help fund the large investment into better and better models, and to become the rare company that is both a frontier lab and a distribution layer. Runway’s commercial success and product prolificness points to a surprisingly wide set of applications for World Models; surprising, that is, unless you read the essay Pim DeWitte and I co-wrote explaining why World Models have the potential to eat everything. In it, we wrote: World models — these systems that learn from watching the world and the actions taken in it — are a fundamentally new kind of foundation model. They can compute what was previously uncomputable. There are a number of different types of models whose makers call them World Models, although many don’t fit our definition; they don’t predict the next state based on intervention. We categorized Fei-Fei Li’s World Labs, which AMD recently agreed to acquire for $8.2 billion, as building “3D Reconstruction and Generation Models.” Runway, we wrote: Is blurring the lines between video generation and world generation, as mentioned in the Video Model section. During its Research Demo Day 2025, Runway co-founder and CTO Anastasis Germanidis explained the company’s evolution that started from “generative AI models [as] viable tools for creative expression”. They then evolved towards World Models (while still making incredible progress in video models.) “To build a World Model,” Germanidis explained, “We first needed to build a really great video model. We believe that’s the right path to building World Models, that teaching models to predict pixels directly is the best way to achieve general purpose simulation.” Runway first mentioned its World Model ambitions in a December 2023 blog post by Germanidis, who is now co-CEO, titled “Introducing General World Models.” This has been the plan for quite some time: teach the models to make videos from watching videos, and then teach them how to bring the actions in video to life. Anastasis describes it as a plan in four chapters: Chapter 1: Media and Entertainment Chapter 2: Broader Content Needs and Full Video Chapter 3: Robotics (We Are Here) Chapter 4: Science The plan is coming together, as evidenced by the recent string of releases, and I have become a World Models convert who believes they have the potential to do a lot of things in the real world that LLMs can’t, so Anastasis and I decided to co-write an essay on the company’s bet that watching a lot of videos can teach models to do almost anything. Learning from Watching One of the things you realize when you have kids is that babies are learning machines. When they’re really young, they just keep changing, picking up new skills, like little Neos. The thing about babies, though, is that they don’t know how to read. They don’t learn like we learn in school, by reading textbooks, because they can’t. Instead, they watch. They watch how we walk, how we talk, how we smile, how we bring food to our mouths, how we pick up toys and how we put them away, they watch how we shake hands, and hug, and run, and jump, and crawl. Then they try it themselves, clumsily at first, they watch again, they try again, they watch again, they try again, and try and try and try until they’ve gotten it. Made with Runway As we grow up, we do go to school, and we do learn from books, but we still learn by watching. We watch how the kids in the older grades dress and talk and interact. We watch how our favorite athletes play. We watch to learn how to interact with and manipulate the world. If you’re a guy, think about the last time you had to tie on a bowtie for a wedding. The best way to do it, other than having the one guy who knows how to tie a bowtie tie it for you, is to watch that guy tying his bowtie. The second best is finding a YouTube video of someone walking you through how to tie a bowtie. At the bottom of the list is reading how to tie a bowtie. I’ve tried it, and it simply doesn’t work, at least not for me. Language is compression. We look at the rich, ever-changing world in front of us, and we can’t possibly capture it all. By the time we’ve come anywhere even close to describing what we see at any given time with any amount of precision, it’s changed. And no matter how hard you try, words will never be enough, because you can’t, for example, see each of the atoms in the scene, track the movement of the electrons and protons and neutrons that make up those atoms, see how electromagnetic and gravity waves interact with them… you get it. Everything in the world changes, then changes again, and each time it does we’d have to start describing it again, like verbal Sisyphuses. Instead of running a constant monologue as we walk down the street, describing all of the motion and appearances and interactions we see, we simply say, “The man walked down the street.” Neither approach captures the full picture, but the latter is easier and allows us to actually live our lives. To jack as many subtle details into our brain as possible, therefore, we don’t read about life, we watch it. Our brain has evolved to pull out the relevant information. Made with Runway A picture is worth a thousand words. A video is worth 1,000 times the number of frames plus however many words the motion connecting them and sounds are worth. But the value of video isn’t simply that it contains more information than words. Predicting what happens next and checking how well you predict, over and over again, creates a pressure to learn reusable relationships. Imagine trying to predict the next frame of that street video. To get it right, it isn’t enough to know what the man looks like. You need to know that he’ll keep walking in roughly the direction he was walking, at roughly the speed he was walking, that his left foot comes after his right, that people walking near him will walk behind him and not through him, that when he passes in front of a moving car, that car still exists and will come out the other side a beat later as the same car, that his shadow moves when he moves, etc. When a model is trained on video, nobody tells the model any of this. There’s no line of code that says “objects still exist when something walks in front of them” or “people can’t walk through people.” The model guesses the next frame, sees how wrong it was, and adjusts. Somewhere along the way, the cheapest way for the model to stop being wrong is to learn something like the rules: things fall, and seem to fall in basically the same way at basically the same rate of acceleration (although feathers seem to flutter back and forth while bricks simply drop); glass shatters and rubber bounces; a ball that rolls behind something comes out the other side looking exactly the same, unless there was a puddle of mud. One video with a muddy ball might confuse the model, but over millions of them, it would learn that the mud is an anomaly. These, incidentally, are things that babies learn during their first year. We play peek-a-boo because it makes them laugh, but also because it teaches them object permanence, this idea that things still exist even when you can’t see them. This is how LLMs work, too. Their goal is to predict the next word, by any means necessary. Ilya Sutskever has been making this point for years: predicting the next token correctly requires understanding something about the reality that produced the token. Swap tokens for frames and you have Runway’s thesis. There’s a theory in neuroscience, which I discussed with Cristóbal Valenzuela, Runway’s co-founder and Anastasis’s co-CEO, called active inference. Karl Friston is its most well-known proponent. The idea is that your brain is a prediction machine, constantly running a simulation of what’s about to happen and checking it against what actually happens. You don’t normally notice it, because you get really good at prediction and it just runs in the background. You only notice it when you get it wrong. You’re walking up the stairs, you go for one more step, the step isn’t there, and for a split second your whole body lurches while the simulation crashes and reboots. “The moment you see it didn’t predict,” Cristóbal said, “you adjust.” A model that sees enough examples of success and failure starts to carry around its own version of that simulation. But until the model runs in a physical body instead of a screen, it can’t feel what it’s like to stumble, trip, or fall. Why video, though, and not another type of data? That’s where the first of Runway’s bets comes in: The Bitter Lesson. The Bitter Lesson and the Big Blob of Video It is practically illegal to write about training models without mentioning The Bitter Lesson. In 2019, Richard Sutton wrote a short essay arguing that the biggest lesson from seventy years of AI research is that “general methods that leverage computation are ultimately the most effective, and by a large margin.” It is bitter because every time researchers have tried to build their own understanding of a domain into a system (chess openings, the structure of grammar, the edges and corners in an image), they’ve eventually been steamrolled by dumber methods that could soak up more compute and more data. For his part, Anthropic CEO Dario Amodei made a similar observation two years earlier, in a recently-partially-leaked internal OpenAI research document he titled “Big Blob of Compute.” The Big Blob of Compute (BBOC) hypothesis, he wrote, “says that creating any given intelligent behavior is mostly about providing a large, minimally structured mass of computational capacity, and then giving it shape and form via interactions with a rich environment and a training process that drives towards behavior appropriate for the task at hand.” Runway’s corollary is that the most scalable source of data tends to win. Call it the Big Blob of Video (BBOV) hypothesis. LLMs won on text because the internet had an absurd amount of it. Robotics has been climbing its own ladder. It used teleoperation data, a specific robot doing a specific task puppeteered by a human. This year, many robotics labs began using a lot more egocentric video from cameras strapped to people’s heads as they perform a task. Runway is betting on something simpler and less human-mediated: vast quantities of video that already exist, and will be filmed (and generated) from here on out. Watching video is the most Bitter Lesson-pilled approach to AI there is. Language is a compression of reality, an incredibly good one that humans have been refining for thousands of years, but a compression nonetheless. As Sutton wrote, “We want AI agents that can discover like we can, not which contain what we have discovered. Building in our discoveries only makes it harder to see how the discovering process can be done.” In other words, we limit agents with our own beliefs. Put more bluntly: if you really believe the Bitter Lesson, you should be working on World Models. There’s a good argument against this thesis, so we might as well make it ourselves. If language is compression, well, compression can be useful! Learning about the world from pixels is less efficient than learning from text. Researchers have shown that you can build a language model out of raw audio, just by predicting the next little chunk of sound, and that language eventually emerges, but that it takes something like three orders of magnitude more compute than learning from text. Text comes predigested. Somebody already did the work of sorting out the most important stuff. Pixels make the model do that work itself, and there are a lot of pixels on which the model must do it. Then there’s Yann LeCun’s critique: predicting every pixel wastes effort on details that don’t matter (every leaf on every tree, every ripple in every puddle), so models should predict in an abstract representation space instead. Runway’s answer is pragmatic: use whatever works. So far, we haven’t seen evidence that JEPA scales better than the diffusion models it uses, and the simplest approach tends to scale best. Plus, it’s really hard to know ahead of time which pixels are noise. The rustling leaf isn’t noise if it tells you that the wind has picked up. The wind, even ever so slightly, will impact how everything else in the frame moves. The efficiency problem is real, though. Runway thinks the answer is more compute, better training to squeeze more from every frame, and better algorithms. As Anastasis said in our first conversation, “Naively scaling what we did two years ago won’t solve these problems. I expect the compute needed to train World Models to eventually exceed what LLMs need, and we’ll need real algorithmic progress on top of that.” Good time to be Jensen Huang. The World Model Backbone Anastasis’s bio on X reads “Simple ideas, pursued maximally,” which is also a pretty good summary of Runway’s research strategy. The simple idea at Runway is next-frame prediction. A language model learns to describe the world by predicting the next token in a sequence of tokens. An autoregressive World Model, as Cristóbal put it, learns to simulate the world by predicting the next frame in a sequence of frames. Autoregressive is a fancy word for a relatively simple idea. The model generates one frame at a time, each based on the frames that came before it, the same way an LLM writes one word at a time. Most video models don’t work like this. They generate a whole clip at once, refining it from static into a picture over dozens of steps, which is great if you’re making a ten-second ad and useless if you’re trying to interact. Interactions stop feeling interactive somewhere around half a second of lag. If you want a model that a person, or an agent, or a robot can actually interact with the way humans do with other humans, it needs to produce the next frame quickly, then the next one, then the next, each taking into account whatever just happened in the previous one and in the reaction to it. That’s what Runway did with GWM-1, which it announced last December. It took Gen-4.5, its most recent base video model, and turned it into a model that generates frame by frame, in real time, and can be steered with actions. In its Solaris announcement, asking, “What if software could generate itself, as you interact with it?,” it laid out the general recipe: Teach the model to generate frames sequentially, Distill dozens of denoising steps down to a few, Train the fast model on its own outputs so the quality holds up over a long session. No code underneath, no layout to translate. The image is the application. It treats your clicks and drags the way our video models treats words: not as commands to look up, but as a signal for what to generate next. Nothing about it is decided in advance, so nothing about it has to stay the same… the scene answers to your intentions. It becomes whatever you ask of it. Actions are the important thing. In my (Packy’s) previous essay on World Models, we drew the line between video models and World Models like this: a video model predicts the next frame, P(xt+1 | xt), while a World Model predicts the next state given an action, P(st+1 | st, at). “That at,” we wrote, “is the magic.” Runway’s last month or so of releases can be viewed as a bunch of different at’s. In GWM Worlds 2, the actions are camera movements plus free-form text addressed to anything in the scene. In Runway Characters, the action is audio, a person talking to an avatar that listens and responds. In Solaris, the actions are clicks and drags. In Praxis-1, the actions are robot commands, expressed as where the robot’s hands should go next. There’s a different vocabulary for each, but they all use the same machinery underneath to predict what happens next. I (Packy) have been fascinated by World Models, and really do think they’re some kind of magic, so when I talked to Cristóbal, I wanted to be sure that I understood: “So all of this really just is the same backbone?” “Correct,” he said. “The backbone is frame prediction at scale, which eventually gives rise to a World Model in the same way that next token prediction at scale gives rise to large language models. And the applications are diverse on top.” This is why the company is moving so fast across seemingly disparate applications right now. About 90% of Runway’s research effort goes into pretraining. The belief is that the capabilities get learned there, by watching, and that post-training mostly brings out what the model already knows. Once the base gets good enough, each new application takes far less effort to stand up than the one before it. You don’t, for example, need to build a robotics model from scratch. You can take a model that has already watched an enormous amount of the world and teach it to speak robot. There’s a lot of room left. Runway estimates that Video and World Models are a year or two behind LLMs, which means pretraining is nowhere near saturated. My (Anastasis’) rule of thumb is the number of pixels a model can generate, which is resolution x duration, roughly doubles every few months. I don’t see any fundamental blocker to hour-long generated video within a year or two. Longer videos aren’t simply a matter of doing the same thing, more. A model that has to keep a scene coherent for an hour has to learn persistence. Turn around, and what was behind you is still there. The mug you set down on the counter is still on the counter twenty minutes later, unless somebody moved it, in which case the model should know who moved it, and where. Those are exactly the things today’s models still struggle with; longer context, the company believes, is how you train those problems away. Longer term, the really big bet is that the field moves toward one model that does language, pixels, audio, and actions together. Even in the near-term, though, the World Model backbone that’s used to generate pixels on a computer screen seems to also be able to help robots navigate and interact with the physical world. Teaching Robots with Video Every robotics company with ambitions of deploying at scale runs into the data wall: real-world data is scarce, expensive to collect, and a general-purpose robot needs a lot of it. In Packy’s Robotics piece with Evan, they wrote a whole section titled, “Robotics is Bottlenecked on Data,” on this idea: LLMs had it relatively easy. The entire internet existed as a pre-built training corpus. There is so much more information on the internet than you could ever imagine. Any question you might ask an LLM, the internet has probably asked and answered. The hard part was building architectures that could learn from it all. Robotics has the opposite problem. So robotics companies have to build their own internet equivalent, one demonstration at a time, often with a human in a VR headset puppeteering an arm through the same task over and over again. For jobs with long tails of weird edge cases, like household robots, Runway argues that there’s no practical way to collect all of the data you’d need, let alone enough of it. Video, on the other hand, is effectively limitless. People film and upload more of everyday life every single day than every robot lab on earth could capture through teleoperation. And as generative models get good enough to make video that’s indistinguishable from the real thing, it’s becoming even more limitless. You can start with real scenes and simulate thousands of variations, each with subtle differences. Packy here. In that same Robotics essay, Evan and I wrote a section called “Video Dreams (and Their Limits)” in which we argued that internet video isn’t enough. Video shows you trajectories and timing and the order of steps, but it doesn’t carry mass, force, friction, or stiffness. Two surfaces can look identical and behave completely differently, and the robot finds out the fun way. We cited a study in which people who watched more videos of moonwalking and juggling became much more confident and not much better. Reading lots of books makes it easier to write a good book, we wrote, but watching lots of golf videos doesn’t do much for your golf game. First, hell yeah. This is why I do these co-written essays. Smart people can disagree. General robotics is still unsolved and people much smarter than me are taking many different paths. Second, though, this isn’t the disagreement it appears to be on the surface. As one piece of evidence: Standard Bots is now testing Runway’s Praxis-1 on its RO1 arm. What resolves the seeming disagreement is that Runway isn’t betting on video instead of robot data, but before robot data. Praxis-1, which Runway unveiled at its AI Summit on September 30th, learns mostly from third-person video, built on the same large-scale pretraining as the company’s World Models. From there, Cristóbal explained, “you can take the learnings of large video data and start fine-tuning it on robotics. So you add a bit more egocentric data, and that egocentric data will help the model understand a bit better the situation that it might be in.” Then you fine-tune on the specific robot. How much robot data does that last step take? About 30 minutes of data teaches the model to control a robot body it hadn’t seen in pretraining, and in new environments it can reach strong performance on some tasks with just minutes of data. You still have to swing the club a bit, so to speak, but you need a lot less practice when you have a robust model of how the world works pre-loaded in your brain. As a test, Runway pretrained robot policies on ordinary video and, separately, on teleoperated robot video, using anywhere from ten to a thousand hours of each. Then it fine-tuned them and measured how close the robot got to putting an object where it was supposed to go. The two lines are basically on top of each other: web video worked as well as robot video. The web-video version ended up with a final placement error of 16.1 centimeters versus 16.0 centimeters for the teleoperated robot video version. That’s not nothing; for high-precision tasks, 0.1 cm might as well be a kilometer. But these models are as bad as they’re ever going to be, and teleop data isn’t getting any cheaper to capture. If they’re roughly interchangeable as a starting point, the robotics data problem looks very different than it did in January. But robots’ bodies and forms vary wildly, both from each other and from humans’. How can a model that’s mostly watched humans control a robot’s arm precisely? Part of the answer is in how Runway represents actions. Praxis-1 expresses actions in a standard end-effector format: where the gripper (the “hand” at the end of the arm) should be in 3D space, and how it should be oriented. A person’s hand reaching for a mug in a YouTube video can be described in exactly the same terms. So can a humanoid’s hand, a six-axis industrial arm, or a two-armed setup. While the embodiment changes, what the model is trying to figure out – where should the hand go next – doesn’t really. To that end, Runway’s model is currently being tested on Noble Machines’ bimanual setup, Standard Bots’ six-degree-of-freedom RO1, and Ultra’s mobile robot. In one demo, the same policy works in a controlled studio and in a home kitchen without retraining. It may be helpful to think about the three ways World Models can be used in robotics. The first is generating training data. Take a small set of real demonstrations and create variations with new objects, new lighting, and new layouts with which to train the robots on a bunch of different situations they might not see in real world training very quickly. The second is simulation, or testing robot policies inside the World Model before you put them on real hardware. This is where Runway has published its most rigorous evidence. In February, its robotics team took eight different manipulation policies from the RoboArena benchmark, all running on the same Franka Panda arm, and ran each one inside GWM-Robotics, an early version of Praxis-1. Each policy made its own decisions based on what the World Model showed it. Human graders then scored 1,450 simulated rollouts, more than 16,000 ratings in all, about ten per rollout. The ranking of the eight policies in simulation matched their ranking in the real world with a correlation of 0.95, and the graders correctly picked out both the best and the worst policy. It’s a test of ranking (which policy is better than which), not of absolute success rates, but for a robotics team deciding which of your twenty candidate policies deserves time on your three physical robots, ranking is exactly what you need. The theory here, and across the robotics work Runway is doing, is to do as much in relatively cheap simulation as you can before moving to the more expensive physical world. Runway’s model only needs a starting image, compared with competing approaches that require scanning and reconstructing every scene in 3D, which makes it relatively cheaper than even other simulation approaches. The third way is to use the World Model as the robot’s brain. This is the one that Cristóbal is most excited about. “One of the most compelling use cases that we found for these models in robots is less on the synthetic data part and more on using the video models as the brains of the robot,” he said. “So they learn not by describing the world, but by simulating the world, and they do it in real-time.” Remember our example of the person tripping on the stairs? A robot with a World Model for a brain is constantly predicting what should happen next and adjusting when it doesn’t, like that human does. That’s why recovering from failure is one of the key capabilities. If the gripper slips, the box is heavier than it looked, or the bag tips over, the model has to notice and adjust in real-time, the way you catch yourself on the stairs. Like humans, then, once embodied, these models are able to start learning both by seeing and by doing. The seeing helps them start doing, and then they learn some more from the doing. Runway thinks the first real deployments will be in enterprise: warehouses and manufacturing, fine-grained manipulation tasks where there are so many ways for things to go wrong that generalization can pay for itself. Useful home robots will take longer. That’s where we landed in the Robotics essay, too. Start where the economic value is, get paid to learn, and keep eating more. And what will the robots doing increasing amounts of work look like? Runway thinks there will be a menagerie of cheaper robots that can use tools to do specific jobs. Simpler, cheaper robots scale better, from a data perspective, than expensive humanoids that break easily. And it will become cheaper to make things to stick on robots to help them do whatever job they need. Long-term, I (Anastasis) don’t think we end up in a world of all humanoids. LLMs make it easy to design components and 3D printers make it easy to produce them, so people will design and fabricate custom robot bodies for specific jobs, all running on the same model layer. This is particularly true if teaching the brain to use new embodiments continues to take 30 minutes across arbitrary embodiments. In this world, the model that is the robot’s brain is the layer at which value accrues, so what do you do to own that layer? Runway thinks you give it away. Runway plans to release Praxis-1 with open weights in the coming months (for now, it’s with early partners), because open models give hardware developers more control over their own robots than closed ones. Open weights also help make Praxis the default starting point for anyone building a body, which means that Runway gets more data, which feeds the next, better Praxis and the next, better World Model on which it’s built. Runway’s acquisition of Kinetix, the Paris-based robotics research lab, earlier in September makes sense in that light, too. It is true, as we wrote in January, that videos don’t contain data on things like force that really matter for making robots work perfectly and predictably. Robots still need to practice in a physical body, like humans do – you’re not going to drive the ball 300 yards the first time you swing a club, no matter how many Tiger Woods videos you’ve watched – and robots, like humans, need to learn from failures in order to anticipate them the next time. But it also seems true that good priors are worth a lot, that you need less practice when you start doing something with an understanding of what great looks like, or as Runway describes it, “how objects behave, how hands move, what a task looks like partway through.” Robotics represents an enormous and obvious opportunity, and one for which World Models seem to be particularly well-suited. Babies, adults, and World Models all take in the scene around them and try to predict what to do next. If Runway can supply robots’ brains, it will build a very big business doing so. But the team is as excited by, if not more excited by, the types of products and markets it can open up if it gives people the tools to simulate every pixel. Robots acting on a physical scene and a person clicking on a digital one, after all, look the same to a World Model. Interfaces, Operating Systems, and Pixel Maximalism If it’s just predicting frames or pixels on a screen, well, why not just predict everything on a screen? A philosophical answer to my question about how Runway thinks it can build an operating system with a World Model via Continuum. When people hear “pixels,” Cristóbal said, they think of video and movies, and then the obvious next steps, like games, ads, content. Runway, though, is a Pixel Maximalism company. An operating system is pixels on a screen. Us humans, we don’t think about using a computer as the same thing as watching a movie, but to a machine, it’s all just pixels. Recall about how LLMs treat tokens. The same LLM that writes poetry can write high-performance C++, because to the model it’s all just tokens. “Models want to generalize about the world,” Cristóbal said. ”And the task itself, how we label the task, doesn’t really matter for the models. It’s just pixels on a screen.” Think about how software gets made today, even when LLMs do it. Someone designs what it should look like. Someone else (or, increasingly, the same person or just the LLM itself) translates that design into code. Everything about it – how it will look, what buttons will do, how it feels to the user – has to be anticipated and programmed ahead of time, then shipped. In the Solaris announcement, Runway describes the result as a lossy compression of all the ways someone might want to interact with a piece of software, “frozen before any user arrives.” Solaris, which Runway released at the end of August, is the first of what it calls Interface World Models. Instead of rendering an app that someone has coded, it generates the interface itself, frame by frame, as you use it. The user’s clicks and drags become conditioning for the next frame, the same way a text prompt or an image would be. (That’s at again.) You can browse a virtual clothing store where the showroom is the interface, pull a shirt off the rack, and drag it onto a photo of yourself, tuck it into your digital pants, tug it back out. You can build a salad by dragging ingredients into a bowl. Remember the guy at the top of this piece, trying on glasses and fixing his hair with his mouse? Nobody coded a “try on glasses” feature. The model just knows what happens when glasses go on a face, because it’s watched more hours of video of people putting glasses on and adjusting them than any human would willingly be subjected to. For now, Solaris pairs a World Model with an LLM. The LLM figures out what you’re asking for and decides what the application should do next (turns your words into an action); the World Model decides how that action impacts the next frame and renders it in real time. This is how Runway thinks things work in the short term. The World Model becomes the rendering layer (replacing the HTML, CSS, and JavaScript presentation layer), and the LLM handles the logic of translating your actions. Runway’s gaming work splits work the same way; the LLM writes the scenario, and the World Model renders it. Compared to the way we’re used to interacting with software, it feels magical. Runway ran a study. It gave Solaris and Claude Opus 5 the same starting image and the same interaction requests, and had 250 people make nearly 7,500 side-by-side judgments. When asked which result better followed the instruction, people picked Solaris 61% of the time to the coded version’s 24% (13% called it a tie). When asked which behaved more naturally within the scene, it was 71% to 21% (7% tie). “The second result highlights the broader difference between the two approaches,” the blog post says. “A coded interface can often reproduce the requested change, but it treats the interaction as an isolated update to the interface. With Interface World Models, because the model already understands how objects, materials and environments behave, it can generate interactions that feel coherent within the scene rather than treating each UI action as an isolated element.” The post also admits that there’s a lot Solaris can’t do well yet. Text is the big one. Stable, legible text is one of the hardest problems in video generation, and interfaces are mostly text. Trust is another; for anything instructional or commercial, a convincing wrong answer is worse than no answer at all, so Runway is researching “conditioning generation on richer verified context as the session unfolds.” Long sessions drift, and Runway is actively researching fixes here, too. Finally, generated interfaces still need to work with screen readers and the rest of the software stack, “so that flexibility doesn’t come at the expense of usability.” Generating every frame fresh is costly, although Solaris is already orders of magnitude cheaper to run than a standard video diffusion model, and the cost curve keeps bending. For now, not every website should or will be generated in real-time, but things are heading in that direction. While we’re at it, though, while we’re glimpsing at the future, why stop at generating individual websites? Why not generate the whole operating system? On October 1st, Runway previewed Project Continuum, “a new operating system research application built around real-time video interfaces.” It is a research preview and very early. But it, too, points to a new way of interacting with software and the world around us. “What it enables you to do,” Cristóbal explained, “is you can make anything an interface. Like, you take anything you want, if you wanna learn about anything in the world, you take a photo of what you want and that itself becomes an interactive application.” In this future, if you’re trying to fix your engine, you can just take a picture of the engine and then turn it around and poke at it in sim; you can even turn the engine itself into an instruction manual. There was this great launch video for the Freckle Phone for kids a couple of weeks ago, that shows kids taking pictures of the world around them and learning about what they see from an LLM; what if the pictures could turn into interactive lessons for kids to play with in whatever way they wish? This isn’t the future of software for everything software does today. Medical records and financial transactions are probably better left in a safe, structured database. “But for many others,” Cristóbal said, “I think that’s the holy grail of computer graphics. It’s like anything becomes an interface.” When the holy grail becomes practical comes down to cost. If generating an interface gets close to the cost of running a traditional app, then the use cases best served by real-time generation will be. First we shape our interfaces, then our interfaces shape us, or something. We humans alive in 2026 never felt, or barely remember, how magical it must have been moving from the command line to a GUI. We did feel what it was like moving from a GUI to a touchscreen, and putting that touchscreen in our pockets has changed the way that we interact with the world, for better and worse. Now, we mostly interact with the most powerful software ever built through a chat interface. Solaris and Continuum are bets that chat isn’t the final interface, that everything you can see might become the interface. That is a wild and beautiful future, and it sounds like an expensive one, too. Someone has to pay for the research that makes it possible, and put it in contact with the messy world of real users. Runway believes that it can solve both issues at once by building an actual business on their models today. A Business and a Lab Runway was founded in 2018 by three people who met at ITP, the graduate program at NYU’s Tisch School of the Arts: Cristóbal, Anastasis, and Alejandro Matamala Ortiz. They were artists and engineers building tools for other artists, back before AI was real. Art school is in the company’s DNA. Multimodality is more multidisciplinary than language modeling. Building models that see and hear well takes taste, that overused attribute that many claim to have but over which art students at least have a more credible claim. Runway has both artists and engineers. It’s part of why Runway is in New York, where you are smushed into a melting pot of people doing all sorts of different things (also, many World Model researchers are in London and Europe, which makes logistics easier from the East Coast than the West). It’s also why Runway decided early on to build a real business selling to customers in addition to its Research Lab. Runway released Gen-1, the first public video model. Its most recent, Gen-4.5, topped the leaderboards. Getting creative tools in creative peoples’ hands lets the company learn faster. But it also gives the company real revenue, which is accelerating faster than at any point in the company’s history. Runway is used by 95%+ of the Fortune 100, is growing ~50% month over month, has seen credit usage grow 5x this year, added over $100M in ARR in Q3 2026, and has 5x-ed its growth rate since. Customers include Dolce & Gabbana, Palo Alto Networks, Lucid Motors, Cloudflare, Yamaha, every major Hollywood studio, and every major advertising agency. Its tools have been used in Oscar-winning movies and Super Bowl ads. In the four chapters framing, Chapter 1 (media and entertainment, making the tools powerful enough for real editing workflows) are bleeding into Chapter 2 (broader content needs: marketing, advertising, full videos, whole campaigns). Runway sizes content production, things like storyboarding, VFX, and post-production, at $600B+, and marketing and advertising at $1T+. Runway Ads is the clearest sign of the move from Chapter 1 to Chapter 2. From the perspective of an art student or a researcher, it’s downright commercial. Designed specifically for performance marketers, it’s a tool Runway built because they needed it themselves. The company said they’ve been using it for a couple months to “scale ad volume 1,000% while doubling our return on ad spend, increase conversion by 34% with click-through rate holding steady, and bring our cost per subscriber down 41%.” They’re now rolling it out to early design partners, with a public launch coming in weeks. Runway describes itself as both a frontier lab and a distribution layer. There are essentially four labs building world-class video generation models: Google, ByteDance, Kuaishou, and Runway. Most other video platforms are aggregators, reselling other people’s models, which squeezes their margins and leaves them one API cutoff away from trouble. Being a lab with its own distribution means Runway sees demand signals across the whole field: what people actually make with the tools, what they’re willing to pay for, what works with customers, etc. This data helps them make both better products and better models. Runway uses the business side to feed and fund its bigger bet on World Models, which is a motion they think others will follow. Anastasis expects every AI lab to turn into a World Model lab eventually. “Our head start is that this was the plan all along. We started working on World Models right after we released our first model.” This is a common structure in deep tech. SpaceX using Starlink to fund Starship and the eventual colonization of Mars is the canonical case study here (see: Casey Handmer’s Starlink is a very big deal), and today, every business that can use near-term data center demand to bootstrap its really big long-term mission, is. Plus, you learn things interacting with the world that you don’t learn in the lab. This is why I’m bullish on Fuse’s approach to fusion – selling radiation on the path to Q>x) relative to many of its research-focused competitors. And while other video model companies may try to back into this approach, Cristóbal thinks culture will be harder to copy. “There’s a different culture in being able to build new things than being able to optimize or copy things that are already in the world,” he told me. “We’re extremely good at building new things and we’re getting much better at optimizing the new thing.” The risk, of course, is that the bigger bet gets very expensive. If World Models end up needing more training compute than LLMs, as Anastasis expects, the creative business has to keep growing fast, in a market where Google (the closest thing to an infinite money printer in tech) and ByteDance are competing for the same customers and the same researchers, and the LLM labs are entering into all sorts of byzantine financial arrangements to fund just the capacity they need. The prize, though, is just as big, if not bigger. Bigger than ads, certainly, and even bigger than robotics, potentially. Simulating the Universe Now we are getting speculative, veering into Anastasis’ vision, so we’re going to let him lay it out himself. Over to Anastasis. DeepMind’s founding mission was to “solve intelligence, and then use that to solve everything else.” For most of the past decade, that’s been the guiding idea at every AGI lab. In 2024, I wrote an essay called Towards Universal Simulation arguing that we should flip it: solve the everything else first, and use it to solve intelligence. The idea, which we’re pursuing, is to train models on a far wider range of data than we do today. Human-made media, yes, but also observations from scientific experiments and raw sensor data, all at once, with the goal of predicting a larger and larger share of everything that can be observed in the world. Then use those models to run experiments virtually, far faster than we ever could in physical reality. Most visions of the “AI scientist” imagine models that propose experiments for humans to run in a wet lab and then interpret the results. I think that gets the bottleneck backwards. Coming up with experiments is the easy part. Running them is what’s slow. The way to accelerate science is to give it an alternative to the real-world experiment, by simulating it. Large-scale model training also has an anthropomorphic bias. We’ve only really scaled up on data made by humans and legible to humans: text, images, video, sound. That makes sense if what you want is a model that talks to people. It’s far too limiting if what you want is a model that understands the universe. This past July, I extended the argument in an essay called Real-World Superintelligence. AI coding tools have been adopted faster than any software in history, and I think people are drawing the wrong lesson from that. Code worked because code comes with a perfect simulator. You write a program, run it, see whether it works, and do that millions of times. Math is the same: AlphaProof got good by generating problem variants and checking candidate proofs against a formal proof checker. I call this sandbox superintelligence: intelligence produced by repeated interaction with closed environments where you can check the answer. Real-world superintelligence is something else. It means acting better than humans in messy, partially observed systems, where you have to predict how the world will respond when you intervene, choose under uncertainty, and learn from outcomes that show up late and noisy. We’re nowhere close to real-world superintelligence. Frontier models can solve Olympiad math problems and still struggle to run a simple business. Anthropic actually tried this; in Project Vend, Claude ran a little shop in Anthropic’s office and, among other adventures, got very into tungsten cubes and lost money. It’s worse in the real world. Drug trials take many years, trading strategies are expensive to test, and company decisions can take months to play out. In many of these cases, good luck figuring out which inputs exactly led to the outcomes. They are wicked problems. Think about the leadup to Wright Brothers’ famous flight. In 1901, their glider produced only about a third of the lift that the accepted aerodynamic data said it should. Testing a redesign meant building a full-size glider, hauling it to Kitty Hawk, waiting for the right wind, and then trying to fly it. So they built a wind tunnel, a box where they could blow air over small model wings and measure lift and drag, and that fall they tested something like two hundred wing shapes without leaving the shop. AI needs wind tunnels for every domain where experiments are slow and expensive. The labs have started building them by hand. Over the past two years, there’s been a massive build-out of reinforcement learning environments, mostly software clones of things like e-commerce sites and airline booking systems, each one with its own dedicated engineering. That can work for a website, particularly now that AI can build those. For most of the things we actually care about, nobody knows how to hand-build a faithful simulator. Learned World Models are the more scalable path. With Learned World Models, we don’t have to program the rules of the world; the model learns them from observation. Fidelity improves with more data and more compute, Bitter Lesson style. And we’ve trained simulators on only a tiny fraction of the observations the world has to offer. To me, our recent releases are wind tunnels. Praxis-1 is a wind tunnel for robot policies. Solaris is a wind tunnel for computer-use agents, which are notoriously bad at simple tasks like booking a hotel, partly because they learn the specific layouts they were trained on and fall apart when the next hotel’s website looks a little different. GWM Worlds 2 is a step toward a general wind tunnel: one model handling navigation, manipulation, and human interaction. In one demo, we generated a world with a humanoid robot and two flags, told an agent to walk the robot to the red flag and then the blue one, and it did it on the first try. David Silver and Richard Sutton call this next phase the Era of Experience. “Agents will inhabit streams of experience, rather than short snippets of interaction,” they write. Their example is AlphaProof, which started from roughly a hundred thousand formal proofs written by humans and went on to generate a hundred million more through its own interaction with the proof checker. Most environments don’t work like that, though. You can’t crash a car a hundred million times. You can simulate it, though. If I’m able to predict 1,000 different versions of a car crash and I can actually see it, I can learn a lot, and a system and a robot can learn a lot about how to prepare for that. There’s a catch, though. Agents cheat. In a foundational 2018 paper, Ha and Schmidhuber asked whether agents can learn inside their own dreams. But when the researchers made the dream too predictable, the agent learned to exploit it. In their Doom-like game, an agent trained in a low-randomness dream survived for 2,086 steps inside the dream and just 193 in the actual game. It had found glitches in the dream that didn’t exist in reality. When they turned the randomness up, the agent survived 1,092 steps in the real game, well past the 750 that they counted as solving it. A dream that’s too easy to game teaches you to game dreams. A useful World Model has to stay accurate when an agent does something rare, something that barely shows up in the training data, or when it actively tries to exploit the simulator’s loopholes. That’s why real-world deployment, and a continuous loop of updating both the policy and the environment, remain necessary. You can’t game physics. We’ve built this into our work in robotics already. Our next chapter, Chapter 4, is Science. Going beyond human-scale observation means feeding models data from other scales of space and time, plus scientific data. I imagine a single model that spans physics, biology, and everything in between, at scales from the subatomic to the observable universe, and from femtoseconds to the lifespans of stars. It’s prudent to start with related modalities. But our ambition is to tackle the whole thing. You may be asking yourself, “Why would a model that has watched a lot of cat videos be any good at fluid dynamics?” Good question. It sounds like a joke, but there’s growing evidence that it isn’t. Pretraining on language helps models generate speech. Pretraining on code helps with broader reasoning. Google’s TimesFM, a forecasting model trained on a mix of time series (traffic, weather, page views, hospital data), performs about as well out of the box on series it has never seen as specialist models trained on those exact series. A group of researchers trained a transformer on continuous glucose monitor and diet data from roughly 10,000 people and found it could predict their health outcomes four years out. Back in 2022, Riffusion took an image model, fine-tuned it on pictures of sound (spectrograms), and got it to make music. I believe physics is the most striking case. In early academic work with computational physics groups, a video model pretrained on basic videos and then adapted to physics simulations beat a model trained directly on the physics data. Independent research points the same way. Last year, a team at UCLA built PhysiX, a model for physics simulation that starts from the weights of a video generation model. Their problem is the one robotics has: the internet has millions of videos, but the largest physics simulation datasets contain only tens of thousands of samples. Starting from a model that had watched the world helped. PhysiX beat task-specific baselines and the previous state of the art on The Well, a standard physics benchmark, and the authors conclude that knowledge learned from natural videos transfers to physics simulation. Transfer works for two reasons. The first is that models trained across many domains get good at learning how to learn. The second is that there are common patterns across the scales of the universe. Waves are waves, whether they’re in a pond or a plasma. Things diffuse the same way whether it’s ink in water or heat through a pan. Scientific data today lives in silos, every lab and team with its own slice, which means there’s probably a lot of untapped upside in training on all of it together. The same idea, action-conditioned prediction, is already showing up in other fields. Arc Institute’s virtual cell model predicts how a cell’s gene expression changes when you perturb it. Researchers have built learned market simulators that model how trades move prices and liquidity, and simulated societies that estimate how people might respond to news or a new policy. This is where I think World Models will ultimately have the biggest impact: anywhere an agent has to act in an unpredictable environment and learn from the data directly, whether that’s a robot, a cell, or a market. Most of the universe looks like this. The TAM is practically infinite. The endgame of a universal simulator of pixels is a universal simulator of reality. And in that end state, I think these models will be more useful than LLMs are today. Simulate Everything Packy here again. When I asked Cristóbal to distill Runway’s vision, he shared: “If every pixel will be generated or simulated, we want to become the platform and the engine in which those pixels are simulated in the first place.” The applications, he said, are impossible to plan that far out. Runway doesn’t have product definitions for next year. “There’s a lot of things that we don’t yet know what we will name and what we will be doing,” Cristóbal said. “Even if you try, it will be impossible to predict.” Naming things too early can actually be anathema to creativity and innovation. “Taxonomy is sometimes where ideas go to die,” he says. People try, though! We try to categorize a new thing using the words of the past. When people first saw daguerreotypes, they described them as mirrors with memories. When cameras followed them, people thought of them as paintbrushes, because a paintbrush was the closest thing they had. The internet, at first, was a place for publishers to put their magazines so you could flip through them on a screen and for advertisers to put banner ads. Eventually, each one became its own medium, with its own vocabulary, doing things that their early namers never could have predicted. Right now, most people put video models in a box labeled “tools for creatives,” which is fair, given where they are today, but which Cristóbal thinks is the latest version of the paintbrush analogy. “People tend to look at video models as another cool, cute, creative thing,” he said. “It’s a toy for some. You should take it much more seriously. This is going to power most intelligent behaviors very soon.” “The work we’ve been doing on World Models isn’t widely understood yet,” Anastasis adds. “People generally underestimate how far video can go.” I, Packy, was probably one of those people, and a few months ago, I was explaining video models as separate from World Models… which they are! What I underestimated, though, was video as a path to World Models, on top of which all sorts of application-specific and embodiment-specific models might be fine tuned. Runway’s bet is that a good enough video model, given actions, can become the World Model, and that on that same backbone you can render a movie, an ad, an entire desktop, a working warehouse, and maybe, eventually, a cell or a universe. The past six weeks are evidence that the bet is at least partly right. All of these applications sprouting from the same backbone, seemingly in real-time. The thing to watch from here, to double-check if the bet is right, if Runway is as good at predicting the next frame as its models seem to be, is whether each new pretrain continues to make everything on top of it better and open up new possibilities, whether the next base model makes robots more dexterous, interfaces more coherent, worlds more persistent, videos longer, and ads more performant, all at the same time, without Runway having to build each improvement by hand. And then whether the company, and its users, can build more new things that you wouldn’t expect from a video company. This is what’s happened with LLMs; while they remain spiky, broadly, each new model gets better at poetry and code and medicine simultaneously. If frame prediction works the same way, Runway’s hot streak will continue far beyond what people, outside of the company at least, are predicting. The idea, though, is that World Models can go even further than LLMs. That if you can simulate pixels, you can simulate nearly everything. Near the end of our first conversation, slowly getting simulate-all-pixels pilled, I asked Anastasis the obvious question: are we living in a simulation? “Whether or not we are, this is a very meaningful universe,” he replied. “Either it’s the root universe, or someone thought it was worth simulating.” Thanks to Anastasis, Cristóbal, Tenaya, Leo, and the Runway team for working with me on this essay! That’s all for today. We’ll be back in your inbox with a Weekly Dose on Friday. Thanks for reading, Packy