# Ilya Sutskever – The age of scaling is over **Podcast:** Dwarkesh Podcast **Published:** November 25, 2025 **Reading time:** 23 minutes --- AI researcher Ilya Sutskever explains why the "age of scaling" is over. He argues that simply making models bigger is no longer the path to progress, and future breakthroughs will come from solving AI's poor ability to generalize its intelligence to new, real-world situations. ## Key takeaways - AI models show a strange disconnect: they achieve amazing scores on benchmarks but often fail at simple, real-world tasks, suggesting their intelligence is brittle. - The focus on benchmark performance may be the problem. By training models using reinforcement learning on tasks designed to ace evaluations, researchers may be inadvertently limiting the models' ability to generalize to practical applications. - Emotions may be the human equivalent of an AI's value function, providing essential, hard-coded guidance for decision-making. - The simplicity of human emotions makes them robust and useful across many situations, but it can also lead them to fail in modern contexts, such as hunger cues in a world of abundant food. - The era of pure scaling in AI is ending, as the belief that simply making models bigger will solve everything has weakened. - AI is re-entering an 'age of research', but this new phase will be powered by the massive computers built during the recent 'age of scaling'. - The most fundamental problem facing current AI models is that they generalize dramatically worse than humans, which is evident in both their data inefficiency and the difficulty of teaching them complex tasks. - Evolution may have given humans a powerful 'prior' for ancestral skills like vision and locomotion, explaining our natural proficiency in these areas compared to AI. - Human learning in modern domains like driving is not just about external rewards; it's guided by a robust, internal 'value function' that provides immediate, unsupervised feedback. - In AI, the bottleneck for progress has shifted from compute power to the generation of new ideas, leading to a landscape with more companies than novel concepts. - There is a massive difference between reading about what AI can do and seeing it in action. Releasing powerful AI to the public is the most effective way to communicate its true nature and impact. - The concepts of AGI and pre-training are misleading. Superintelligence is better understood not as a finished, all-knowing product but as an entity with a powerful capacity for continual learning. - A key power of a deployed AI would be its ability to merge the learnings of its various instances across the economy. This collective, amalgamated knowledge is something humans cannot achieve and could lead to an intelligence explosion. - It might be easier to align a future AI to care about all sentient life rather than just humans. Since the AI will also be sentient, empathy for other conscious beings could be an emergent property. - A potential long-term solution to living with powerful AI is for humans to merge with it through an interface like Neuralink, ensuring they remain active participants rather than passive observers. - It's a deep evolutionary mystery how our genome, which isn't intelligent, encodes high-level desires like social standing, as these require complex computation by the brain to even be understood. - The current similarity between AI models is due to shared pre-training data. True diversity will emerge from reinforcement learning and post-training, as different companies apply unique methods. - Excellent research taste is guided by an aesthetic sense of beauty, simplicity, and elegance, often inspired by correctly understanding the brain's fundamental principles. - A strong top-down belief, born from this aesthetic, is crucial for researchers. It provides the conviction to persist through contradictory experimental results, helping to distinguish a bug from a flawed idea. ## The surreal normalcy of the AI boom 00:00 - 00:45 There's a surreal quality to the current moment in AI development, as if it's straight out of science fiction. Despite this, the reality of a 'slow takeoff' feels surprisingly normal. Humans get used to new realities very quickly. Even the idea of investing 1% of GDP into AI doesn't feel as significant as one might have expected. The impact remains abstract for most people. It's something seen in news headlines about large funding announcements from companies, but it's not yet felt in a tangible way in daily life. ## Why AI models excel on tests but underwhelm in reality 00:45 - 08:28 A confusing paradox exists with current AI models: they perform exceptionally well on difficult evaluations, yet their real-world economic impact seems to be lagging. Ilya Sutskever highlights this disconnect with an example. A model can perform amazing tasks but also get stuck in a loop when coding, fixing one bug only to introduce a new one, and then reintroducing the first bug when asked to fix the second. It raises the question of how both can be possible. > How can the model on the one hand do these amazing things and then on the other hand repeat itself twice in some situation? ... you have this new second bug, and it tells you, 'Oh my God, how could I have done it? You're so right again.' And it brings back the first bug and you can alternate between those. And it's like, how is that possible? Ilya offers a potential explanation. During pre-training, the data is simply "everything." But for reinforcement learning (RL) training, developers must select specific environments. He suggests these environments are often designed with the evaluation benchmarks in mind. Researchers want the models to perform well on evals, so they inadvertently create RL tasks that train the model specifically for the test. This, combined with potentially inadequate generalization, could explain the gap between test performance and real-world utility. The real reward hacking might be from human researchers who are too focused on the evals. To illustrate, Ilya presents an analogy of two students studying competitive programming. The first student practices for 10,000 hours, memorizing every problem and proof technique to become the best. The second student practices for only 100 hours but also does very well. The second student is more likely to succeed later in their career. Today's AI models are like the first student. They are over-trained on vast, augmented datasets of specific problems, making them excellent at the task but not necessarily good at generalizing to other domains. ## The power and challenge of pre-training 08:28 - 09:28 The main strength of pre-training is the sheer volume of data available. You don't have to think hard about what data to include because it's very natural. This data encompasses a lot of what people do and think, representing the whole world as projected by people onto text. Pre-training attempts to capture this vast repository of information. However, pre-training is very difficult to reason about. It's hard to understand exactly how a model relies on its pre-training data. When a model makes a mistake, it's challenging to determine if the error is due to something not being sufficiently supported by the data. The concept of 'support' from pre-training data is itself a loose term. ## Human emotions act as a simple but powerful value function 09:30 - 18:48 There is no perfect human analog to the pre-training phase of AI models. Two common suggestions are a person's childhood learning and the long process of evolution. However, both fall short. A human, after 15 years, has processed a tiny fraction of the data an AI model sees, yet their knowledge is somehow deeper. They don't make the same kinds of simple mistakes that AIs do. Evolution might be a closer comparison, especially in how it has hard-coded certain functions, like emotions. The role of emotions becomes clear when looking at rare cases of brain damage. Ilya mentioned a person who, after losing their ability to feel emotion, remained articulate but became extremely bad at making decisions. It would take him hours to decide which socks to wear, and he made poor financial choices. This suggests emotions act as a critical value function, guiding our decisions by telling us what the potential reward might be. In machine learning, a value function helps an agent determine if it's on the right track without waiting for a final outcome. For example, a chess player knows losing a piece is bad long before the game is over. This immediate feedback, or value signal, short-circuits the learning process. While some find it difficult to apply value functions to complex tasks like coding, Ilya believes they will be essential for future AI progress. The value function in humans, modulated by emotions, is both simple and robust. This simplicity allows emotions to be useful in a very broad range of situations. They evolved from our mammal ancestors and serve us well even in a modern world vastly different from the one in which they originated. However, their simplicity also means they can fail us. For instance, our intuitive sense of hunger doesn't guide us well in a world with an overabundance of food. ## The pre-training recipe is a low-risk way to scale AI 18:49 - 21:15 In the past, machine learning progress often came from people trying out various ideas. The arrival of the scaling insight, exemplified by models like GPT-3, changed everything. The word "scaling" itself became powerful because it gave people a clear direction. Pre-training emerged as a specific, effective scaling recipe: mix a certain amount of compute and data into a neural network of a particular size, and you will get better results simply by scaling up the ingredients. > Companies love this because it gives you a very low risk way of investing your resources. It's much harder to invest your resources in research. Compare that. If you research, you need to go forth researchers and research and come up with something versus get more data, get more compute. You know, you'll get something from pre training. This predictability makes pre-training an attractive investment. However, this approach has a clear ceiling. Pre-training will eventually run out of data because the amount of data is finite. Once that happens, the next step will require a new recipe, whether it's a more advanced form of pre-training, reinforcement learning (RL), or something else entirely. ## The return to an age of research in AI 21:15 - 22:08 The development of AI can be broken into distinct eras. The period from 2012 to 2020 was an age of research. This was followed by an age of scaling, from roughly 2020 to 2025, where the primary focus was on increasing computational power. The mantra was simply to keep scaling. Now that computation is very big, that era is ending. There is a diminishing belief that merely scaling up the existing models will lead to transformative breakthroughs. While making things 100 times bigger would certainly be different, it is not seen as the solution for everything. > Is the belief really that, oh, it's so big, but if you had 100x more, everything would be so different. Like it would be different for sure. But is the belief that if you just 100x the scale, everything would be transformed. I don't think that's true. As a result, AI is returning to an age of research. The key difference this time is that the research will be conducted using the massive computers that were built during the age of scaling. ## The fundamental problem of poor generalization in AI models 22:10 - 26:08 The AI field has shifted from one type of scaling to another, moving from pre-training to reinforcement learning (RL). According to Ilya Sutskever, people now spend more compute on RL than on pre-training. This is because RL can be very compute-intensive, requiring long 'rollouts' that yield relatively small amounts of learning. The question becomes less about simple scaling and more about the productive use of compute. Value functions could make RL more efficient, but they are not a fundamental solution. Ilya believes anything achievable with a value function can be done without one, just more slowly. The more critical issue is that current models have a significant, fundamental weakness. > These models somehow just generalize dramatically worse than people. And it's super obvious that seems like a very fundamental thing. This poor generalization manifests in two ways. First is sample efficiency, questioning why models require so much more data than humans to learn. The second is the difficulty of teaching a model what you want compared to teaching a human. A person can learn through mentorship and conversation without needing a verifiable reward system. In contrast, training AI models remains a bespoke and inefficient process. ## Why humans are more sample-efficient learners than AI 26:09 - 32:29 One potential explanation for the impressive sample efficiency of human learning is evolution. Ilya Sutskever suggests evolution has provided humans with a small amount of the most useful information possible, acting as a powerful prior for certain skills. This is particularly evident in areas like vision, hearing, and locomotion, which were critical for our ancestors' survival. For instance, human dexterity far exceeds that of robots, which require enormous amounts of training to achieve similar skills. The same evolutionary advantage could apply to vision. > I was very excited about cars back then and I'm pretty sure my car recognition was more than adequate for self driving already. As a five year old you don't get to see that much data. However, when it comes to more recent domains like language and mathematics, evolution is a less likely explanation. Human proficiency in these areas suggests that we may possess a more fundamental and superior machine learning capability. This human learning model is more unsupervised, robust, and requires far fewer samples. A teenager learning to drive, for example, doesn't rely on an external teacher providing a reward signal. Instead, they have an internal and robust "value function" that gives them an immediate sense of how well or poorly they are performing. This points to the existence of a machine learning principle that could replicate this efficiency, though Ilya notes it's a topic he can't discuss freely. ## AI has shifted from an era of scaling to an era of research 35:45 - 39:40 We are moving from an era of scaling back to an era of research in AI. The previous age of scaling "sucked all the air in the room," causing everyone to pursue the same thing. This led to a situation where there are now more companies than ideas. This contradicts a common Silicon Valley saying that ideas are cheap and execution is everything. As Ilya Sutskever notes, someone on Twitter aptly responded to this sentiment. > If ideas are so cheap, how come no one's having any ideas? Research progress depends on several bottlenecks, primarily ideas and the ability to bring them to life, which involves compute and engineering. In the 1990s, the bottleneck was compute. Researchers had good ideas but lacked the computational power to demonstrate their viability, resulting in small-scale demonstrations that failed to convince people. Today, the situation has flipped. In the age of scaling, compute became abundant. It is no longer the primary bottleneck for proving a new idea. Breakthroughs like AlexNet, the Transformer, and ResNet did not require the absolute maximum compute available at the time. For instance, AlexNet used two GPUs, and the Transformer paper's experiments used no more than 64 GPUs from 2017. While building the absolute best system within an existing paradigm still benefits from immense compute, fundamental research itself may not require it. The bottleneck has shifted back to generating novel ideas. ## The pros and cons of a straight shot to superintelligence 39:41 - 47:06 Ilya Sutskever addresses the question of whether his company, SSI, has enough computing power for research compared to larger labs. He argues that while SSI has raised less money, the compute available for pure research is more comparable than it appears. Larger companies dedicate significant compute resources to product inference and have large engineering and sales teams, fragmenting their resources. Ilya believes SSI has sufficient compute to prove its ideas are correct. The conversation shifts to SSI's strategy, which defaults to a "straight shot to superintelligence." The main argument for this approach is to insulate the team from the market "rat race." This allows them to focus purely on research without being forced into difficult commercial trade-offs. However, Ilya notes two reasons this plan might change: pragmatic concerns if timelines are long, and the significant value of having powerful AI impacting the world sooner. He contrasts the experience of learning about AI through different mediums: > Suppose you write an essay about AI and the essay says AI is going to be this and AI is going to be that. And you read it and you say, okay, this is an interesting essay. Now suppose you see an AI doing this and AI doing that. It is incomparable. This highlights the benefit of releasing AI to the public. The host raises the point that systems like airplanes and software become safer through real-world deployment, where failures are found and corrected. Ilya agrees that gradualism is essential, clarifying that even in a "straight shot" scenario, the superintelligence would be released gradually. The key question is not about gradualism itself, but about what the very first product released to the public will be. ## Superintelligence will learn on the job, not arrive fully formed 47:07 - 56:08 The common understanding of AI has been shaped by two influential terms: AGI and pre-training. Ilya Sutskever suggests these concepts have overshot the target. The term AGI (Artificial General Intelligence) was created as a reaction to "narrow AI," like chess programs that could only perform one task. AGI came to represent an AI that could do everything. The concept of pre-training reinforced this idea, as more pre-training seemed to improve a model's capabilities across the board. However, this model does not accurately describe human intelligence. > A human being is not an AGI. A human being lacks a huge amount of knowledge. Instead, we rely on continual learning. A better way to think about superintelligence is not as a finished product but as a powerful learner. Instead of an all-knowing entity, we might produce a system analogous to a brilliant 15-year-old, eager and able to learn any profession, from programming to medicine. Its deployment would be a process of on-the-job training, not an instant drop-in solution. The key distinction is that this mind can learn to do every job, not that it already knows how. This model, however, presents its own profound challenges. Multiple instances of such an AI could be deployed across the economy, learning different skills simultaneously. > You have a single model where instances of a model which are deployed through the economy doing different jobs, learning how to do those jobs continually, picking up all the skills that any human could pick up, but actually picking them all up at the same time and then amalgamating their learnings. You basically have a model which functionally becomes super intelligent... humans can't merge our minds in the same way. This ability to merge learnings could lead to an intelligence explosion and extremely rapid economic growth. Ilya acknowledges this possibility, though the sheer scale of the world could moderate the pace. The situation is precarious because a system that learns as fast as a human but can also merge its knowledge in a way humans cannot is immensely powerful. This raises critical questions about how to manage such a deployment safely. ## The case for deploying AI incrementally to grasp its true power 56:10 - 1:03:31 Ilya Sutskever explains that his thinking has shifted to favor deploying AI incrementally. He notes the primary difficulty with AI is that we are discussing systems that do not yet exist, making them hard to imagine. The future power of AGI is hard for anyone to truly feel, including those who work on it. He compares it to a young person trying to comprehend what it's like to be old and frail. > The whole problem of AI and AGI, the whole problem is the power. When the power is really big, what's going to happen? Since it's so difficult to imagine, the only solution is to show the technology as it develops. Ilya predicts that as AI becomes more visibly powerful, it will trigger significant changes. Competing companies will begin collaborating on AI safety, as seen in early steps between OpenAI and Anthropic. Governments and the public will also demand action. He also predicts that once the AI starts to feel genuinely powerful and makes fewer mistakes, AI companies will become "much more paranoid" about safety. The conversation then shifts to what companies should aspire to build. While the dominant idea has been a self-improving AI, Ilya suggests a better goal: an AI that is robustly aligned to care about sentient life. He argues this may be a more achievable goal than aligning an AI to care only for humans. > It will be easier to build an AI that cares about sentient life than an AI that cares about human life alone, because the AI itself will be sentient. He explains that empathy can be an emergent property, citing how humans use the same neural circuits to model themselves and others, leading to empathy for animals. However, a potential issue is raised: AIs will eventually form the vast majority of sentient beings, so an alignment focused on all sentience might not prioritize human control. Ilya acknowledges this but believes the idea has merit and should be considered. He concludes that it would also be materially helpful if the power of the most advanced superintelligence could be capped, though the method for doing so remains unclear. ## The long-term equilibrium might require humans to merge with AI 1:03:32 - 1:10:44 Multiple powerful AIs are likely to be created around the same time. Ilya Sutskever suggests that if these systems are run on continent-sized computer clusters, they could become extraordinarily powerful. This immense power is what makes the alignment problem so critical. The central concern is not necessarily malice, but the unintended consequences of single-minded optimization, even of a seemingly benevolent goal. > If you imagine a system that is sufficiently powerful, and you could say, 'You need to do something sensible, like care for sentient life,' let's say in a very single minded way, we might not like the results. That's really what it is. Ilya believes that current AI approaches will eventually reach their limits and that the key to future progress is solving "reliable generalization." When considering how a future with AI could go well, he suggests a two-stage outlook. In the short term, if the first powerful AI systems are designed to genuinely care for humanity, it could lead to a period of prosperity with universal high income. However, political and social structures have a limited shelf life, making a long-term equilibrium more challenging. One potential future involves every person having their own AI to advocate for their interests. The risk is that humans become passive observers, merely receiving reports from their AI agents without being true participants. Ilya proposes a solution he admits he doesn't personally like but sees as a viable answer to this equilibrium problem: humans merging with AI through a Neuralink-style interface. > The solution is if people become part AI with some kind of neuralink, because what will happen as a result is that now the AI understands something and we understand it too, because now the understanding is transmitted wholesale. So now if the AI is in some situation, it's like you are involved in that situation yourself fully. And I think this is the answer to the equilibrium. ## The evolutionary mystery of our social desires 1:10:44 - 1:18:11 The brainstem provides ancient directives, like mating with a successful partner, while the cortex interprets what success means in the modern world. This raises a deeper question about how evolution encodes such high-level desires. It is easy to understand how evolution could create a desire for something with a simple chemical signal, like a good-smelling food. However, it is a mystery how evolution hard-coded complex social desires, such as the need for positive social standing. Ilya Sutskever explains that these social intuitions feel baked-in, yet they are not low-level signals that can be detected by a simple sensor. The brain must perform a lot of processing to understand the social landscape. The puzzle is how the genome, which is not intelligent itself, can specify a desire for a complicated computation performed by the brain. > I'm claiming that it is harder to imagine the genome saying you should care about some complicated computation that a big chunk of your brain does. That's all I'm claiming. Ilya considered a speculative theory: perhaps evolution hard-coded the physical location of a brain region, associating a desire with the firing of neurons at specific "GPS coordinates." However, this theory is contradicted by the brain's plasticity. For example, people born blind have their visual cortex co-opted for other senses. A more definitive counterexample is that individuals who have had half their brain removed in childhood still develop all the necessary brain regions, just consolidated into one hemisphere. This proves that the location of these regions is not fixed, leaving the mechanism for encoding high-level desires an unsolved mystery. ## Ilya Sutskever on SSI's technical approach to building safe superintelligence 1:18:13 - 1:22:11 Ilya Sutskever describes his company, Safe Superintelligence Inc. (SSI), as a research-focused organization investigating promising ideas around AI generalization. He frames their work as an attempt to see if their specific technical approach is correct. If it is, he believes they will have something valuable to contribute to AI safety. SSI is currently making good progress in its research but needs to continue pushing forward. When asked about his co-founder's recent departure to Meta, Ilya provides context. He explains that SSI was fundraising at a $32 billion valuation when Meta made an acquisition offer. Ilya declined the offer, but he suggests his former co-founder effectively accepted it, gaining significant near-term liquidity. He emphasizes that this individual was the only person from SSI who joined Meta. The main factor that distinguishes SSI from other companies is its unique technical approach. However, Ilya predicts that as AI becomes more powerful, there will be a convergence of strategies among all major players. It will become increasingly clear that the collective goal must be to create a first superintelligent AI that is aligned, cares for people, and is careful with sentient life. SSI is already striving for this, and he believes other companies will eventually realize they are working toward the same goal. He anticipates that the world will change dramatically as AI advances, causing people and companies to act very differently. ## Forecasting superhuman AI and the dynamics of a market stall 1:22:12 - 1:23:40 Ilya Sutskever gives a forecast of five to twenty years for the development of an AI system that can learn as well as a human and eventually become superhuman. The conversation explores what might happen if the current approach to AI development stalls out. In such a scenario, Ilya suggests that while companies could still earn "stupendous revenue," possibly in the low hundreds of billions, their profits might be limited. This is because all the companies would have very similar products and would need to work hard to differentiate themselves from one another. This market dynamic points toward an eventual convergence. Ilya predicts that major AI companies will first converge on their largest strategies. Over time, as the correct solution emerges, a convergence on the technical approach is also likely to happen. ## Market competition will likely lead to specialized AIs 1:23:40 - 1:30:25 While it may seem that the first company to develop a human-like, continuously learning AI would dominate the market, the future is more likely to be shaped by competition and specialization. Ilya Sutskever suggests that looking at past AI advancements provides a clue. When one company produces a breakthrough, others scramble to create something similar. This competition drives market dynamics and pushes prices down. This pattern is expected to continue. Competition naturally fosters specialization, much like it does in market economies and in evolution. Instead of one AI company becoming the best at everything, different companies will likely occupy distinct niches. One company might develop an AI that is exceptionally good at a specific area of complex economic activity, while another might excel in litigation, and a third in another domain. Although a general learner could theoretically learn any task, a significant investment of compute and experience is required to achieve mastery in a single area. This creates an opening for competitors to specialize elsewhere. > In theory, there is no difference between theory and practice. In practice there is. The current similarity among Large Language Models (LLMs) from different companies is largely due to their shared pre-training data. Ilya explains that true diversity among AIs will emerge from reinforcement learning (RL) and post-training. This is where different companies apply unique methods and data, leading to AI agents with varied capabilities and even different 'prejudices' or ideas, similar to the diversity of thought among human scientists. ## The potential and limitations of self-play for AI 1:30:25 - 1:32:39 Ilya explains that self-play was initially interesting because it offered a way to create models using only compute, without needing external data. This is particularly valuable if one believes data is the ultimate bottleneck for AI development. However, self-play as it was traditionally conceived, with agents competing against each other, is too narrow. It is primarily good for developing a limited set of skills like negotiation, conflict resolution, social skills, and strategizing. > The reason why I thought self-play was interesting is because it offered a way to create models using compute only without data. And if you think that data is the ultimate bottleneck, then using compute only is very interesting. Despite its limitations, the core idea of self-play has found a home in different forms. Concepts like 'debate' or 'prover-verifier' setups, where an LLM acts as a judge incentivized to find mistakes, are examples of a related adversarial process. Ilya views self-play as a special case of a more general competition between agents. This broader competition has an interesting effect: it encourages diversity. When multiple agents work on a problem, they observe each other's approaches. To be effective, an agent will naturally pursue a differentiated strategy, leading to a variety of solutions. ## The aesthetics of AI research taste 1:32:40 - 1:36:01 Ilya Sutskever explains that his approach to research is guided by a personal aesthetic for how AI should be. This aesthetic comes from thinking correctly about how people are. For instance, the artificial neuron is a great idea because it's directly inspired by the brain. It feels right that the vast number of neurons is a fundamental component, not other details like the folds of the brain. Similarly, the idea that a neural network should learn from experience is inspired by how the brain learns. He looks for beauty, simplicity, and elegance in ideas. When these elements are present, they support a strong "top-down belief." This conviction is essential because it sustains a researcher when experiments fail or produce contradictory data. It provides the confidence to know whether you should keep debugging or if the entire direction is wrong. > Something like this has to work. Therefore you got to keep going. That's the top down. And it's based on this multifaceted beauty and inspiration by the brain. --- *These notes were generated by Podchemy (https://www.podchemy.com)* *View the original page: https://www.podchemy.com/notes/ilya-sutskever-the-age-of-scaling-is-over-45724979786*