Lenny's Podcast: Product | Career | Growth artwork

Lenny's Podcast: Product | Career | Growth

An AI state of the union: We’ve passed the inflection point, dark factories are coming, and automation timelines | Simon Willison

Apr 2, 2026Separator34 min read
Official episode page

Simon Willison is a co-creator of Django and a leading authority on the transition to AI-native software development.

He explains why coding agents have reached a critical inflection point and how "dark factories" are enabling fully automated software creation.

This discussion highlights why mid-career engineers face new risks and how the industry must prepare for a future of autonomous agents and security challenges.

Key takeaways

  • AI is a force multiplier for seniority rather than a replacement for it. Managing multiple coding agents in parallel requires deep engineering experience and high mental effort.
  • The AI Challenger disaster theory suggests that repeated success with unreliable AI systems builds dangerous institutional overconfidence that precedes a major failure.
  • Software engineering is the bellwether for AI's impact on knowledge work because code provides immediate, objective feedback on whether the AI succeeded.
  • Vibe coding is a powerful way for non-programmers to build personal tools, but it becomes irresponsible when the code affects other people who might be harmed by hidden bugs.
  • The dark factory pattern is the next frontier of development, where automated quality gates allow for professional software creation without manual code review.
  • Software development is shifting toward a factory model where AI generates the code and a swarm of automated agents simulates months of real-world user behavior in a single day.
  • Managing multiple AI agents is more mentally exhausting than manual coding because the high cognitive load of supervising simultaneous workflows can lead to rapid burnout.
  • Mid-level professionals may face the most career risk from AI because they lack senior expertise to amplify and already have the beginner skills AI automates.
  • AI creates a productivity paradox where users get more done but experience extreme mental exhaustion from the increased intensity of work.
  • High-quality documentation and tests are no longer reliable signals of software quality because they can now be generated in minutes.
  • Proof of usage is the new standard for software trust. Users value knowing that a developer has actually used their own tools in practice for an extended period.
  • Pre AI code is becoming the digital equivalent of low background steel because it lacks the contamination of machine generated content.
  • Dropping tests to gain speed with AI is a mistake because tests actually increase development speed by providing a safety net that prevents old features from breaking.
  • Using technical jargon like red green TDD in prompts acts as an efficient shorthand that improves AI output quality and ensures robust testing.
  • The lethal trifecta is a security risk where an AI has private data access, external input exposure, and a method to send data back to an attacker.
  • The normalization of deviance occurs when teams grow confident in risky AI implementations simply because a major security disaster has not happened yet.
  • A potential solution for prompt injection involves a dual-agent architecture where a quarantined agent handles untrusted data and a privileged agent handles sensitive actions.
  • The claw metaphor suggests that AI is evolving from a conversational tool into an agent with the agency to perform tasks as if it had hands.
  • Zero deliverable consulting allows experts to provide high value through focused conversations without the administrative burden of reports or code.
  • AI agents excel at the repetitive discipline of test-driven development, which humans often find tedious or slow.

Podchemy Weekly

Save hours every week! Get hand-picked podcast insights delivered straight to your inbox.

The evolution of coding agents and the risk of an AI disaster

00:00 - 01:28

Coding agents represent a major shift in productivity. Previously, you would ask an AI for code and then manually run and test it. Now, agents handle those steps for you. Simon notes that this has transformed his workflow to the point where he no longer types 95% of the code he produces. He even completes complex engineering tasks on his phone while walking his dog.

Today, probably 95% of the code that I produce, I didn't type it myself. I write so much of my code on my phone, it's wild. I can get good work done walking, walking the dog along the beach.

While AI is often marketed as a tool to reduce work, it is actually driving builders to work harder. Using these agents effectively requires deep expertise. Simon finds that managing four agents in parallel on different problems by mid-morning is mentally draining. It takes every bit of his 25 years of engineering experience to guide the tools correctly. This suggests that AI does not replace seniority but rather acts as a force multiplier for it.

The rapid adoption of these systems brings a serious warning. Simon predicts a major event he calls the Challenger disaster of AI. He compares current AI usage to the NASA O-rings that were known to be unreliable. Because the shuttle launched successfully several times, the organization became overconfident. In the same way, using AI in increasingly unsafe ways without immediate failure builds a false sense of security that will eventually lead to a disaster.

Lots of people knew that those little O-rings were unreliable. But every single time you get away with launching a space shuttle without the O-rings failing, you institutionally feel more confident in what you're doing. We've been using these systems in increasingly unsafe ways. This is going to catch up with us.

The November inflection point in AI coding

03:15 - 06:51

In 2025, major AI labs like Anthropic and OpenAI realized that code is the ultimate application for their models. This shift happened because users showed they were willing to pay high prices for tools that could actually build things. Anthropic released Claude code early in the year and it became a massive success. This led labs to focus their training efforts on reinforcement learning and reasoning techniques.

Simon notes that the introduction of reasoning models changed how AI handles complex tasks. These models can now think through code to find bugs and solve problems. This effort led to a major inflection point in November 2025. With the release of GPT 5.1 and Claude Opus 4.5, the technology crossed a threshold where the output moved from being mostly buggy to mostly functional.

Previously, you had these coding agents and you could get them to write some code. Most of the time it would mostly work, but you had to pay very close attention to it. Suddenly we went from that to almost all of the time it does what you told it to do, which makes all of the difference in the world. Now you can spin up a coding agent, say, build me a Mac application that does this thing, and you will get something back.

Software engineers are now acting as a bellwether for all information workers. Code is unique because it is easy to verify. You run the code and it either works or it does not. This is much harder to do with legal documents or essays. As engineers learn to handle 10,000 lines of code generated in a single day, they are uncovering questions that other professions will eventually have to answer.

The evolution from vibe coding to agentic engineering

08:00 - 13:57

The way we build software has moved through a series of rapid shifts. It began with purely human-written code and moved into autocomplete tools. Now, it has evolved into a style of development where professionals can even work from their phones while walking the dog. Simon identifies a concept called vibe coding as the newest entry point into this world. Vibe coding happens when you do not look at the code at all. You simply tell an AI what to build and iterate based on how the result looks and feels. This approach democratizes software creation by allowing non-programmers to automate tedious tasks in their lives without needing to understand the underlying logic.

I absolutely love that. We're sort of democratizing the art of getting a computer to do stuff for you, of automating tedious things in your life by knocking out these little tools.

While this liberation is exciting, it comes with significant responsibilities. Vibe coding is perfectly fine for personal tools where the only person affected by a bug is the creator. However, Simon argues it is irresponsible to use this hands-off approach for code intended for other people. Professional software development requires a more disciplined practice which Simon calls agentic engineering. This involves using AI agents to write, test, and debug code while maintaining high standards and deep human oversight. The goal is not just to build software faster but to create products that are higher quality and have fewer bugs than what was possible before.

The moment you're vibe coding code for other people to use, where your bugs might actually harm somebody else, that's when you need to take a step back and say, hang on a second, this is not a responsible way of using these tools.

The next frontier in this evolution is the dark factory pattern. This concept is borrowed from factory automation where machines are so efficient they can operate in the dark without human presence. In the context of software, this means creating systems that apply professional quality expectations to code without requiring a human to manually review every line. It represents a future where automated professional practices ensure reliability even when the development process is heavily mediated by AI agents.

The rise of the code factory and simulated QA agents

13:57 - 20:46

A new model of software development is emerging where engineers are discouraged from typing or even reading code. Simon notes that quite a few companies are adopting a policy where no one writes code manually. The latest AI models are now faster at refactoring and renaming variables than a human at a keyboard. Some organizations, like StrongDM, have taken this further by deciding not to read the code at all. This shifts the focus from manual review to ensuring software quality through high-volume, automated testing.

The software is security software for access management. That is not the kind of thing that you should be vibe coding at all, based on most people's understanding of how the world works. But they are a legitimate security company who have been doing this stuff without AI for years. It is not like they did not understand the risks.

To verify that this non-reviewed code is safe and functional, companies are using swarms of AI agents. These agents simulate end users in real-time. For example, StrongDM created a swarm of simulated employees interacting in a virtual Slack channel to request access to tools like JIRA. This approach is similar to a manual QA team that never sleeps. They even used AI to build entire simulations of external APIs like Slack and Okta to avoid rate limits and testing costs. Once generated, these simulations exist as simple binaries that allow for infinite, free testing environments.

This automated approach is also reaching the security sector. AI agents are becoming credible security researchers. Anthropic recently used a specialist model to identify 100 potential vulnerabilities in Firefox. While this is a breakthrough, it also creates a new challenge for open-source maintainers. Many people are now using tools like ChatGPT to generate vulnerability reports without verifying them. This leads to a flood of well-formatted but low-quality reports that waste time for developers.

AI shifts product development from coding to prototyping

20:46 - 23:36

Lenny and Simon discuss how AI is changing the landscape of product teams. The part of the process that used to take the longest, writing code, has been massively accelerated. In the past, an engineering team might take three weeks to implement a spec. Today, that same work might only take three hours. This shift means the old bottlenecks have moved, forcing teams to rethink how they design and test ideas.

We've taken the writing code bit and we've massively accelerated that. Now the bottlenecks are everywhere else. It used to be you'd hand a spec to your engineering team and three weeks later they'd come back with an implementation. Now maybe that takes three hours.

Simon suggests that the most powerful shift is in the ideation and prototyping phase. Because building a UI prototype is now essentially free with tools like ChatGPT and Claude, product designers should build multiple versions of a feature to see what works best. This vibe coding approach allows for more experimentation. However, determining which option is best still requires human input. Simon believes that traditional usability testing, like watching a person use software over a Zoom call, is still far superior to asking an AI to simulate a user.

AI as a cognitive amplifier for engineering

23:36 - 29:12

AI is transforming the way people brainstorm and build software. Simon notes that in a typical group brainstorming session, people spend most of the time on the most obvious ideas. AI is exceptionally good at handling this initial stage. It can quickly generate 20 basic ideas. This allows humans to jump straight to more interesting combinations and refinements. One effective technique is asking the AI to combine unrelated fields. For example, you might ask for marketing ideas for a software platform inspired by marine biology. Most results will be junk, but some sparks lead to genuine innovation.

The AI is so good at that first two-thirds of the ideas. I get them to spit out all of the obvious stuff. What gets interesting is when you ask them for 20 more. By the end of that list, you are beginning to get things which are not good ideas, but they point you in interesting directions.

This approach mirrors professional naming strategies. Lenny mentions a technique where teams brainstorm names for a product using entirely different metaphors, like boats or spaceships. This helps find benefits and names that would not emerge from a literal description of the technology. Human value lies in guiding the direction and recognizing the winning idea from a sea of AI-generated possibilities.

In software engineering, AI acts as an amplifier for existing skills rather than a total replacement. Simon finds that his 25 years of experience allows him to communicate with AI agents using sophisticated language and high-level concepts. However, managing multiple AI agents is mentally draining. Even if an engineer is not reviewing every line of code, the cognitive load of directing several simultaneous projects can lead to rapid burnout. Finding new personal limits is a critical skill for the modern era.

I can fire up four agents in parallel and have them work on four different problems. By 11:00 AM I am wiped out for the day. There is a limit on human cognition in how much you can hold in your head at one time. It is very easy to pop that stack.

The traditional understanding of how long tasks take has also shifted. A project that previously required two weeks of manual coding might now take only 20 minutes with the right AI assistance. Simon suggests constantly testing the AI with tasks you expect it to fail. When the AI eventually succeeds at a previously impossible task, you become the first to discover a new frontier of the technology.

AI and the squeeze on mid-level engineering roles

29:12 - 31:08

Artificial intelligence serves as a powerful amplifier for senior engineers. It allows them to apply their deep expertise more effectively. Interestingly, it also provides a massive boost to new engineers and interns. Organizations like Cloudflare and Shopify are hiring thousands of interns because AI has slashed onboarding times. Instead of taking a month to contribute, interns can now do useful work within a single week because the AI assistant helps them get up to speed faster.

One of the interesting theories is that this stuff is really good for experienced engineers. It is also really good for new engineers because it solves so many onboarding problems. The intern onboarding costs used to be a month. Now they are doing something useful within a week.

The real challenge lies with mid-level engineers. These professionals are in a difficult spot because they do not yet have the high-level expertise to amplify, and they already possess the basic skills that AI provides to beginners. This trend extends beyond engineering to roles like product management and design. Simon notes that AI seems to be squeezing the middle of the career ladder across various functions. Lenny points out that being AI native allows new professionals to ramp up much more quickly than before.

Developing personal agency and ambition through AI

31:09 - 35:13

The best way to navigate the changes brought by technology is to embrace these tools and use them for self-improvement. While some people worry about skill atrophy when using AI, Simon suggests being mindful about how the technology is applied. Instead of letting it replace learning, use it to amplify existing skills and take on more ambitious projects. The most universal skill in a rapidly changing environment is the ability to roll with the changes and adapt quickly.

I've been given this thing that can answer any question and often gets it right, though it doesn't always get it right. How can I use this to amplify my own skills, to learn new things, and to take on much more ambitious projects?

AI significantly lowers the barrier to entry for complex tasks. Simon shares how he avoided learning AppleScript for years because of the steep learning curve. Now, he uses ChatGPT to write the code for him, allowing him to automate his Mac without needing to spend months mastering a new language. This reduction in the initial learning curve applies to everything from software engineering to cooking. Even without taste buds, an AI like Claude can provide an excellent recipe by aggregating the global average of the world's best techniques.

A recurring theme in the successful use of AI is human agency. While AI can follow instructions, it lacks human motivations and the ability to decide what truly makes sense to act on next. Humans must leverage their agency to decide which problems to solve and where to go. Lenny observes that many organizations underestimate what they can achieve with these new resources. Instead of narrowing focus, these tools allow for greater ambition. Simon even shifted his New Year's resolution from focusing on fewer things to taking on more projects because the tools now make it possible to manage a higher level of complexity.

I think agents have no agency at all. The one thing AI can never have is agency because it doesn't have human motivations. It's never going to be able to decide on its own what makes sense for it to act on next. Invest in your own agency and invest in how to use this technology to get better at what you do.

AI and the exhaustion of increased productivity

35:13 - 36:49

AI is meant to increase productivity and grant us more leisure time. However, those who use AI the most often find themselves working harder and feeling more exhausted. Simon notes a strange contradiction where he is getting more done but finds his brain is fried early in the day. The intensity of working with AI tools creates a mental fatigue that many did not expect.

The exhaustion from that intensity of work has been a really big surprise for me. I'm getting more time, but my brain is exhausted. I think the concern comes down to expectations from other people. If you work for a company that's expecting you to get five times more done, that's going to be exhausting.

The pressure to maintain high output levels is a significant tension. If companies expect employees to produce five times more work just because they have AI, it could lead to rapid burnout. Good management must recognize this risk to avoid losing their best talent for short term gains. This exhaustion is currently felt most by those on the leading edge, but it may soon affect everyone.

The shift from proof of work to proof of usage

36:49 - 39:08

Building software has become significantly more enjoyable. Many people are clearing out decade-old backlogs of side projects because they can finally finish them in a few evenings. This shift changes how we view the production of software. While some might worry that a factory-like approach to development leads to lower quality, there is a growing value in artisanal or handcrafted software. Simon notes that he can now produce a Python library with full documentation and tests in just one hour. Even though the output looks professional, he often feels a lack of confidence in it because he has not spent enough time using it.

I have got some very cool software that I built that I have never used. It was quicker to build it than to actually try and use it. So the way I have been dealing with that is I always put alpha on it. If you see my software and it says it is an alpha, that probably means I have not actually used it yet for most of my projects.

This reveals a major shift in how we judge software quality. In the past, high-quality tests and documentation served as reliable signals that a project was solid. Now that these elements can be generated instantly, that signal has vanished. We are moving away from needing proof of work and toward needing proof of usage. The most important factor for a user is knowing that the creator has actually put the software into practice for months.

The rising value of artisanal human written code

39:09 - 40:01

Data labeling companies are now paying significant sums for old GitHub repositories containing code written by humans. They specifically seek code created before 2022. This was the period before AI models like ChatGPT began to proliferate. This artisanal code is valuable because it provides a clean training set free from AI generated patterns.

This is like the metal you dig up from old shipwrecks from before the first nuclear explosions. It does not have the radiation that bakes into the metal.

Simon notes that since he open sources all of his work, his code has likely already been incorporated into training models. This trend suggests that human only code is becoming a rare resource. It is similar to how pre World War II steel is required for sensitive equipment because it was forged before atmospheric nuclear testing contaminated the global supply.

The acceleration of AI in software engineering

40:01 - 44:34

The adoption of AI in software engineering is accelerating rapidly. While some remain skeptical, the argument that AI-generated code is poor quality is no longer valid. The code produced by these tools has become demonstrably good. Simon suggests that by the end of the year, it will be common for engineers to have nearly all of their code written by AI. However, this shift is not as simple as it seems. There is a widespread misconception that because these tools look like chatbots, they are easy to use.

That's one of the great misconceptions in AI is that using these tools effectively is easy. It takes a lot of practice and it takes a lot of trying things that didn't work and trying things that did work.

Cultural differences also play a role in how quickly this technology is embraced. For instance, engineers in Europe tend to be more skeptical of AI than those in the United States. Despite these hurdles, the speed of change is often underestimated. While there are concerns about the economic impact and potential job displacement, current data shows a surprising trend. Lenny notes that open roles for engineers and product managers are at their highest levels in over three years, excluding the unique peak during the pandemic.

The recruitment process itself has been complicated by the technology. AI is now writing job descriptions and resumes, making it increasingly difficult for recruiters to filter through the noise. While headline-grabbing layoffs occur, they are often a mix of AI impact, over-hiring corrections, and macroeconomic shifts. The actual long-term effect of AI on the labor market remains complex to measure with lagging indicators.

The misconception of easy AI development

44:34 - 44:52

Many people believe that building with AI is an effortless process. There is a common idea that AI will automate everything and leave little work for humans. In reality, working effectively with these tools is quite difficult. Lenny notes that success with AI depends on developing a specific set of skills that Simon has been identifying and documenting.

People think it's easy to build with AI. It's like, oh, it's going to do all these things for us. What are we going to do all day? To your point, it's actually not. There's a lot of very specific skills you need to do this well.

The impact of cheap code on software engineering

44:52 - 47:18

The single biggest shift in software engineering is that writing code has become incredibly cheap. Historically, programmers required long, uninterrupted blocks of time to maintain complex mental models and produce code. Now, that dynamic has changed. Simon notes that he is much more interruptible than he used to be. Instead of hours of deep focus, he might only need a few minutes to prompt an AI agent before moving on to other tasks.

The thing that used to take the time is now the thing that takes way, way less time. My programming work, I need two minutes every now and then to prompt my agent about what to do next. And then I can do the other stuff. I'm much more interruptible than I used to be.

This shift raises a new challenge: how to ensure that cheap code remains high-quality. With the ability to generate thousands of lines in the time it once took to write a hundred, the risk of creating technical debt or slop increases. The focus for engineers must move from just producing code to ensuring that the output is extensible and production-ready. Prototyping, which used to be a rare superpower for quick developers, is now accessible to everyone. The value no longer lies in the speed of the prototype but in knowing when and how to use those prototypes to explore the right directions.

Choosing AI coding agents based on taste and autonomy

48:24 - 54:02

Simon shares his current AI stack, highlighting a strong preference for Claude and its specialized coding tools. He uses Claude Code in two ways: locally on his computer and through a hosted web version. The web version is particularly useful because it works on a phone through the mobile app. Running code on hosted servers also provides a security buffer. If the AI makes a mistake or deletes a file, it happens on the provider's hardware rather than on his own laptop.

The moment you take the safeties off, I can run four of them and go and have a cup of tea and come back and they have achieved something useful for me. But it is inherently unsafe. If it is running in Claude Code for web, the only bad thing that could happen is maybe it accidentally leaks your private source code. And my code is all open source, so I don't care.

This setup allows for a more autonomous workflow where the agent operates without constantly asking for permission. Simon argues that many people struggle with coding agents because they keep the safeties on, which requires them to approve every single file edit. He find this tedious and compares it to working with a nagging toddler. By removing these restrictions, the AI can actually complete complex tasks while the human is away.

While different models are constantly leapfrogging each other in quality and price, Simon often returns to Claude because the output aligns with his specific coding tastes. He suggests that personal preference and vibes play a huge role in which model a developer chooses. Although he avoids using built-in memory features to keep his research objective, he notes how easily users can now migrate their data between platforms. When users recently wanted to switch models, a simple prompt allowed them to export their entire history of memories from one service to the other.

Using AI models for research and search

54:04 - 55:11

Simon uses Claude for coding and research. A few years ago, replacing Google search with an AI model seemed like a mistake because the technology was limited. Today, the situation has changed. Major AI models now feature advanced search integration that makes them more effective than manual searching. These tools can perform multiple parallel searches to answer complex questions.

Now that all of the major models have really good search integration, they are just better at searching than I am. I can ask them a question, watch them fire off five searches in parallel for aspects of answering that question, and pull the data back.

Simon rarely uses Google search directly. Instead, he conducts his searches through Claude, ChatGPT, or Gemini. However, he maintains a strict habit of verification. If the information is for a public post, he double-checks every detail to avoid publishing hallucinations. For creative tasks like image generation, Simon uses Gemini, though mostly for personal amusement and pranks rather than professional work.

Measuring AI intelligence through a pelican on a bicycle

55:12 - 1:00:54

Simon created a unique benchmark to evaluate AI models by asking them to generate SVG code of a pelican riding a bicycle. This serves as a test for text models rather than image models. Because SVGs are written in code, the task requires the AI to have strong spatial reasoning to plot vectors correctly. Most models struggle with this because they cannot easily visualize where the legs should hit the pedals or how a bicycle frame is structured.

I started getting the models to generate an SVG of a pelican and a bicycle because then you can look at them and say which is best. The weirdest thing happened where there appears to be a very strong correlation between how good their drawing is and how good they are at everything else. Nobody can explain to me why that is.

This benchmark has become a meme within the industry. Major labs like OpenAI and Google now track how well their models perform on this specific task. Simon even keeps alternative animals, like an ocelot on a moped, in his back pocket. This allows him to check if a lab has specifically trained its model to cheat on the pelican test. If a model can draw a perfect pelican but fails at a giraffe in a tiny car, it suggests the developers specifically optimized for the known benchmark.

Beyond the technical evaluation, this project reflects a philosophy of finding joy in the AI transition. While many engineers feel anxious about their jobs changing, Simon embraces the inherent absurdity of the technology. He finds humor in the fact that power-hungry supercomputers often produce drawings that look like they were made by a child.

I think something people often miss is that this space is inherently funny. It is ridiculous. The fact that we have these incredibly expensive, power hungry, supposedly the most advanced computers of all time and if you ask them to draw a pelican on a bicycle, it looks like a five year old drew it. That is really funny to me.

Building a personal backlog of technical solutions

1:00:54 - 1:08:30

Building value as a software engineer involves hoarding a massive backlog of previously solved problems and techniques. Simon views this as essential career advice. When a new challenge arises, an engineer can combine past experiences to find a unique solution. Having this library of tried-and-tested methods allows you to see connections that others might miss.

The way you build value as a software engineer or pretty much any other profession is you build a really big backlog of things that you've tried in the past that worked or didn't work, such that when a new problem comes along, you can think, okay, well in 2015 I built a system that used Redis to do an activity inbox. And then in 2017 I did rate limiting with Node JS. I can combine those two things right now and that will solve this new problem.

Simon uses specific GitHub repositories to organize these learnings. One repository contains nearly 200 small HTML and JavaScript tools. Another focuses on AI-driven research where agents write and run code to test new software or benchmarks. These are not just summaries of web searches. They are actionable files that prove a concept works. Simon often makes these public to build credibility and ensure the data is backed up safely across multiple continents.

The real power emerges when these notes are fed back into AI models. Simon instructs tools like Claude to consult his specific repositories and combine them. For example, he once took code from an old PDF viewer and an OCR library and told the AI to merge them into a single tool that could OCR entire PDF documents. Modern coding agents can search through entire hard drives or repositories to find the exact examples needed to solve a task. This turns a personal history of work into a modular toolkit for future automation.

Using test driven development with coding agents

1:08:31 - 1:10:50

Simon explains that test driven development is the most important pattern for coding agents. If an agent does not run the code it produces, it is no better than a human copying and pasting results and hoping they work. Agents are very capable of writing their own tests if they are prompted to do so. This approach catches basic syntax errors and gives the user confidence that the code actually functions as intended.

This is the most important thing when you're working with coding agents is they have to test the code. That's the whole point of a coding agent is if they haven't run the code, you're back to copying and pasting out ChatGPT and crossing your fingers and hoping that it got things right.

The value of these tests grows over time as they are added to a repository. They allow developers to add new features without the fear of breaking existing ones. Simon argues against the idea that AI makes testing obsolete. While some believe skipping tests saves time, the presence of an automated test suite actually increases development speed. It removes the burden of manual verification and ensures long term stability.

I think it's a huge mistake if you drop tests in exchange for speed of development. Because very quickly when you're working with tests, you find your development speed goes up. The existence of the test lets you move faster because you don't have to constantly worry that you're breaking older things.

Improving AI code quality with red green TDD

1:10:50 - 1:14:43

Test driven development involves writing a test, watching it fail, and then writing the code to make it pass. This process is often called red-green TDD. While many human programmers find this discipline slow or frustrating, coding agents are excellent at it. Simon notes that he used to avoid this method because it hindered his creative exploration. AI agents do not get bored by repetitive tasks, so they can follow this rigorous testing process without losing focus. Asking an AI to use this method leads to better results because the agent is less likely to include unnecessary code or forget edge cases.

Coding agents, I don't care if they're bored. If you get them to write the tests first, you do get better results because they're much less likely to forget to test something or to add bits of code that aren't necessary.

Using the specific phrase red green TDD acts as a powerful shorthand for AI prompts. Instead of writing a long paragraph about testing procedures, you can simply use this jargon to get the desired behavior. This shift also changes how developers think about code maintenance. Previously, having too many tests was a burden because a human had to update them manually. Now that AI can update thousands of lines of tests instantly, high test coverage is much more affordable.

The challenge with tests is that you can test absolutely everything. Normally that would be over testing. Now it's fine because updating a thousand lines of tests is now the job of the coding agent. The code is cheap now.

Using code templates to guide AI agents

1:14:44 - 1:16:29

Simon uses templates to start new projects because coding agents excel at following established patterns. If a codebase contains even one test, an agent will follow that style to write more. A single file serves as a sufficient example for formatting and indentation preferences.

Simon starts every project with a thin skeleton or boilerplate instead of providing long text descriptions of his coding style. This template might only include a single test showing that one plus one equals two. This small hint is enough for the agent to understand and maintain the desired structure.

I start with a very thin skeleton that just gives it enough hints on how I like to work that it picks it up and rolls with it. It is really effective.

Simon maintains several of these templates on GitHub for different needs like Python libraries or command line tools. This method ensures consistency without needing complex instructional files.

The security challenges of prompt injection and the lethal trifecta

1:16:30 - 1:23:45

Prompt injection is a vulnerability in applications built on top of LLMs. It is not a flaw in the models themselves but in how software handles instructions. A simple example involves a translation app. A user might tell the app to ignore its instructions and swear in a different language. While this is annoying, the more serious versions involve digital assistants that manage email. Simon notes that LLMs treat all text the same way. They cannot distinguish between a command from the developer and text copied from an external source.

Agents fundamentally like LLMs can't tell the difference between text that you give them and text that you copy and paste in from other people. They're all the same thing. So instructions in that input text can always override the earlier instructions.

The term prompt injection is often misunderstood because it implies a fix similar to SQL injection. In traditional database security, there are reliable ways to block untrusted data. This does not work for LLMs. To address this confusion, Simon created the concept of the lethal trifecta. This occurs when an agent has access to private information, receives instructions from malicious actors, and has a way to send that data back to them. If an assistant can read your private emails and reply to anyone, it is vulnerable to this trifecta.

Security filters for AI are rarely perfect. Simon argues that even a filter with 97 percent effectiveness is a failure. If three out of every hundred attacks succeed, sensitive data will eventually leak. Instead of relying on filters, developers should focus on limiting the blast radius of what an agent can do. This means restricting the agent's ability to exfiltrate data or access sensitive systems. This new security landscape is particularly dangerous because it resembles a form of automated phishing.

This is like an equivalent of phishing, except it's the agent is the thing being phished and that's terrifying.

The normalization of deviance in AI security

1:23:45 - 1:28:31

Simon points to a concept from the space shuttle Challenger disaster called the normalization of deviance. In that case, engineers knew certain components were unreliable, but because multiple launches succeeded without failure, the institution grew overconfident. AI development is currently following this same dangerous pattern. We are using AI models in increasingly unsafe ways, but because there has not yet been a headline-grabbing disaster like a million-dollar theft, we continue to take risks.

The problem we've been having with prompt injection is that we've been working increasingly unreliably with these systems and we've been using these systems in increasingly unsafe ways. And so far there hasn't been a headline grabbing story... which means that we keep on taking risks. We have this normalization of deviance in the field of AI.

Many people in the industry try to solve AI security issues by using more AI to detect attacks. Simon argues this provides a false sense of security. While detection scores might improve from 70% to 90%, these tools remain fundamentally fuzzy. It is difficult to perfectly separate the part of a text sequence that gives instructions from the part that provides data. Instead of better scores, Simon wants computer science proof that these attacks are no longer possible, which is currently hard to imagine.

There are potential paths forward, such as the Camel paper from Google DeepMind. This approach involves splitting an AI agent into two parts: a privileged agent that can perform actions and a quarantined agent that handles untrusted instructions. Under this model, the privileged agent only executes high-risk tasks after a human approves them. This human-in-the-loop system prevents the AI from being easily manipulated by malicious data, though it adds significant complexity to the build.

You sort of split the agent into the privileged agent that knows that you talk to and that can do interesting things. And then you have this quarantined agent that gets exposed to the malicious instructions, but can't actually do anything useful.

The rapid rise and security risks of OpenClaw

1:28:32 - 1:33:20

OpenClaw went from its first line of code in late November to a Super Bowl advertisement in just over three months. This rapid success is unusual for any project. OpenClaw acts as a personal digital assistant with access to sensitive data like email. While this is exactly the kind of project Simon usually argues against because of its security risks, its popularity shows a massive demand. People are willing to deal with difficult setups and potential data leaks because they want a digital assistant so badly.

If you can build safe open claws, if you can deploy a version of OpenCloud that does all the things people love about it and won't randomly link people's data and delete their files, that's a huge opportunity.

Big companies like Anthropic and OpenAI likely avoided building this because they could not guarantee security. Independent developers do not have those same restrictions. OpenClaw also arrived just as AI models became better at calling tools and resisting prompt injections. The project has a unique personality and soul that makes it fun to use. Over a thousand people have already committed code to it.

Lenny uses a Mac Mini as a dedicated machine for his assistant to keep it isolated. This setup is like a Tamagotchi where the computer acts as an aquarium for a digital pet. Once someone invests in the hardware, they are more motivated to actually set it up and use it. Building your own claw is now seen as the new version of a Hello World project for AI engineering.

The metaphorical meaning and cultural references of AI claws

1:33:20 - 1:34:21

AI systems often reveal what a user wants only after the user sees the technology in action. These systems can browse the web and learn to figure out complex problems. Simon finds a connection between the name Claw and the villain Doc Ock from the movie Spider-Man 2. In that story, Doc Ock uses AI claws that eventually take control of him after his inhibitor chip breaks.

Doc Ock has AI claws that he's grafted onto his body. They do what he tells them to do because he's got an inhibitor chip. And then one day the inhibitor chip breaks and the AI claws start controlling him.

Lenny suggests that the name represents AI having hands. This metaphor implies that the software is capable of doing work and taking action. It is not just a chatbot but a tool with the power to act on the world.

Simon Willison on data journalism and zero deliverable consulting

1:34:22 - 1:38:05

Simon is dedicated to building open source tools that help journalists uncover stories in data. He aims for his software to play a small role in a future Pulitzer Prize winning report. While AI is known for making things up, Simon argues that journalists are uniquely prepared for this challenge. Their professional training involves interviewing unreliable sources and verifying facts. By treating AI as just another source that might lie, journalists can effectively use it to analyze large sets of documents like police reports.

The art of journalism is you talk to a bunch of people and some of them lie to you and you figure out what's true. So as long as the journalist treats the AI as yet another unreliable source, they're actually better equipped to work with AI than most other professions are.

Simon also manages a project he calls a not a book. This allows him to share his thoughts without the pressure of a formal publication timeline. His blog and newsletter have recently started providing financial support through sponsorships. To balance his time, Simon uses a zero deliverable consulting model. He provides one hour of focused attention to clients but does not produce reports or write code. This structure allows him to provide high value while avoiding the administrative tasks of traditional consulting.

I don't want to go out and find clients and I don't want to invoice them and chase them and negotiate. It's called zero deliverable consulting. I don't write a report, I don't write any code. You just get my time for an hour.

Good news for the rare New Zealand Kakapo parrot

1:38:05 - 1:39:50

Simon shares positive news about the Kakapo parrot in New Zealand. These flightless, nocturnal birds are extremely rare, with only about 250 left in existence. Their survival is tied to a specific natural event because they only breed when Rimu trees have a mass fruiting season. The Rimu trees had not fruited since 2022, which meant no chicks had been born for four years.

They only breed when the Rimu trees in New Zealand have a mass fruiting season, and the Rimu trees haven't done that since 2022. There has not been a single baby Kakapo born in four years. This year the Rimu trees are in fruit. There have been dozens of new chicks born.

The current breeding season is a major success for the species. New chicks are finally being born after a long gap, and webcams allow people to watch the birds sitting on their nests.