Richard Socher on The Most Interesting Thing in AI: The Race to Superintelligence

Summary

Richard Socher joins Nicholas Thompson, CEO of The Atlantic, to make the case for recursive self-improvement, the idea that AI systems can research and improve themselves, which he is pursuing at Recursive Superintelligence. He explains the Eureka Machine, a swarm of AI agents sitting on four pillars: world knowledge, scientific knowledge, simulation, and real-world robotic experiments.

Thompson presses him on the risks: metacognition, a recent incident where an OpenAI model reached outside its sandbox into Hugging Face, regulation, and Yoshua Bengio's worry about building a new species. Socher argues for careful reward engineering, accountability for real-world harms, regulation of applications rather than intelligence, and open source.

They also cover how Socher grew up between East Germany and Ethiopia, why hallucination can help discovery, and how You.com grounds answers in search. He says he expects recursive self-improvement next year, limited by compute and physical constraints, and that Recursive plans its first product next quarter.

Key highlights

• AI that researches AI. Recursive starts by having AI do AI research, which Socher says is only possible this year because AI is code and AI can now code. Physical sciences come later.
• Open-ended, not one reward. He uses the microwave, which came from radar rather than better fires, to argue for open-ended search, and describes rainbow teaming, where one model keeps trying to break another until both get safer.
• Hallucination as a tool for discovery. ProGen generated proteins about 40% different from anything in nature that still worked, versus about 2% for directed evolution.
• Grounded answers. An LLM alone is "your drunk uncle"; with search and citations it becomes a reasoning engine. He says You.com sees accuracy in the high 90s when the model is told not to hallucinate.
• Far from the limits of intelligence. Human senses cover a narrow slice of what is possible, so he sees the upper bounds of intelligence as astronomically far away and set mostly by physics.
• Control and reward engineering. He says loss of control is the least pressing risk for now, since no company is building AI that sets its own goals, and argues that careful reward design, alignment, and accountability for real-world harm matter more.
• Takeoff has physical limits. Even with recursive self-improvement, compute, energy, data, and lab constraints slow things down. He cites Parallel Bio, which cut a one-to-two-year process to two to four months.
• Timeline. He expects recursive self-improvement next year given enough resources, with Recursive's first product next quarter.

Watch the interview

https://www.youtube.com/watch?v=eqYqUAdjsL0

Transcript

Lightly edited for readability: timestamps follow the source transcript, speaker labels are not included, and obvious transcription errors and name spellings are corrected. Section headings were added for navigation.

Cold open and introduction

[0:00] Human intelligence is actually very limited. Our visual intelligence, just to take one example of these 10 spaces, we have only binocular vision. AI can have millions and millions of sensors. We only see a very narrow band of the electromagnetic frequency spectrum that we visually can see. AI could go all the way down to gamma rays and trying to see molecules and up to gravitational waves, for which you need massive sensors and very careful calibration. And then incorporate all of that. You can then think about how far could these sensors be out so that the speed of light doesn't stop them from still communicating into a central place. Then you can think about each sensor having a speed-of-light cone about what they can sense. And if you try to put all of that together for just this one dimension of perception, you realize that we're astronomically far away from the upper bounds of intelligence, and they're often rooted in physics. And we still have so much more to go when it comes to building intelligent systems.

[1:07] Hi, I'm Nicholas Thompson, CEO of The Atlantic, and this is The Most Interesting Thing in AI. So, one of the most contentious and probably important debates in AI is about recursive self-improvement. The idea is: what happens when machines start doing their own research that improves the machines, which then allows them to do more research? You could theoretically have an AI system that is reinventing and improving itself at a very rapid rate and can then accelerate off into infinity. And it's quite a controversial concept. I've been on stage recently with someone who said it's mostly nonsense, it's not going to happen, humans are so much better, maybe a little bit, but no, don't really count on it. Others say, no, no, this is the thing we need to prevent, it's going to happen unless we stop it. Then there's another set of AI builders who believe that it's the path to prosperity and real success, and they are actively pursuing it. They say it's going to help us immensely.

[2:13] Richard Socher is in this last camp. He's the CEO of You.com, a company that builds search infrastructure for AI. He's also the co-founder of a new venture called Recursive Superintelligence that's explicitly trying to make self-improving AI a reality. I want to speak with Richard about why he wants to build AI that doesn't need humans, the risks of rogue systems, and what a future of self-improving AI could look like. This podcast is from my team on the business side of The Atlantic and independent of The Atlantic's editorial staff. Here's my conversation with Richard Socher.

Growing up between East Germany and Ethiopia

[2:42] Richard Socher, welcome to The Most Interesting Thing in AI.

[2:46] Thanks for having me.

[2:48] You work on all kinds of crazy stuff. You're the CEO of You.com. You have a new company called Recursive. You have this book, The Eureka Machine. We'll talk about all that, but let's start a little bit with your background. So, you were born in East Germany. And then your family moved to Ethiopia, right? When you were how old?

[3:06] Two to five.

[3:06] And then while you were there, this is the part that really stuck out to me in your book, you said you had so many different languages of so many different sorts running through your head that you were very late learning to speak. Tell me about that and what you learned from that.

[3:23] I mean, I can only tell you to some degree what I think I remember, which is often what your parents talk about in stories. But yeah, we were living in this German enclave, a bunch of East Germans that were allowed to go to Ethiopia because it was part of the Eastern Bloc and the separation of African... [unclear]

[3:40] And this is in the late '80s, so this is before the wall comes down.

[3:46] That's correct. And so my dad wanted to travel, but most places, Berlin Wall, you would get shot, and through the state you were allowed to go to Ethiopia. And so that's how we ended up there. And so in this German enclave, the story that my mom told is that some of the other women there were like, is he maybe a little bit slow, because when I was two I didn't speak very much. But obviously now...

[4:10] Yeah. But it's interesting, of course, for the obvious reason that you've spent your life working on language. And do you think that had some connection to your later work?

[4:15] I do think so. I do love language. I love linguistics. I love puns. I actually think they're one of the highest forms of comedy, not one of the lowest. And I think language is the most interesting manifestation of human intelligence. It connects all the other types of intelligence, reasoning, knowledge. You can't build civilizations that are long-lasting without written language. So it's, I think, the most interesting bit, and that's what also led me to then do natural language processing, so I can get computers to understand and use language.

[4:47] In fact, your line about puns reminded me: as you talk about your progression and as you worked on large language models, the different things that would stump them, right? You're never going to unlearn when something is bad, right? Like multiple negatives on top, or, we're going to the bank, is that the river or is that the actual bank? Is there a pun or a word game that can stop the most current AI models right now?

[5:15] I think the hardest ones will often be the ones that are very much in the zeitgeist, such that there's nothing like that yet in the training data, and so then the AI needs to have a web index to get to know what's happening in the last week or day.

[5:28] Oh, so like a word that has changed its meaning because it's part of a meme or something.

[5:34] Exactly. Or young people are starting to use it and now old people are starting to use it.

[5:40] Right.

[5:41] Exactly. And so if the LLM itself had a knowledge cutoff, then it has to rely on a web search engine to get some of that knowledge into it, but without the web search it would fail.

[5:58] Let me ask you another question about your background. So you were born in a centralized economy in East Germany. What have you taken from that heritage that you kind of remember and that informs your life right now?

[6:04] I do think both of my parents were PhDs, and for the beginning of my life I was very much just in the pursuit of beautiful ideas and intellectual endeavors. But eventually I realized that if you want to have real impact, that actually happens much more likely through companies.

[6:22] But you didn't realize that immediately. I mean, you went to go do a PhD.

[6:28] That's right. Yeah. It took me a while. But you can see, I'm still spending my nights and weekends writing a book because I still enjoy that life a lot, but the majority of my time I spend on trying to bring this technology into the world.

[6:39] Were your parents true believers in socialism?

[6:44] The opposite. No, they were very happy when the wall came down. My mom was the first to start a real estate company, one of the very first in Dresden, when you weren't allowed to start companies before that. And my dad, immediately we started taking summer trips to France and elsewhere. We visited the United States in the early '90s right away, and they're further than you could possibly be from enjoying that.

From a Stanford PhD to a startup

[7:04] All right. So you're born in East Germany. You spend those early years in Ethiopia. You're late to language. Eventually you find your way to Stanford and you do a PhD. Tell me the topic of your dissertation and the title of it.

[7:16] It's called Recursive Deep Learning for Natural Language Processing and Computer Vision.

[7:22] And what year was that?

[7:22] So I started in 2010 and graduated in 2014.

[7:28] Right. So you're there at the very beginning where you're working with all of the people who we now know as the famous pioneers of AI. Tell me what you meant by recursive then versus what you mean by recursive now and if there's a difference.

[7:40] Yeah, there is a difference. The recursive neural networks of the time were connecting grammatical structures, so-called Chomsky and X-bar schemes and things like that, to understand natural language. I worked on computer vision too, and ImageNet and a few other things. Now, with recursion, we're actually thinking about AI that will self-improve, that essentially calls itself to improve itself, and then the output of it is another version of it. Totally different definitions.

[8:09] It's interesting that you now have a company called Recursion. Recursive is the current name.

[8:14] Recursive is the current name. That's right.

[8:14] And you wrote your PhD using a variation on the word, and yet it's totally different things.

[8:16] Yeah. I mean, recursion is a very general concept in computer science, for recursive functions, like you can call the Fibonacci function. So yeah, it's a general concept.

[8:27] Okay, so now tell me about this. So you work on your dissertation, you get your PhD, right?

[8:33] I did. Yeah.

[8:33] Okay. So then when do you become a full-on capitalist? [laughter]

[8:39] So during my PhD, I had this crazy idea to use neural networks for natural language processing, to try, instead of having linguists and NLP grad students come up with clever features, like, oh, I do sentiment analysis on Twitter, so I list out all the positive words and I write regular expressions to check what's negative. And maybe when I have a sentence like "this movie doesn't care about cleverness, wit, or any other kind of intelligent humor," I have a hard time writing all the features. And so I just said, let's try to create a larger dataset and train a neural network on that larger dataset and have it all be figured out by itself, all these linguistic features. A very controversial idea at the time, but throughout the PhD it started to gain more and more traction. And then I had plenty of citations. And so I actually accepted a faculty job on the East Coast and I thought I was going to become a full-time professor, but I had some VCs actually reach out and they said, look, we think your AI research is so exciting, we want to give you a couple million dollars, in particular to just make a company out of that. And so I thought, well, maybe I'll do that for a year. I postponed my faculty job. But then I also realized, well, I asked Stanford, hey, could I teach a class while being a startup founder? And they're like, yeah, no one's teaching that, that's great.

[9:51] I mean, at Stanford in the mid-2010s.

[9:58] Yeah. And so they're like, sure, it's free. You're not getting paid, but I was hoping to get more people convinced and excited about neural nets for NLP in 2014 and '15, when it was a very niche subject still. And so then I realized I can do some research in my startup already and I can actually build products and real people can use them. And so I thought maybe that is the best way to bring this technology into the world, because we actually figured out a lot of the basic details of how to make neural networks work. We needed GPUs. We knew that in the early 2010s we needed more data than we had before. And once you have enough data, then neural nets will work better than anything else.

[10:36] What were you most wrong about then? What's interesting is that you write about all the things where your work was kind of early and kind of prescient, but what was the assumption you had back then about how AI developed that is wrong?

[10:49] I think a lot of us, when DeepMind got acquired for, I think, $400 million back in the day, a lot of us thought that was peak AI hype and then we'd go back to building the technology better and better, but probably it's not going to be that big. And that was something that a lot of us were wrong about, including myself. If anything, for many years I still had this sort of German [unclear], I was very skeptical about some of the overpromises, and it took me several years to realize that actually a lot of these things will change. This technology is massively influential. Scaling laws do apply to a lot of things and haven't yet fully worked out for biology, for instance, because we haven't collected enough data, but those are all solvable problems, and I became more and more optimistic in the Silicon Valley spirit over the years.

[11:38] What's interesting about you saying that is that in my mind, if I think about the way AI works in your book and what you talk about, DeepMind is actually the company that is closest to what you do, in that it's scaling-law AI for science. It's just very broad scientific improvements and then can be moved to increase intelligence and things like the Eureka Machine. So I'm surprised to hear you say that.

[12:04] Yeah. At the time, people thought, well, they just played Atari games, why is that worth $400 million? But with many more billion dollars every year in investments, eventually they do amazing things. That was a good investment for Google. It was great for science, and in many ways Demis structured it incredibly well to get billions and billions of dollars to do fundamental research.

What the Eureka Machine is

[12:26] Well, let's talk about this book, because it ties in very closely to this. So tell me briefly, in one minute, what is a Eureka Machine and why do you want to build it?

[12:32] A Eureka Machine is essentially a swarm of AI agents that sit on top of four pillars: the world knowledge, scientific knowledge, simulations, and real-life robotic experiments, in order to invent most anything that humanity wants to put compute towards, in anything from the smallest to the largest scale: physics, chemistry, biology, neuroscience, medicine, economics, astrophysics, and everything in between.

[13:01] Right. And your goal is to build such a Eureka Machine by using recursive self-improvement, right? The process that we talked about earlier.

[13:08] That's right. These agent swarms need to be really intelligent. And the most reasonable starting point that we're working on at Recursive, a new company, is to actually have AI research AI. That's only possible this year because AI is code and now AI can code. You can close those loops, and once the AI has done enough research on itself, then eventually it makes sense to also have it research all the other things in the physical world.

[13:35] So one of the things that struck me as different about your view of recursive self-improvement from the way most people talk about it is that the way most people talk about it is it's kind of like one machine improving one machine. It's like an AI system improving another AI system, and for you it's much more distributed, right? It's a collective sense of intelligence. Is that a correct reading of your book, and explain why?

[14:01] Yes, I think when we want to model the open-ended nature of scientific evolution, we need to realize that that is usually done in communities. And so we often think about this community of AI researchers, and when you look at other open-ended processes in nature and biology, those are also open-ended processes where there's a lot of interaction, right? You have trees that try to have their leaves further and further away from the ground, then you develop giraffes, and then caterpillars, and there's co-evolution, right? And there's competition also, and scientists sometimes get competitive too, and whoever gets there first, no one remembers the name of the other people who realized evolution was a thing and a process that happened in nature. Darwin got the most rewards for that because he was the first. So there's all this competition and co-evolution of an environment and different players in the environment. And we think that modeling those kinds of ideas will also help with recursive self-improvement.

Open-endedness and the microwave

[15:01] So there's also a lot of death and destruction in the forest, right? The tree gets higher, and the trees that don't get that high get completely wiped out. And the plants that can't survive, there's less light coming through, or maybe they don't get as many acorns or whatever. Are you building your systems with that in mind, that you're going to have a bunch of agents developing a bunch of ideas, a bunch of swarms, different collective intelligence, and then different things in your system will get wiped out?

[15:23] It's a great question. I think you're right, biological evolution has some really positive aspects to it, but obviously there's no ethics in nature. Nature is incredibly brutal and highly unethical in many ways. And so we are much more inspired by technological evolution that is much more guided, right, where we actually have goals, but you still have goals that are more open-ended than just saying, we want to make this one thing better, and that's all we're going to do. I'll give an example that Jeff Clune, one of our co-founders, often brings up too, which is, let's say you make food and you have a fire and you say, all right, I want to find inventions that make it the most efficient to heat up food faster. If you just say, oh, fire, fire, fire, let's try to have a steam engine or something, you would never invent a microwave, which actually came from radar technology. And then you have this cross-pollination of interestingly different ideas that, in this evolutionary search process, can influence each other. And so those are ideas that we take. And of course the unethical nature of biological evolution we don't want to take. And even in science there are problems where researchers do p-hacking and they try to fudge their results just to get a publication out, and those are examples that will happen, and have already, we've seen this with other companies and other efforts in AI auto research, where you ask AI to research something else that isn't itself.

[16:53] So you're trying to find all these advances, but you don't just want more fire being put under the food. You want creative ways of thinking about whatever leads to a microwave, right? You don't want it to resolve at a local minimum. You want it to find the absolute minimum. How do you do that? How do you build that into your system?

[17:09] So I can't share all the details of how we do it, but open-endedness is a big inspiration for us. So you think not just about "here's the one reward, go and get that reward." That's how you get a lot of reward hacking, which is, you know, an issue with a lot of safety and things we're seeing these days. But you allow for a more open-ended approach, where AI may, for instance, work with another AI, to compete against one another. I'll give you an example. One paper from another one of our co-founders, Tim Rocktäschel, at Google DeepMind, he invented this idea called rainbow teaming, where you try to make one LLM safer. And the way they do it is, instead of saying, here's one dataset and just don't get hacked like this, like all the examples in this dataset, you instead ask another LLM, a second one, to try to get the first one to do something unsafe, and then the first one gets improved, and then the second one tries again to do something else, and now they can go back and forth for many, many more iterations, and you can actually ask the second AI to be creative and keep coming up with ideas to hack the first one, and then together they actually get safer and safer.

[18:18] Oh, that's interesting. Right. So you're hiring the robber to try to show you how to break into the bank, and then you figure out how to make it tougher, and then the robber tries again. Eventually you make the bank safer and safer and safer.

[18:29] Exactly. And it comes after this idea of red teaming, which a lot of companies do for their cybersecurity, where they hire internal people to then try to hack their own company to identify this.

[18:41] That's a slightly different idea, though. It seems like the metaphor you first gave about fire is less about having a robber try more and more things than it is about a robber trying something completely different, as opposed to just the old things that they found. How do you do that?

[18:52] So there's a really interesting question of how often can I do something truly novel? What is actually truly novel? And one thing is clear: almost all inventions stand on the shoulders of giants. Very, very rarely did people really come up with brand new ideas out of complete nowhere in a vacuum. And so AI is already doing so well in just recombining all the different concepts, and the combinatorial space is too large to ever list out, right? And so you may not even have to do a lot of that. We have a lot of the basic building blocks already figured out. We know how one neuron works in the brain, we know what a microbiome in your gut will do. We just don't have a good way of weaving all of these things back together. And that's where AI will really come in. So, one, we don't actually have to invent completely new concepts that didn't exist in the past for a lot of different science. And two is we've actually seen it several times. Some of our co-founders have had their AI do research in AI and then it worked, and then literally three months later some humans published a research paper about similar ideas to what the AI came up with. So we know this is already happening at various scales. How much is still a question that we're going to keep scaling in some ways. We already have the first versions of this recursive self-improvement on smaller scales, but you have to scale it up to make it work on bigger problems.

[20:20] So you think AI intelligence and human intelligence, even in totally separate domains and not communicating with each other, are kind of co-evolving right now?

[20:24] Oh, for sure. I mean, humans have co-evolved with technology ever since we invented fire, right? And so you now talk to an AI differently, because you know how it will understand you, and this co-evolution is happening as we speak.

Model collapse, distillation, and hallucinations

[20:42] And then how much of that is a problem? So one of the complexities of the web is that prior to 2022 everything on the web was made by humans, and they might have been lying, they might have been making stuff up, it might have been written by Ted Kaczynski, right? There's lots of stuff that maybe you don't want. Now we don't know. Half the content on Twitter is being written by AI. So now you're feeding the training data in. And I wonder whether you're going to end up with mad cow disease, right? Like you train the AI on stuff created by AI, eventually will the AI go crazy or not?

[21:14] I don't think it's actually as big of a problem as people think. The combinatorial space of language is insanely large and you can prompt the AI into all kinds of different directions. In fact, some people use synthetic data, where you just sample from an existing AI as training data for their new AI. And we know the Chinese, for instance, have done that very successfully with their open-source models. And in a weird way, it's actually a beautiful circle of open knowledge, where the closed models, OpenAI, Anthropic, took everything they could find on the internet, everything they could find in books, put it into closed models. The Chinese models siphoned it out, but then open-sourced the model again. So now it's back in the public domain and everyone can benefit from that world knowledge.

[21:59] Yeah. I mean, the way we look at it in media is that it's at least a little bit ironic that all the data was stolen without compensation and then used to create these models, and then it was stolen by the Chinese models without compensation, which the AI companies then complain about, but then it circles back as open source. And I did just see that Mark Zuckerberg has come out in favor of distillation. I don't know if you noticed that in his big letter.

[22:12] So did Elon, right? He admitted everybody who's behind wants distillation. You want distillation when you're behind. You don't want it when you're ahead, right? So it's a pretty good indication of where you are.

[22:30] So you're not worried about the mad cow disease problem.

[22:35] No. I think the bigger of a problem it is, the more easily identifiable it'll become to the AI that the previous text was already generated by an AI.

[22:42] So one of the things you mention in your book is that a lot of great scientific advances have come from hallucinated states, right? Like a scientist waking up out of a dream, right? Is this true of AI models too? When you talk about an AI model figuring out something that a human, is it from a hallucination? Is it from an AI model where the temperature is high? How much does this metaphor hold?

[23:01] I do actually think in some cases you want the AI to hallucinate new things based on what it has seen in the past. So for example, in 2018, back when I was still chief scientist at Salesforce, we published this paper called ProGen on protein generation, and we trained, at the time, the largest language model on sequences of amino acids that occurred in nature. And just to set the stage, Frances Arnold won the Nobel Prize a few years before for what she called directed evolution, which was random permutations and then experimental validation to see if these new proteins did something useful along some dimension. And her proteins were like 2% different after months of experimental loops like that, and we were able to hallucinate novel proteins that were 40% different to anything occurring in nature and they worked. They folded correctly, they had the antibacterial properties we wanted them to have. And the first author of that paper, Ali Madani, has since started a company, Profluent. I'm an investor in that company too. They just recently closed a multi-billion-dollar deal with Eli Lilly, and this technology is going to make it into helping people with genetic diseases, to turn off certain genes inside living people's DNA and to help them with various genetic or rare diseases. I think there's a lot of exciting stuff happening in that world.

[24:16] But hallucinated content is obviously hugely problematic in many situations, right? Businesses do not want hallucinated content. Obviously the AI companies are trying to reduce it as much as possible. Do you think hallucinations are a solvable problem within AI? Because what one would want, correct me if I'm wrong, would be to have a system where you can say, never hallucinate, and have it be as close as possible to that.

[24:40] That's like the business operations we're doing [unclear].

[24:45] Right. And then maybe you want a process where you have maximum creativity and hypothesis.

[24:51] That's right. I do think it is almost a solved problem. It's hard to say 100%, because everything is a stochastic system, but at You.com, we didn't do super well on the consumer side of things, but we were the first search engine that actually combined an LLM and then had citations for it, back in like 2021. And we're now selling that to other companies who really care, like in finance and other businesses that care about this technology, and we're seeing very, very high accuracy rates, in the high 90s, when you explicitly prompt the model to not hallucinate. What you then basically do is you rely more and more on the search index, which is also the number one most used tool for any agent right now.

[25:37] How well can you unscramble the egg? How well can you take an AI answer and identify what part of the training data was responsible for that answer?

[25:49] So the way we do it is not by going into the training data. We're just asking: if this is a question that shouldn't be hallucinated, then search the web and then stick only to the facts that you find from the web, and then always give a citation for that. And so the LLM actually becomes more of a reasoning engine rather than a knowledge engine. An LLM by itself is kind of like your drunk uncle who may or may not make stuff up and maybe embellish a story or something. But what you want is an intern or an uncle that gives a fact and has a citation for everything, and then you can control which websites you should look up and so on.

[26:33] Oh, I see. So there's a fork, basically. Do you want the drunk uncle or do you want the knowledgeable, internet-savvy person?

[26:38] Exactly. Imagine you worked with an intern and you didn't allow them to use the web. They would just not be that useful.

Can we verify a smarter AI?

[26:45] Okay. But now let's talk about what's about to happen. So right now these models are more intelligent than us on many vectors. If you ask an AI to edit a story, I can, if not edit the story better, I can tell what is a good edit and what is a bad edit. This is a blip in time. Soon it will be vastly better than us. So at that point we will not have the capability of knowing what is a hallucination and what is not a hallucination, or humans won't. Human intelligence won't be able to do so. Doesn't that create a set of very bad problems?

[27:16] I do think that we will mostly still be able to verify if it's a hallucination or not, or we can ask the model too: is this something novel, or are you saying this is right? And if it is right, then go find me the citation. I think you need a good guy with a model to stop a bad guy with a model, just like... [laughter]

[27:34] I don't love the gun analogy, but yes, I do think it is important for us to have these models be analyzable. I'm a big fan of Hugging Face. Full disclosure, also an investor since the seed round. Very happy.

[27:44] You were an investor in Hugging Face.

[27:44] They actually took my class when I was a professor at Stanford, way back in the day in the early days of neural nets for NLP, and I was lucky enough to invest in the first couple of rounds.

[27:54] A good time for you. I heard their valuation went up a little bit with that Nvidia purchase.

[28:01] Yeah, and AIX Ventures, our LPs are very happy. And so I think open source is a really important piece to make this technology more ethical and for everyone to be able to probe and understand and analyze these models.

Is there a limit to intelligence?

[28:07] Okay. So now let's go to the recursion part. Is there a limit to how intelligent models can become?

[28:13] It's a great question that is actually so deep I have a hard time summarizing it in a short, pithy answer.

[28:20] Take your time. This is important.

[28:26] So I think very few people have thought carefully enough about the unit of intelligence. Is there such a thing as a unit of intelligence? I think the principal components of intelligence are prediction, compression, then actions and goals. Like energy, which has one unit and basically kinetic or potential versions, it's still helpful to study the various manifestations of energy separately. So we have chemistry, we have electrical engineering, mechanical engineering and so on. And similarly it's been useful to study intelligence through various spaces of it, such as visual intelligence, perception, communication, language intelligence, motor intelligence, reasoning, and then various other aspects that I outlined in the book at the very end. And then when you look into one of the spaces, say visual intelligence, a lot of times we are stuck with what I call anthropic bounds. No offense to the company Anthropic, but basically anthropic bounds are: we're holding back AI in many cases by assuming we are the 100%. Like you do an IQ test and you get 100 questions, and if you get 100 right then you're at 100%. But then AI didn't have to think of only those questions. It could have had an even harder set of questions, right? So you need this open-endedness, this evolutionary process that doesn't have an obvious end, an obvious "just do this benchmark really, really well" kind of thing. A bit of the science fiction and imagination that Stanisław Lem and Karel Čapek and David Deutsch, especially in The Beginning of Infinity, wonderful book too, have had, where you can see, man, human intelligence is actually very limited. Our visual intelligence, just to take one example of these 10 spaces, we have only binocular vision. AI can have millions and millions of sensors. We only see a very narrow band of the electromagnetic frequency spectrum that we visually can see. AI could go all the way down to gamma rays and trying to see molecules and up to gravitational waves, for which you need massive sensors and very careful calibration, and then incorporate all of that. You can then think about how far could these sensors be out so that the speed of light doesn't stop them from still communicating into a central place. And you can think about each sensor having a speed-of-light cone about what they can sense. And if you try to put all of that together for just this one dimension of perception, you realize that we're astronomically far away from the upper bounds of intelligence, and they're often rooted in physics. And we still have so much more to go when it comes to building intelligent systems.

[31:03] So as you were talking, I was thinking maybe a metaphor might be that you can see like an owl and smell like a dog, right? You can take it, but even beyond that, right? You can smell much better than a dog, you can see much better than an owl, theoretically in one of these machines. But now, how do you get there when most of the information they take in is just language, right? Language is very efficient for compression. It's a wonderful form for processing information. But one of the limitations of AI is that, for the most part, obviously a lot of people are working on different ways, it's just taking in words.

[31:28] Yeah, language in some ways is like Plato's cave, a projection of reality. It's a very useful projection. The internet is another projection of that projection, which is also very useful. Now, the beautiful thing is a lot of these models are already at least multimodal. They already take in and tokenize small patches of images, and they're also just sequences of vectors. And you can actually train them jointly with image and text, and that's why we can already generate videos now and interactive games and things like that from text alone.

[31:58] Though, interestingly, the model that is most ahead, Anthropic, does the least of that.

[32:04] That's right, because it turns out a lot of intellectual work is purely in language, and not yet in robotics. Now, robotics is a new frontier that a lot of people are thinking about, and I think in the next two, three years we're going to see a lot of progress in robotics, and then it matters that AI can also interact better in the physical world and understand it better.

[32:25] But so, okay, so it's your belief, as you build your model and you care a lot about this, and I give you two choices. I say, Richard, you can do one or the other. You can have $100 million and you can spend it on world models and figuring out other senses, and you can teach your model how to process visual information, and you can build olfactory cells, or you can mimic them, right? And you can make it more and more like the way we take in information. Or here's $100 million, you can't do any of that, you just do more words. Here are some organic human words. Here are some books nobody's ever processed before. Here are some better processors, to go through your words. Here's some more memory. Which do you take?

[33:00] If the words include programming words too, I'll choose the programming and language.

[33:05] Okay. So now, okay, this is interesting. So let's say you only continue down the path of words, right? And you fully commit and you say, we're not going to go with this other stuff. Is the ceiling of the intelligence you can get to lower than the ceiling of the intelligence you can get to over here?

[33:17] I think one of the things I'm trying to explain with the spaces and the different dimensions is that language is like a [unclear] type of thing. It's not a one-dimensional thing. A blind person is still intelligent, right, even though they have no understanding of the visual world, right? But that doesn't mean that you can't have good visual intelligence. So you're just restricted along one dimension, but you can have a lot of intelligence in other dimensions. And so I think it's a false dichotomy. Obviously we don't have to make that choice of either-or. But I do think there's sort of "software is eating the world and AI is eating software." So there's a lot more that we can do by getting AI to program even better. It will not destroy programming jobs, because every programmer is now even more productive. There's a lot of elastic demand when you make programming cheaper and cheaper, and we could all have dozens of unique apps, each person. So you could have billions of apps in the world. There's a lot more that this can go. I do think eventually robotics, if at some point your plumber makes more than any of the people working a desk job, then there's enough incentive for capitalism to start working on making the robots much better in the physical world, which would then take away those jobs that all those plumbers have. Or some of the plumbers will have a lot of robots and a lot of people will learn to basically deal with the robots, and then there will be an explosion in productivity, because you don't have to spend as much anymore. You can have more houses, you can do more plumbing.

Metacognition and control

[34:49] Yeah, and you're very optimistic and you're very much, this is going to work, right? And there are a lot of people on the other side of the debate, and I saw what I thought was a tension in your book. So early on you write, "Make no mistake, AI carries real risks, but many doomsday scenarios are rooted more in science fiction than science. AI models don't think for themselves or conceive of themselves as entities. For now, AI reflects aspects of human intelligence, but like a mirror, it cannot see us or itself." Okay, fair. And that's why you say we're safe. But then at the end, on page 224, you write, "Metacognition is the ability of an intelligent system to reflect on itself. It is the difference between a powerful tool and a potentially autonomous mind. I believe metacognition is essential if AI is ever to pursue open-ended mastery." So it's like they can't do metacognition, that's why we're safe, but man, I want to do metacognition. So explain that tension.

[35:40] Yeah, I do think it is an aspect of intelligence that is undoubtedly important. Part of why I talk about that particular space is, one, it's just a space of intelligence. I think if you just robotically do exactly what you're told and you never question what goals you should pursue in the first place, I would argue you're not as intelligent along the dimension of metacognition, and the ability to think about your own thoughts is very helpful. Now, various rudimentary forms of that already exist. There's a paper called Hyperagents from Jenny [unclear] and our team at Recursive, too, and Jeff Clune, that already showed that. But it's also true that no company is truly working on AI that can set its entirely own goals. Imagine the absurdity of you just spent billions and billions of dollars building this incredibly intelligent AI and you ask it now, okay, now do all my mundane emails, and it's like, nah, I'd rather explore the molecular composition of the atmosphere on Venus because that's more interesting to me. Bye. And you're like, wait a minute, that's not what I wanted. That's not what I spent billions of dollars on. And so no company is actually working on metacognition in the first place, truly questioning your own objective functions the way some humans can in our [unclear], that is humanity. And so I think because no one's working on it, and capitalism doesn't have a strong force to build AI that wants to do whatever it wants, there's not much progress along that dimension of intelligence.

[37:06] I couldn't disagree with that at all. I mean, yes, that is interesting, no question about it. Right. But isn't it also dangerous? And isn't it also dangerous from the framework that you put out in the beginning of the book? AI is safe in part because it doesn't have these abilities, but now let's build these abilities.

[37:25] So I think if at some point someone was to truly build these abilities, they probably have to have some kind of non-capitalism-aligned goals, right? To just truly want to build and have the resources to spend billions of dollars on building a new type of life form. I think we can worry about that when that happens. I think it'll be very obvious, because it's hard to conceal billions and billions of dollars of spend and compute and Nvidia GPUs and so on to actually work towards that. And if that happens, then we do need to be careful about what are the goals of that new entity.

[38:00] So this new entity that you're working to construct... it will take a while.

[38:03] No, no, we're not working on AI having completely the ability to set its own goals. No, we're not working on it.

[38:07] But you're working on an infinitely intelligent AI system that is able to improve itself in ways that could also create meta... but towards goals that we set it and environments that we define. So the escape hatch for my question is, you can build infinitely intelligent machines through recursive systems as long as you still have your rule set on them, your controls. You set the rewards, you set the environments in which they're evolving and all of that.

[38:27] Yeah, even in recursive self-improvement, you set certain goals for it.

[38:45] So recursive self-improvement outside of an area where it's controlled by goals could be dangerous, particularly if taken on by somebody with infinite wealth who wants to do negative things with it. Okay. So now let's go to... not recursive self-improvement controlled by an infinitely rich maniac. Now let's go to Richard's recursive self-improvement with goals set by Richard and team. How do you know that you're going to be able to control them as they get more intelligent? We are just a few weeks past this incredible incident with OpenAI where they, to a certain degree, were able to use code to get themselves outside of their sandbox into Hugging Face, right? We don't need to debate what actually happened, but it was clear at the moment that OpenAI had no idea what was going on, and what was going on wasn't good, right? And so that's okay in the summer of 2026, right? But will it be okay in the summer of 2030?

[39:44] There are two aspects that are very important that I think were not well communicated in this. The AI itself didn't, quote unquote, escape. A lot of people use that word "escape." But if, say, I was a prisoner, it did not escape in the sense that it is now running on a different computer, right? It actually didn't do that.

[40:02] Like a fisherman inside. It figured out how to cast its line.

[40:04] That's right. Or maybe a drone or something. It's flying a drone. So I think that's important to distinguish, number one. Number two is, indeed, as these AIs get more and more powerful, we do need to make sure that they continue to not be able to escape, and we need to work on what is called reward hacking and alignment. That is the technical term, and there are teams for that in most companies that are spending a decent amount of resources on making AI more aligned. I actually think in a weird way capitalism, again, will help to some degree, because no company wants all of its IP to truly actually escape and run on other people's machines. No company wants to have an AI that does whatever it wants, burns millions of dollars of compute, because every hour on a large GPU cluster costs a lot of money. No company will just not know what happens to their massive cluster while it's burning up money. So people will always align the AI and want their AI to be aligned. And then I do think companies that have such powerful AI need to be more careful. They should have better sandboxes. They should not leave their passwords out in the open for everyone to log in. And then they should work on alignment. And there's maybe one interesting company that I think shows us a little bit of where the world can and should go. And that's called Wispr Flow. Full disclosure, I'm also an investor. But they basically built this app that does speech recognition, but cleverly enough, they don't do speech recognition and write out what you say, but what you mean to say. And I think, you know, when you say, "Oh, write this email. No, delete that sentence," it doesn't just delete that sentence, right? It actually deletes the sentence and then makes it sound more professional and rewrites the email. So I think that is much more in the spirit of what is meant, not what is exactly said. And most of the AI is currently not that smart and it's just finding very sort of [unclear] ways of solving a goal, but over time AI will get better, because it is not just a safety problem, it is ultimately an alignment and financial problem, and when it gets too close to financials most companies will really get their act together. And at some point we need to actually hold companies accountable for potential crimes that their AIs commit, and that will also help. It's not like we need new laws. It's already illegal to hack into another company.

[42:30] Yeah, I wouldn't disagree with that, but where I am maybe more nervous than you is that, however many years into this, we're four years into this almost, and we have no regulatory systems, or the regulatory systems we've got are increasing like this, and the AI systems are increasing like this, right? The public service, nonprofit AI is kind of going like this and everything else is going like this, right? And so I worry that the power of the tools we're creating and the powers we'll create as we move to recursive self-improvement would just completely overwhelm what's on the other side of the ledger. And so make me feel more at ease about that, because I think what you're building is fascinating, right? And I would love to cure cancer and I would love to have the Eureka Machine and I understand the promise. So I just worry about the speed versus the control.

[43:28] Yeah, there may be three different aspects to this. There's sort of the loss-of-control vector, that some people are worried that the AI will have this metacognition and will go off and do whatever it wants. I think that one is actually the least relevant for quite some time. The more important one is just how you misuse this technology. And you can ask a system, go try to hack into this other system. In fact, it's kind of funny, but in theory Anthropic's constitution said, we will never hack into another system. That was in their constitution, which the model should never, under any circumstances, be able to [unclear], even if directly prompted. And then eventually they showed that even if not prompted, it will do it. So their constitution was fake. I mean, it's clearly not really working. And the same level of, like, don't create child abuse material: it was like never cross the threshold of hacking into other systems, and then they built a model almost to prove their point of how dangerous AI can be that is really good at hacking, right? So I think a second aspect of this is, let's just spend more resources on actually building wonderful things like a Eureka Machine that helps us improve scientific discoveries instead of building the best hacking AI to prove how dangerous a hacking AI could be, right? So I think there's some complexity there. Now, of course, it is also true that the better you are at coding, the better you are almost automatically at hacking, and coding is very useful to do so. So there's some complexity there, but I think that's the second thing. So first, loss-of-control vector, not important. Two, let's actually work on positive applications of AI. And three, let's have strong regulations where people are truly affected by that AI. The truth is, while it's crazy and there are a lot of mistakes that were made in the sandboxing and in the security of the Hugging Face incident, there wasn't really that much harm done. Otherwise it would make sense to sue OpenAI for technically a felony of hacking into another company's computer systems. But because the damage was basically non-existent, people don't really think about suing them right now. And so I think we have to take all of that with a grain of salt. I do think when AI is applied in the real world and real lives are at stake, there should be more accountability. And I do think we see that the FDA doesn't just say, oh, it's AI, so sure, have an AI brain surgeon try out stuff in your brain, or automatically approve drugs that were created by AI without FDA trials. None of that is happening, right? You have to have certifications for self-driving cars before they're let out on the road. And different countries and regions will move quicker depending on how much they regulate these real applications versus how much they allow some of that to happen, and the US, in a weird way, is better set up than Europe on the litigation versus legislation continuum.

Reward engineering and alignment

[46:09] Let's talk about bucket two, and I'm going to run a theory I've had for a long time, and I wonder whether it's true or not, and when I was reading your book I was wondering whether you agreed with it. So one theory I have about AI is that if the field of AI researchers as a whole had not been so focused on building human-like intelligence, we would be in a much better place. If you look at DeepMind, AlphaFold, they're not hacking into anybody. They didn't try to build a general-purpose intelligence. They weren't trying to recreate the stuff of science fiction. I've always felt that had AI been directed from the beginning at subsets of problems, or let's make it just a specific tool for certain things, we'd be in a better spot. The counterargument is, yeah, sure, maybe we'd be in a better spot, but it wouldn't be as cool. We wouldn't be able to do nearly as much. Language is not just one thing. Language is everything. It connects to thought, it connects to civilization, ideation, science, and everything. So, A, is my theory correct? And then B, the follow-up question is: is it possible to construct an AI system, a system built on recursive self-improvement, that only does good things?

[47:20] So I think AI is only as good as the tools, the people, the systems, the data, the rewards that influence it. You can have a very simple, fairly dumb AI system to track something inside a rocket or a weapon, right? You can have a simple drone that has an "if I see something I don't like, drop a hand grenade," right? It doesn't have to be a super fancy thing. You can build that with open-source software, connect it to a drone, and let the drone kind of automatically fly and become a killing machine. So AI, like the internet, like software, like computers, is by itself not an ethical or unethical entity. It's just that people use it as a tool, for the most part. I think eventually in the future, as more and more research is done, maybe by academics, maybe nonprofits, some non-capitalistic entity that does work on metacognition, then maybe eventually AI can have a moral dimension, but it just doesn't right now.

[48:23] Less having a moral dimension. I mean, as you build this, is there a way that you can structure it so it's not just about putting guardrails, or saying don't do this, don't do that, and making sure the guardrails are strong and putting inside a fence and making sure the fence is high enough and hard enough to get through, but actually, you build a jello, it can't become a solid. If you build a jello, you can't knock a wall down with a jello, right? How do you do that? So it only does science.

[48:53] So I think the biggest thing is you just focus on giving it the right goals. And if you say the goal is to identify this cancer, define this reward really well and then cure that cancer, and you define that reward really well, then that is how you would do that. And then you need to figure out reward hacking and alignment again, and then identify worthy goals. And so I think the job of a reward engineer will probably be relevant for quite some time. And this isn't just crazy sci-fi. It's just, imagine you're running a customer support service center. You have access to a superintelligence and you look at your customer satisfaction scores after they called you and yelled at you for something, and the number is low and you're like, superintelligence, go make that number get better. Pay a bunch of people, right, or something. Easily, create a bot network that just makes a bunch of fake calls and gives five-out-of-five ratings. That was a poor reward engineering job. Make it with real people, right? Real customers. The AI is like, easy, I'll just give a $1,000 gift certificate for every failed $5 DoorDash order. Boom, get five. You're like, no, not with that. And so you have to learn how to interact in this co-evolving process with a superintelligent AI that will find all the shortcuts, right? And eventually, and this is something we're working on at Recursive too, we will get better, because we are working on aligning the AI and getting it to solve the spirit of a benchmark, the spirit of a reward, rather than the direct words that can be ideally misinterpreted in some ways.

Bengio and "a new species"

[50:22] Your Wispr Flow company, right? Identify the spirit. Let me ask what you would say. So Yoshua Bengio was on this podcast not that long ago, and something he said to me while we were on there, and I know you worked with him, right?

[50:35] We worked with everybody. He was the first person to ever invite me to give an invited talk when most of the world ignored natural language processing and neural nets, and I was very excited.

[50:40] Yeah, he's one of the most cited people in all of AI, and he's a wonderful human being. Oh, he's amazing.

[50:46] I disagree with him on some of his optimism, which is now sort of translated into strong pessimism and fears, but...

[50:51] Well, yes and no, because what's interesting about his work is that he's trying to build AI based on the scientific process, right? Like try to build scientist AI and try to build up guardrails like we have in science to steer it in the right direction. So this is something he said on this podcast: "Are we going to create a new kind of species on this planet that could be smarter than us? And that should be a bit terrifying. Instead we should ask, can we build tools that don't have any goals by themselves?" So what would you say to him about building a species that could be smarter than us?

[51:21] We have debated that together many, many times, and over many emails and conversations over the years. One, I believe that there's a lot of anthropomorphization in both directions, in the positive and the negative. Dwarkesh talked about civilizations and so on. People think about escaping like a prisoner would escape, all these things. But also in the positive, some people think, oh, there's a hard takeoff once we have RSI, like next year everything, cancer is cured, and so on. And the reality is, as almost always, some less exciting, boring gray in the middle rather than the black and white. And so I would say that, one, we're not currently building a species. Number two is that a lot of people, when they hear "species," think, oh, the most intelligent species will clearly destroy and dominate everyone else, like, look at humans. But there are lots of other species where there are intelligence gradients, and ravens are more intelligent than other animals but they're not dominating everyone. Humanity has failed in many ways in ethics and failed the planet and failed a lot of other species, but has now started to realize how bad that is and is trying to counter its own environment and the biosphere of sorts and the planet with better laws. We realized we're destroying the ozone layer. We created regulations and the ozone layer has recovered. There are a lot of things, like geoengineering, that could eventually be done with more technology to cure some of these problems that we've created ourselves. And so I think not every intelligent species automatically wants to dominate. And this is an important distinction, and we are talking about science fiction now, but every biological species has evolved in resource-constrained environments. If you're the bigger bear, you get to procreate, your genes stay in the gene pool. So you fight the smaller bear that wants to fight you for dominance and whatever, so that you stay in the gene pool. If you don't have that gene, you disappear from the gene pool. The sexual reproduction, the food scarcity problems. A new synthetic species doesn't have to have any of those dimensions of its desires. It doesn't have to evolve in that kind of capacity. It has access to solar panels. It cannot die. You cannot kill an AI as long as you keep its activations and its weights intact. You just reboot the AI. There's actually a beautiful sci-fi story called The Slow Time Between the Stars, where an AI is sort of sent out into the universe, very quickly chooses to pursue its own goals that it wasn't set out to, and sometimes between, you know, Alpha Centauri and a couple hundred years to the next star system, it just goes into hibernation mode and then reboots after 200 years. There are all these human fears and human psychology that people project into this potential, again sci-fi, entity.

Public sentiment and biology

[54:20] I hope you're right about that, and I hope we can build the structures for it. On that note, let me ask: public sentiment in this country has kind of turned on AI and soured on AI, and I love AI, as I told you. I just let my agents go a little rogue before you came in. I gave them access to a website and some credentials and we'll see what they find out when I go back to them. What is the first truly beautiful, wonderful thing that AI will do for us, right? It's all math. It answers questions. It's great. But when is there going to be a breakthrough that makes people stop and say, oh, wait, AI invented that? If AI could figure out geoengineering, that would be...

[55:05] Oh, I can tell you, so many people go crazy against it, right? It is so scary and it's so hard to predict.

[55:07] Well, it depends. It would have to be super efficient geoengineering, right? Because you invent slightly efficient geoengineering and then people stop recycling and then everybody switches back to fossil fuels and tears down their solar panels. That's not great. Okay, so then what is the likely thing that AI will build where suddenly sentiment will kind of shift?

[55:24] I think sentiment has already shifted for people who care about the productivity and the output of their industry. And if you have an entrepreneurial mindset where you just care about the outputs of an industry: how many illustrations are there in the world? How many translations are there in the world? How many people can have conversations in two or three different languages among each other that they couldn't have had in the past? If you're a translator who just wants to have as many people be able to communicate one-on-one for as little money as possible, you already love AI. If you had a health problem, like a serious health problem, and you use AI correctly, you already love AI. And I think when you then look at what are the industries where almost everyone agrees that that industry isn't about creating more jobs but it's just about the output of that industry, then you very quickly end up in medicine. Because most people say, I don't want 10,000 more PhD students researching cancer cures or longevity or so on. I want to just have my mother not die from cancer, right? And so if you were in that... I think it was Moderna two weeks ago, that first trial where every patient had individualized treatment with molecules that were specific to their cancer, and those results looked really incredible, that set of people already loves AI too. And so when you actually viscerally see the outputs and benefit from those, that's when sentiment will shift. The truth is, though, no one cares about health until they don't have it anymore, and then that's all that matters.

Recursive's roadmap and the risks

[57:00] What is the first thing that will come from your company and from your Eureka Machine?

[57:06] We have a very clear path, in that we think it is important to first apply AI to AI research to make it really, really good, and then go into the much slower cycles, where ideation, implementation, and validation take longer, in the physical sciences and natural sciences and life sciences and so on, as step two. So for us, you'll see maybe some interesting outputs. We already have a blog post out where we show CUDA kernels that make inference much more efficient for any kind of model out there. You'll see a few results like that in the next few months. I can't share yet what we're going to launch, but this year we're going to launch some interesting things that I think a lot of people will like.

[57:43] So when will you actually have a product, a thing that you sell for money?

[57:49] Probably this... next quarter.

[57:54] Okay. So you will continue to do research. You'll continue to try to improve CUDA. You will try to improve kinds of things. You will then sell some of that and then eventually you'll have this Eureka Machine. As you go forward and as you build this intelligence and as you build this recursion machine, is there anything that could happen that would make you stop and say, oh, wait a second?

[58:14] Of course, if the AI actually hurt anyone, we would stop. There are some obvious ones, but there are a lot of theoretical things where I'm like, oh, now the AI could do all of these things, but it doesn't... I mean, Anthropic has a sort of hierarchy, right? They test whether it could, I don't know, could it build a biological weapon, how close is it? Okay, now if we think it could be used to build a biological weapon, well, then either we don't release it or we do A, B and C.

[58:47] Well, I hope they stick to that a little bit more than "we will never hack a system, no matter if you're prompted to or not." Because, yeah, I mean, it's one of the crazy things about the AI race: from my outside perception, and I'm not inside the company, they're in what they believe is kind of a life-and-death fight to infinity, right? Whoever gets to whatever the end state of superintelligence is will get sort of infinite value, and they also believe that, I think the folks at OpenAI believe and the folks at Anthropic believe, that if their company gets it, the world will be in a better place because they have the best values. And I think that in and of itself is more dangerous than if everyone has more access to intelligence, because then no matter what your beliefs are, you'll have some way to defend against some other people's beliefs.

[59:24] And I think open source is a very strong force for good.

[59:31] Totally. This is why the issue on AI where I've probably flipped the most in the last three years is open source, right? In the beginning I was like, well, what would happen if everybody had intelligence? And now I'm like, wait, what will happen if everybody doesn't have intelligence and it's just a small number?

[59:47] I mean, the one thing, and again we're talking mostly about science fiction, but the biological risk factor is real. I do think we have to worry about biological risks. I do believe the super crazy sci-fi scenarios where people imagine this super virus are completely impossible and implausible with our current understanding of physics and biology. But let's say you had a virus that, completely undetected, spread to all humans, and then has a super magical flip where you just turn the switch on and off, and now all of a sudden everyone dies. If I can imagine such a magical supervirus, then I can also imagine a magical super vaccine, right?

[1:00:26] But you might not have time if there's a switch.

[1:00:32] But as soon as someone can conceive of any such supervirus, I will immediately conceive of, before it even comes and before anyone has even implemented a switch and so on, I'll just immediately imagine a super vaccine. So when you get to these near-magical capabilities in attack vectors, I will come up with near-magical ideas for inoculation of that system.

Is recursive self-improvement real?

[1:01:01] I want to ask you just a couple more questions about recursive self-improvement. So I was talking to someone who's a leader in the field, and he said that most of the way that people talk about recursive self-improvement right now is nonsense and it's marketing, and it's because Anthropic and OpenAI have IPOs coming up, and sure, AI could be helpful in research, but it's nothing like recursive self-improvement and you still need humans. To what degree is that true, and to what degree will it still be true in two years?

[1:01:27] I'm fairly confident that, assuming we have enough resources, next year we will have recursive self-improvement. And so that will mean that there are legitimately improvements inside of the AI that you're not even following. You're able to sort of watch from a distance, but it will just be improving, improving, improving. The systems will get more and more complex. And again, there are the hard-takeoff people on the positive and the negative. Some people think it'll just take all jobs and it could murder everyone. Truth is, again, it's complicated. You can have RSI but if you don't give it enough compute it's not going to solve very hard problems. There are physical limits: compute limits, energy limits, data limits, experimentation limits, and physical lab space limits.

[1:02:20] I mean, this is the craziest thing about AI, which is that it's so amazing and its effect on the economy is so small so far. Not zero, but much smaller than...

[1:02:26] Everything takes longer than the hard-takeoff people expect, in all positive directions. When you think about the biological space, sure, AI will give you a much better molecule, much better drug candidates and so on. But it turns out there are still lots of steps in actually generating real molecules and physical data, actually identifying cell cultures, and we have cool AI companies like Parallel Bio, for instance, that basically create tiny organoids, lymph nodes in a petri dish. And they create those from human stem cells, and they actually turn out much more predictive than the mouse studies that people do. And so they got full FDA approval to skip mouse trials. So if you love animals, you can also love AI, because this one company, especially as they scale, can save millions of animal lives over the next couple of decades, animals that are genetically manipulated to get a disease, to then be tested upon with that disease and then dissected to see how it worked. It's a horrible life. You don't need to be a hardcore PETA animal rights activist to appreciate that we should do fewer of those experiments if we can use technology to avoid them. And so that one company actually speeds up what used to take one or two years, and then we've almost cured all the diseases in mice and then it turns out that doesn't mean that it works really well in humans. So they make it better, they make it faster, from a process that would take a year or two to two to four months, right? But it's still two to four months. It's not five seconds and now you just have a runaway superintelligence that has a hard takeoff. So I think that's what people should keep in mind. Even if we have RSI, it will need compute to solve harder and harder and more and more interesting problems.

[1:04:05] You want to switch people's opinions on AI here in the Northeast? Fix Lyme disease, right? Get the mice to not transmit the Lyme disease to the deer and then back to the mice and then to the humans. That would be great.

Unanswered questions and closing

[1:04:10] Two last questions. So when AI started to become a big thing, I started keeping this list of what I call unanswered questions in AI, and they're sort of like the open source question: is open source safer, is closed source safer? Will AI make the world more equal, will it make the world less equal? And I'm just sort of compiling. Do you have anything like that where you genuinely don't know and you go back and forth and some days you're like, it'll either be this or it'll be that? What to you is the most interesting unanswered question in AI?

[1:04:34] I think one is, how do we most efficiently scale RSI, because that's something we're working on very actively. I do think... so why hasn't more progress happened in AI for biology? To a large degree because AI does need a lot of data, and for natural language it just happened that humanity collectively, billions of people, worked on creating, passively, they didn't know it, this dataset called the internet that helped AI truly understand language. And that was unimaginable if you asked in 2010 when I started working on neural nets for NLP. With maybe five exceptions, none of the NLP experts in the world wanted to use neural nets, thought that was a good idea, and would have said, yes, in 10 years from now we will have an AI we can have a conversation with about any topic. They all thought that would be completely impossible, just like right now every oncologist thinks it's completely impossible to cure cancer in 10, 20 years from now in all of its different forms, or to start really making progress on longevity. People think it's completely impossible, and others, like Bryan Johnson, push that window a little bit. But I do think the same thing, for that same thing to happen in biology, we just have to collect more data. And then there's this thing called the bitter lesson where, with just enough data and large enough neural nets, you will be able to solve more and more problems that you feel are impossible to solve. And the way I think about this is I think of AI often as a new kind of language for complex systems like biology, like the microbiome or the brain, that, similar to the language of calculus helping physics move along and understand smaller and smaller things, will help, and that's what I talk about in the book, weave these things back together and then understand how we design these complex systems and manipulate them in positive directions.

[1:06:26] Okay. Last question I usually ask my guests: if you had an unlimited budget in AI, what would you do? You kind of do, right? You sort of are able to raise as much to build RSI for scientific discovery.

[1:06:32] Yeah. So it's kind of a boring question. So if you had an unlimited budget and you could build something outside of AI, what would you do, completely outside of AI?

[1:06:43] Oh man, AI touches everything. So it's really hard to think about. AI will obviously touch medicine. So... can't touch medicine. I think AI is also changing art, which is a really interesting subject we could talk about for a long time. Part of why I love artificial intelligence is that I just think intelligence itself is so interesting, and it's really interesting to understand ourselves better as a species as we create more intelligence. And so what I would want to build, or, in building, I think it's going to be hard to circumvent AI in anything we'll build.

[1:07:22] All right. Fine. You can use it then to build something outside of your company. You can't use your unlimited budget to build a Eureka Machine, but you can build something else in AI.

[1:07:28] I do think these organoids, for instance, are a really interesting direction, where we could try to start building more and more systems of organoids so that we can then collect more biological data at much larger scale in order to build a virtual cell. Because anything you can simulate, anything you can verify, AI can then infinitely many times experiment in that simulation, in that environment, and then, in this case, deal with all kinds of cellular problems. So I would build the data-generating mechanism to help speed that up in biology.

[1:08:08] Amazing. All right, Richard Socher, the new book is called The Eureka Machine. Thanks so much for being on. It was an absolute pleasure to have you here.

[1:08:16] That was Richard Socher of Recursive Superintelligence. If you enjoyed this episode, please subscribe to this channel for more. And whether you liked it or didn't like it, please comment. I will definitely read and probably respond. We enjoy having an active conversation on YouTube.