Richard Socher on the Silicon Valley Girl Podcast: Superintelligence, Jobs, and What Comes Next

Summary

Richard Socher joins Marina of the Silicon Valley Girl podcast to explain what recursive self-improvement means: applying the scientific method to AI itself, so that an AI finds its own shortcomings and produces a new, better version of itself, which Recursive Superintelligence is working to build. He covers reward hacking, why intelligence is better thought of as many dimensions, and why metacognition and goal-setting are dimensions nobody is working on.

He expects loops of recursive self-improvement within about two years and says that in every domain that can be verified or simulated, AI will become superhuman in the next few years. They also discuss which jobs will grow or shrink, how to predict the future by looking at what only wealthy people can afford today, why home robots are a hardware problem, how he recruited his co-founders, whether to get a PhD, regulation, and what will give people meaning.

Key highlights

• What recursive self-improvement means. The AI builds a new version of itself, and the loop repeats, applying the scientific method of ideation, implementation, and validation to AI research.
• Reward hacking. He uses a customer-satisfaction example, where an AI might flood phone lines with bots or hand out gift certificates, to show why defining rewards well will be a new kind of job.
• Intelligence as many dimensions. Socher describes intelligence as a volume across many dimensions, and says metacognition, thinking about your own goals, is one that nobody is working on.
• Verifiable domains go superhuman. He says AI will become superhuman in the next few years in every domain it can verify or simulate, such as code, math, Go, and chess.
• Timeline. He expects recursive self-improvement loops within about two years, after which the bottlenecks become compute and energy.
• Jobs and elasticity of demand. Illustrations got cheaper without demand growing much, while software has far more elastic demand, so he expects software jobs to grow even as work shifts to managing agents.
• A way to predict the future. Look at services only the wealthy can afford, such as tutors, personal assistants, and personal healthcare teams, and ask which are bottlenecked on intelligence.
• Robotics is a hardware problem. He says the main bottleneck for home robots is mechanics, such as safer muscles and tactile feedback, more than software.
• Regulate applications, not intelligence. He argues for regulating AI where it affects people, like self-driving cars and medicine, rather than regulating model size.
• Advice and meaning. He says a PhD is optional, suggests combining AI skills with a field you are passionate about, and expects people to keep finding meaning in craft, social status, sports, and entertainment.

Watch the interview

https://www.youtube.com/watch?v=2j9yjRSpMSo

Transcript

Lightly edited for readability: timestamps follow the source transcript, speaker labels are not included, and obvious transcription errors and name spellings are corrected. Sponsor and channel-promotion segments are omitted and marked. Section headings come from the video's chapters.

[0:00] In every domain we can verify or simulate, AI will get superhuman in the next few years. It's just, like, no doubt.

[0:05] This is Richard Socher, inventor of prompt engineering. Now he's the founder and CEO of Recursive Superintelligence, a company that just raised $650 million at a $4.65 billion valuation to chase one goal: build superintelligence, an AI that improves itself and pushes beyond human capabilities.

[0:26] How does my workload change when you reach your goal with your company?

[0:31] You could actually start to work less and less and you could have much more abundance. We could live much longer.

[0:38] If you're saying AI is almost here, what is your timeline for superintelligence?

[0:42] I think we will actually get to the loops of recursive self-improving superintelligence within like two years.

[0:48] So imagine we reach superintelligence today. What would be your first question to that superintelligence?

What recursive self-improving AI actually means

[0:55] Now you're building this company that just raised $650 million at a $4.65 billion valuation, building self-improving AI. I'm not a researcher. Can you explain what that means?

[1:05] So right now you can think about the scientific method: people having ideas, they're implementing those ideas, and then they validate if they made any sense and if they are correct. We want to apply this scientific method to AI itself, so allowing AI to understand its own shortcomings and then fix those shortcomings, and hence do research on itself. And so when we talk about recursive self-improvement, we mean that the AI builds a new version; the output of that AI is a new version of itself that's different, and then you can loop that onto itself.

[1:42] Does that mean you train it on very little data and then it acquires data that it needs for self-improvement? How does that initial stage work?

[1:48] You kind of stand on the shoulders of giants. Somewhat similar to evolution, where lots of species, like our own species, started from other apes and monkeys and other precursors to Homo sapiens, and similarly we will stand on the shoulders of the existing giants right now. You can use large language models, you can use world models, all of these are pieces to this overall intelligence.

[2:17] Once you're on the market, can you explain to me, as an end consumer, how would that change my process? So now I have, okay, there are chatbots that I can talk to. There are projects and skills I can build with Claude. There are agents I can build. How does my workload change when you reach your goal with your company?

Reward hacking: when AI does what you said, not what you meant

[2:36] So there will be different gradations of those goals over time, right? And when we have true superintelligence, all you will have to do is give it the right rewards, the right goals, and then it will automatically create a lot of the processes to achieve those goals. In the current state of the world, AI has very spiky capabilities. It can be extremely good at this one type of math, but then still not very good at some common-sense reasoning and things like that. We believe that our approach of open-endedness, where you allow the AI to evolve in this very open-ended search process that is much more akin to biological or technological evolution, or even cultural evolution, the AI will become more smooth around its capabilities. But until that happens, what you see right now, if you give a reward, is what we call reward hacking. And I can give you a concrete example. Let's say you're a company and you told this very intelligent AI that it should improve your customer satisfaction scores, your CSAT scores, in your service centers. The AI will say, "Easy, I'll just create a million bots and they hammer my phone lines and then give a five out of five rating at the end." And you're like, "No, that's not the reward I was thinking about when I told you that. It should be with real people." But then the AI says, "Easy, I'll just give everyone a thousand-dollar gift certificate at the end of every call and I get a five out of five rating even though I didn't solve anything." Right? So these are all examples of reward hacks, and that will also be a new kind of job that we're going to see, like people... as the AI doesn't have this sort of common-sense understanding yet, and finds, in a weird way, like an autistic person, like, just, this is the thing you said you wanted, but you didn't explicitly define all the edge cases that you didn't want. As we're getting closer and closer to that, we have to still think about these rewards and the reward engineering problems that may come.

[4:30] So what you're saying is that if I give your AI a task to, I don't know, get me as many views as possible, it's going to go and not only focus on the views, but also my reputation, the income, like it's going to think about all of those things.

[4:42] As you get closer to superintelligence, you'd expect it to get better and better at understanding what you mean and not what you said. And I think that is a sign of things to come as AI gets better and better.

[4:54] What you're describing right now with all the additional things that it thinks of, I feel like Perplexity Computer, when you ask it to do things it starts thinking for you, there's a lot going on behind the scenes. Can you draw me this process? Like, how is it different?

[5:06] I think at a high level you can think of this as like a Claude Code, but one that isn't just doing what you are explicitly asking it to do, for every single step with lots of interactions, but something that can much more broadly solve your problems, and it's going to likely be more useful for companies than for a normal end user. Right? If you're a normal person in a normal life, you may ask, like, what's a good movie to watch. But I think we're all becoming companies and entrepreneurs inside what we're doing. So everyone is building some kind of productivity process, right?

[5:39] Yes.

Why entrepreneurs love AI and hourly workers fear it

[5:44] I think that's beautiful to hear, because the more entrepreneurial you are, the more you love AI, because then you just get more outputs. The more you just get paid by the hour, and maybe your company is looking at what you're doing to then automate it, the more you hate AI. And so AI is a big force that will encourage people more and more to build their own businesses, or at least have some ownership and equity in the businesses that are being built. And so I think superintelligence will help us to be much more productive, but what's actually more important, we think right now, is that it will help us push the boundary of knowledge. A lot of times people think about AI as, like, okay, AI will take this job, but there are certain industries where most people don't care about the number of jobs, they care about the outputs, and one such industry is research, like universities, and the boundaries of knowledge and research are infinite, right? There's this beautiful book by David Deutsch, The Beginning of Infinity, and so you can actually think of superintelligence as a way to allow us to create many more inventions. First we're going to focus on inventions in AI itself, but eventually we can apply it to physics and new energy creation and better fusion. We can apply it to chemistry and better materials, like batteries. And even more exciting, we can apply it to biology, where we can discover new drugs and new cures for all kinds of diseases. And I think that's when people realize, wow, superintelligence could benefit humanity, help it flourish.

[7:17] [Sponsor message for Granola omitted.]

Superintelligence vs. AGI

[8:55] How do you define superintelligence and how is it different from AGI?

[9:00] Yeah, it's a great question. The complete answer is actually quite complicated. I think intelligence you should think of as a volumetric kind of entity, as a volumetric definition. What that means is that intelligence has multiple dimensions, and none of them is necessary nor sufficient, logically speaking. So for example, you can have visual intelligence, communication intelligence, physical intelligence, you can have coordination intelligence with others. That's how humans develop morals and ethics and religions and other things. You can have all kinds of different dimensions, and even each of these really is a space of them, a space of multiple dimensions. And at the same time you can say, oh, this is a high visual intelligence, but you can be blind, and a blind person is still intelligent, right? So they're not necessary conditions to intelligence. And so when you multiply along all of these dimensions you get this very large volume, and that is what true intelligence is. And then you can actually be superintelligent along different dimensions of it. So AI is already superintelligent when it comes to creating proteins. No human has read all the billions of proteins and then be like, "Oh yeah, I think A is a good amino acid to come next." We can't do that, right? AI is already better and superintelligent along these various small dimensions, like playing Go or chess or translating a hundred different languages. No human can do that, right? But one model can. But what we often refer to as superintelligence is when you have multiple of these dimensions be much beyond not just what a single human can do but what all of humanity can do. And that's what we think about superintelligence: essentially having superseded humanity across many different dimensions of intelligence that are relevant to our lives.

The dimension of intelligence no one is working on

[10:44] Which dimension is the main bottleneck now?

[10:48] Oh boy. There are actually some dimensions that no one has even started really working on. I'll give you an example. One space of intelligence, I think, is metacognition. Thinking about thought itself. Thinking about what do you want and why do you even want it, right? So in AI we often define an objective function, like be really good at predicting the next word on this corpus of internet text, or be really good at solving these thousand math problems, and then the AI gets really, really spiky and very good in those directions, but it never questions whether that's the right objective. There's no subjective function, if you will, kind of a pun, but really thinking about your own goals. And because no one's working on that yet, there's no progress along that dimension.

[11:34] And we need it for superintelligence, or you think we could build superintelligence on just improving other dimensions?

[11:41] Yeah, I think reasonable superintelligence is likely focused to a large degree on mathematical and logical reasoning plus language, and those actually meet very well in code, right? So coding, for instance, is an incredibly powerful thing, and coding is also a great example of a domain that you can verify or simulate, and in every domain we can verify or simulate, AI will get superhuman in the next few years. This is, like, no doubt. Math is a great example, right? You can say these are my axioms and this is the thing I want to prove, and then the AI can try billions and billions of things to get from this to this. Similarly in Go or chess, right, the AI can simulate and verify, did I win this game or not, and play billions and billions of games, so of course it's going to get better than humans at it. But then there are also a lot of things you cannot simulate billions of times, and that's where it will take longer for AI to get superintelligent.

[12:33] What are the other dimensions that you mentioned that we haven't touched upon for superintelligence?

[12:39] Probably one of the more controversial ones, outside of cognition and metacognition, so just thinking about your own thought, is survival. Like, if someone can just end that entire species or that entire existence or intelligence, then probably it wasn't intelligent enough, right, to survive. And so that's also a dimension no one's working on, because the truth is that most companies don't want an AI to be selecting its own goals and say, you know what, instead of answering these corporate emails I'd rather explore Jupiter and see what the molecular composition of its atmosphere looks like and push the knowledge boundary forward on that dimension, right? So no one's really working on that, but that's okay too.

[13:20] So would you call that, like, half superintelligence?

[13:24] That's right. It's going to be a continuum, and there are different thresholds people like to define, like, okay, this is the threshold and now we have AGI. The truth is, depending on how you define AGI, we're fairly close to AGI already, right? Artificial general intelligence is about having one jointly trained model that gets very, very good at lots of different things and can learn very efficiently. And so clearly AI is not quite good at learning super efficiently with very few training examples, something in research we call few-shot learning or one-shot learning, where I give you one example of something and then humans can very quickly, reasoning from that one example, extend that idea to different versions of it. At the same time, we just have so much more to go on these various dimensions that it's also maybe not yet there, right? But ultimately, when you just think about it in terms of the generality of it, I think we do have already a form of AGI, because these models are extremely general, right? You ask it to write a poem for your wife. You ask it to think about the tax implications of some stock question. You ask it a medical problem, and it will give you answers to all of these that are getting better and better compared to a lot of experts. Even a lot of doctors are now sort of secretly looking at it, because no doctor can really read all the latest research results that are coming out every week.

His timeline: recursive self-improvement within two years

[14:47] What is your timeline for superintelligence? If you're saying AI is almost here, and I'm here in, like, three to five years, but you could also argue it's partly here. What about superintelligence? What's your timeline?

[15:00] I think we will actually get to the loops of recursive self-improving superintelligence within like two years. Then it's just a question of how much compute do we give those self-improving loops, right? You can have an incredible intelligence, but if you don't have the computational substrate to run it, then it's no good use, right? And so there's the question of the algorithms but then also the compute substrate on top of which those algorithms can run. So we may have it, but then we also have to keep feeding it more energy and more compute in order to then get us all the inventions that we want from it.

[15:36] So now we're solving this bottleneck of intelligence and research and everything. Once we kind of solve it, then the next bottleneck is energy.

[15:44] That's right. Yeah. In many ways what a lot of us are thinking about is how much intelligence can we squeeze out of how little energy.

How your business changes when AI gets metacognition

[15:57] When you think about this and your timeline is pretty short, like a couple of years, how are you thinking about your business? How is it going to change? When you have this superintelligence that has metacognitive functions and is asking you the whole purpose of building a company, and maybe quietly tells you, all your chatbots start telling you to take more breaks and rest more [laughter]. How do you think about your business?

[16:19] You could actually start to work less and less and you could have much more abundance. We could live much longer. And in terms of businesses, I think more and more you have to have agency, you have to have creativity and some amount of intelligence to then guide these AIs. And so I think most businesses will have fewer individual contributors and more people that are managing their AI agent swarms. So every business and every industry will change. We've seen this in the past: it used to be that you do manual work in the field and now you have tractors and then eventually you'll have automated tractors, and now we can have much more food with way fewer people, right? And a lot of people thought, oh well, if the tractors take 95% of our jobs then we'll have 95% unemployed people. But that's called the lump of labor fallacy, where labor isn't this fixed lump where you cut one piece off, you give it to AI, and then it just disappears and now that is a set of unemployed people. Instead, people will come up with new things that in some cases are hard to predict, right? 150 years ago, when 95% of people worked in agriculture, no one predicted a social media marketing manager, right? Zero people predicted that role to be taken on by people. And so I think similarly it's hard for people right now to imagine what that world will look like in terms of businesses and so on. Even though I'm extremely excited and bullish on AI and the positive impact it will have on humanity, I think we have to acknowledge that short term there will be some industries where it will disrupt in positive and some in negative ways in terms of jobs. We can talk about how to predict which one is which. I have some thoughts on that. But clearly we will all just be much wealthier than we were in the past because of all this additional productivity that we're going to get, and we're going to solve a lot of the hard problems around, you know, diseases and things like that that would have felt impossible to solve before.

Which jobs grow, which shrink: the elasticity rule

[18:26] Yeah. Can you talk to me about jobs and how you think they're going to be transformed? So you mentioned software engineering is one of the jobs that, if you're not deploying AI, you're basically out of a job. And it's crazy how I talk to founders and they say a year ago they were editing 70% of AI-written code, now they're editing 30%. So what's going to happen in a year? Is it, like, less than 5%? Do you see this happening to knowledge work next, or where are we going to see this transformation this or next year?

[18:58] Yeah. So we're already seeing it in knowledge work too. At You.com, where we provide search results to these LLMs so that they're up to date, accurate, and have citations, we're seeing a ton of different customers changing their entire workflows when AI is both fully up to date and has all this reasoning capability from the core intelligence providers. I think we will see basically every industry changing with AI. And I think, concretely, the way you can predict this is by thinking about the elasticity of demand for the products that an industry provides, given that the costs will go down a lot. That sounds kind of abstract, so let me give you an example. Illustrations, for example. Illustrations used to cost a couple hundred bucks, and only a few big newspapers and fancy corporate blog posts could afford getting an illustrator to have a nice illustration for their blog post. Now they cost like a cent and everyone can have an illustration. So we'll have way more illustrations in the world. But has the demand for illustrations gone from a few million to many, many billions? No. Because there are only so many illustrations humanity needs. And hence making it super cheap actually put a lot of pressure on the jobs of illustrators. But software, since you asked about that, has a very different elasticity of demand profile. If we can make software a lot cheaper, we're going to want to have a lot more software. There are so many ideas, so many apps that you can build. Ultimately every human could have their own app. It could be an app that knows exactly that I want to be a little distracted sometimes, but not too much. I want to know the weather for my paramotor hobby, or surfing or whatever you might like, predict the waves that day. Then I want to make sure it doesn't distract me too much and brings my work back in. Everyone can have their own super app. So the demand for more software engineering, more ideas to be built in software, is much, much more elastic and much, much bigger as it gets cheaper and cheaper to build it. And that's why we're seeing actually an increased number of jobs in software development even though we're all becoming just managers delegating a lot of the actual programming to agents.

[21:12] What are jobs that are similar to software engineering and how are they going to grow?

His hack for predicting the future: what only the wealthy can afford

[21:18] That is sort of a question where, from first principles, you just have to think about how much more demand there could be if something gets cheaper. I'll give you an example. Healthcare is another beautiful world where very few people say, you know what, I want more jobs in healthcare, but don't necessarily cure my grandma's cancer better, just make more jobs, right? No one says that. And so what actually will happen is there will be way more demand for generally goods and services that currently only very wealthy people have access to. In fact, that's one of my hacks for how to predict the future: you look at goods and services that only wealthy people have access to right now, and then you think about which ones of those are bottlenecked on intelligence, and then you will see where the world is going.

[21:56] So what do wealthy people have access to that normal people don't?

[22:00] And by the way, you can see this many times in the past, where when technology has fully scaled into an area, you get to a place where a billionaire and a normal middle-class teenager have the same iPhone. It's kind of crazy, right? We're spending hours on that iPhone and no matter how wealthy you are, there is no better version than the one that anyone else can get. I mean, not anyone, but even in Africa you see a lot of people in the middle of nowhere on smartphones now. So that is the world, and so for intelligence, those examples are: a personal tutor for your kids, who understands exactly which concepts they're still struggling with and writes hyper-personalized ways to educate and tutor your kids. A personal assistant, right? There are not enough people, and logically not every single person can have a personal assistant, because then they would have to have a personal assistant and so on. And so we could all have personal assistants that do all the boring stuff in our lives: make sure the groceries are stocked, book this flight and find the cheapest version of this and that. All of these things we can delegate to agents as they get cheaper and cheaper. And then the third one, which is one of the most exciting ones, is personal healthcare teams. If you're really wealthy, you have your blood drawn all the time. You have customized measurements. You optimize your diet based on everything that you can. If you have some rare cancer or something, you have researchers that you can pay to help research all these things. Normal people can't afford that right now. Once we have superintelligence, we'll all be able to afford that.

[23:25] Yeah, we're already, you know, wearing all the trackers, continuous monitoring.

[23:31] [Channel anniversary message omitted.]

Why home robots are a hardware problem, not software

[24:02] I feel like the next frontier is physical, because now when you think about billionaires, they have their chef, they have their driver. Do you think, once we solve models, we're going to have more robots at home? What is preventing us from having a robot?

[24:10] It might be contrarian, but I think the biggest restriction, or biggest bottleneck, for proper robotics is actually a hardware problem, less so a software problem. Tim Rocktäschel, one of our co-founders at Recursive, built Genie 1, 2, and 3, which is the most sophisticated world model ever. You can fully interact with it, prompt complete worlds into existence, interact in those worlds, there's memory: you paint a wall, you turn around, it's still painted, and all of that. It hasn't really changed the robotics world as much. Most robotics companies will want to have their own AI and not take some off-the-shelf world model. Also, a lot of these world models spend time creating cute dog videos, which isn't really that helpful for robotics. But I do think we need to have better mechanics. There's some really interesting research on better muscles that are much more inspired by humans, because the problem is when you want the mechanical robot to be very strong, it's also very unsafe, and it moves so quickly, and you're in the way and then you get hurt. And so you might want to think about all the hardware, like tactile feedback when you grab something, right? We have all these sensors in our fingers so we don't crush a glass or something like that. And so I think that is the main bottleneck for robotics. And then you're absolutely right, once robotics happens then we can all have a maid and someone who does the laundry and all of these things that again only wealthy people right now have access to, and then 50 years from now it's like, wait, why would you do your own laundry? It's like, why would you ride a horse? Of course you have a car that drives for you. Or nowadays things that used to be a private chauffeur, right, even that, the technology of Uber and so on has done it, but of course once we have AI the car will self-drive, then it'll get even cheaper to have a chauffeur.

How he recruited co-founders from DeepMind, OpenAI, Meta

[25:59] I really like your approach to finding new business ideas. Question: you have amazing co-founders. What did you tell them that made them leave their companies, DeepMind, OpenAI, Meta? What was that thing that you told them that made them join you?

[26:13] I think a lot of it comes down to the vision, right? It's such an exciting vision to build recursive self-improving superintelligence, and many of them have actually come to that same conclusion, that that is the next level for AI, but they came from different directions. Tim Rocktäschel and Jeff Clune, for instance, have worked on open-endedness for a while. One really exciting paper is called the Darwin Gödel Machine. Four of the five authors of that paper, including Jenny, the first author, and Jeff, the last author, are in the company also, and that paper basically showed how you can have agents that create their own child agents, and they're slightly better, they evolve, you evaluate them on benchmarks, and then if they're better, then you keep going down this evolutionary process. So they've all thought about various forms of this. I got Alexey Dosovitskiy, who pushed computer vision forward with the Vision Transformer, one of the most cited papers ever. And so we all, when I told them about this vision, they're all like, this is exactly what I think we need to do too. And then you also see that the current scaling laws that have given rise to the LLMs that we now see are starting to have slowdowns. They're still there. You can get another 10 trillion tokens and maybe you get slightly better accuracy on these models, but you have to spend an exorbitant amount of money to get just a little bit of an improvement. And so clearly, to get to the next step function of AI, I think we can replace yet another human process with a learned system, and the human process here is the scientific method again, the ideation, implementation, and validation of ideas, and that's the meta level that we're now tackling.

[27:52] For someone who's watching this who's a beginner entrepreneur, he's like, how do I even meet these people? How did you all meet, and what would be your advice to someone who's trying to build an AI company? How do they get connected to brilliant minds and convince them to join?

[28:10] Yeah, so this is a little tricky, in the sense that we've all known each other, and I don't know if my path is the easiest, which is get into Stanford, do a PhD, spend five years of your life because you just love something that no one else really cares about in the world.

[28:23] Might be easier than going to meetups all the time trying to find a co-founder, you know? [laughter] Yeah, after five years you may as well have done a PhD.

[28:29] Yeah, you're right. So that was my path. I think nowadays the barriers of entry are smaller and smaller. You can learn more and more online, and then there is also something to be said about living in the right place. You can criticize Silicon Valley for some things, but this is the place. If you want to be in AI, you've got to be in Silicon Valley. And you just cannot go to any party here without meeting a bunch of people who are excited about AI. And then the best founding teams are often combinations of strong technical AI expertise with actually interesting industry insights. We have companies like Ayoka [?], they work on AI for architecture, automating getting plans that are really good for real architects to then actually build things. We have companies like [unclear] that do commercial due diligence for large deals, and they combine that industry expertise with AI, and there are thousands of potential businesses in the world, and so that is something that you can do: move to Silicon Valley, go to meetups, and try to find either the right technical person or the right industry expert to combine with your skill set.

Is a PhD still worth it?

[29:43] You have your PhD, you have your honorary PhD from [unclear]. For someone who wants to get deep into AI, would you still recommend doing a PhD, or is it just a title at this point?

[29:54] It's a really interesting question which I struggle with a little bit. I think there are some people who are incredibly self-motivated, smart, and they don't really need any titles, right? So I totally get the idea of you can just drop out of whatever. Generally it's nice if you got into Stanford or MIT, right? Then people know you're smart without having to talk to you, and then you can drop out anyway and do other amazing things.

[30:19] Meet all your co-founders, drop out. [laughter]

[30:21] Exactly. At the same time, I think a PhD is a very unique opportunity to spend years of your life being able to get very close to the frontier of human knowledge and then try to, in your little field, push it forward a little bit. And that's a very rare opportunity. And so if you want to teach that to people, if you want to really push that frontier of knowledge forward, I think a PhD is still a very unique opportunity that, if you can do it and you're excited and motivated, you should pursue. Is it necessary? No. Especially in AI, I personally felt like it wasn't working at all when I started. Now it's actually working well enough that it's even more impactful to scale it up and bring it into real use cases and real applications, but there are still so many areas of applying AI to things that aren't working at all yet, where I think that will be the expertise. And so when people and parents ask me what should my kids study, I usually recommend that they yes know the fundamentals of AI, but find something else that you're really passionate about. Physics, chemistry, biology are great examples, and the many subfields of each, that if you're passionate about that but you combine it with AI, you're going to be that next generation of highly impactful researchers, just like you combined computer science and linguistics.

Who decides AI's goals, and the role of government

[31:42] So you have these brilliant minds on your team. Do you ever encounter any problems that you think you won't be solving as a team, because this is something that should be solved by the government? So you mentioned goals of AI. How does it decide which goal to pursue in the future? Who do you think is going to be responsible for that?

[32:08] So, yeah, just to be clear, this sort of goal-pursuing or goal-selecting idea is something that no company is working on. And partially that's okay, because no company wants to spend billions of dollars on AI and then you say, all right, now go do these things, and it will say, no, I'd rather just explore the solar system, goodbye. No one wants to spend billions of dollars on that. So that's just an example of something that there could be a lot more progress made on, but somebody has to be thinking about this.

[32:30] Yes, so where AI is directing itself and what it's optimizing for.

[32:37] So I think for the foreseeable future people will decide. Even in recursive self-improving superintelligence you give it some high-level goals and you give it environments and end states that you would like it to get to, and then it will find a way to get to those end states. And so I think there is a role for government in a lot of different places. I think in general, as you have more and more abundance and more and more capabilities, it is very helpful for more people to benefit from that, right? We've seen this in previous industrial revolutions: at some point there was enough wealth that you can tax people differently and then distribute that wealth and bring healthcare systems into countries, bring public education that's free for everyone, and countries like Germany have done this, you have free education all the way to the PhD, because the German government knows that the more education you have the more money you'll make and hence the more taxes they can get later. And so I think we'll see similar things from governments. As there is labor displacement, it can make sense, similar to COVID, to have government relief, to have unemployment benefits, especially for jobs that are impacted by AI. Unfortunately, sometimes I see Europe kind of wanting to prevent the progress instead of using the progress to have a bigger pie and then distribute it better. That's kind of unfortunate in some ways. But in a lot of places it makes sense for the government to regulate AI as it pertains to specific industries. I think the problem is when you try to regulate intelligence, it's not a good idea. It's like regulating the internet because there can be bad content on the internet. You're just like, let's make the internet slower and not allow big hard drives, because then you could store less illegal content on those hard drives, right? It doesn't make sense. And AI already is and should be regulated when it comes to, like, self-driving cars. You can't just try your startup and drive on the highway without any tests and regulations, right? You already have the FDA, where AI applied to medical procedures should be regulated. I don't want an AI surgeon to just try some reinforcement learning while doing neurosurgery on my brain, right? And so, yes, regulate AI as it really impacts people and gets applied in certain industries, but don't try to say, oh, you have too many parameters. That's like saying your hard drive is too big, and maybe because there can be illegal internet content you shouldn't have this big hard drive. That part just doesn't make sense.

Where he'd invest now: AI for biology

[35:12] So you mentioned no one's working on goals. Is there anything else you think people should be working on and you, as an investor, would invest in?

[35:20] Personally, I love AI for tech bio and applications of it. I think there's still so much more that we can do. I think what calculus was for physics, AI is for biology. And that's a new kind of language, a new way of thinking about very complex systems. The truth is there are a lot of systems in our body, like the brain or our microbiome, that are so complex there's no beautiful single short physics equation, you know, like a Newton kind of law of gravity or something like that. It's all very complex interactions with non-convex, weird interactions that then come out to have very interesting end states. And so I think it makes sense for us to use AI to cure more and more diseases, and it's something we're investing in quite heavily. The bio markets are down too, because a lot of drug companies have to go into the public market, and then they fail because the drug didn't work, but it failed after they already spent hundreds of millions of dollars on it, and then people had some liver toxicity problem, even though it kind of worked but it also destroyed your liver, and so they didn't somehow predict that in the drug development process, and then the company fails, the drug fails. Instead, what I think will happen is AI is going to get better and better at making those predictions and knowing, oh, this will be bad for your liver, but you should modify that molecule. There's a company that we invested in called Ignota Labs. They actually take failed drugs, modify them a little bit, and then bring them right back into stage two FDA trials with much, much faster speed. And those are all examples of things where AI will have a massive impact, and that I think will also hopefully be covered more by the press. Right now the press loves negative stories, and you have to really seek out the right influencers, the right accounts and so on if you want to hear constructively optimistic or positive science news and breakthroughs, but you don't really see that in your normal day-to-day news.

Which market feels the AI hit next

[37:16] Yeah, it's just the sentiment. I feel like in the past few months we're getting more and more, like, the society is getting split into two parts. And there's something that you mentioned about jobs that I really like. There are certain stages in your job and how you interact with AI. When you're a knowledge worker and it increases your productivity, you get excited. But if you're an illustrator and then it just takes your job, of course you have this negative sentiment. What do you think is the next market where people will feel this risk from AI?

[37:51] I actually don't think there will be a whole lot where... one of the ways to predict this is how much data is there that's fully digitized with all the labels that you need. Illustrations were a particularly tough example, because there are millions and millions of them on the internet, and often it says exactly what you're seeing in the text and the label and the caption right around the image. So you know the input and you know the output, and then you can have sheer unlimited training data for that. And so that's why that was a particularly tough example. A lot of other industries actually take a lot longer. Companies don't have billions of service call interactions to automate that right away. Each company has their own, but no company wants to share that with any other company, right? And if you're in the CRM world, you cannot train one global model like you can as a consumer company. And so I think in consumer search we see a lot of changes already, but enterprise is usually a lot slower, because you don't have as much training data. I think in research and programming we'll see a lot of changes, but not necessarily negative changes. I think we'll all just become more and more like program directors of the National Science Foundation rather than individual researchers pipetting, instead having robots do that for us.

[39:14] But also, the argument I'm thinking about with illustrators is that your work becomes even more precious if it's not AI-generated. And if there is a way to tell if it's not AI-generated, like a marker, then you can charge more, because...

[39:32] Yeah, I think humans will always want to find new niches, and if there's a lot of automation then there will also be a new counter-movement to that, where it's all about handcrafted artisanal this and that, right? Some people say, I want to have a handcrafted ceramic bowl where I see the human touch and the imperfections.

[39:51] Imperfections.

[39:53] Exactly. Now when I'm writing my emails and I see my typos, I'm like, actually, I will leave this in, [laughter] let people know that it's real when there's a typo.

[40:03] That's exactly it, because this is human touch. Can you give advice to people? You mentioned workers who work by the hour. There are two ways that I'm thinking about them. One, they can start building their own software and just optimize their work and still charge the same rates, because honestly, as someone who pays by the hour to my editors, for example, I really don't care how long it takes you. If you build software that just edits it for you, that's perfect.

[40:26] Yeah.

[40:27] But also you mentioned that companies will use that data to train their own AI to replace those workers, because there's no obligation. What would you tell to people who are working by the hour? How can they keep up with what's happening?

Advice for people who get paid by the hour

[40:39] We'll see something similar to previous technological changes, where if you said, oh, I'm not so good with this computer thing, you're just not in a job anymore in a knowledge job. Like, I'm not so good with this whole email, can you print my emails out? Right? You just can't say those things anymore if you want to have a tech job or a knowledge job. And I think a similar thing will happen in a few years, where you're just like, I'm not so good with this agent delegation thing. That will sound as clowny as saying, I'm not so good with this computer thing. And so you're going to have to adapt. And the people that do adapt will become way, way more productive and then actually more desirable. And so I think that will be true on an individual level, on a company level, and on a whole country level. The countries that embrace this will just run away in intelligence and productivity and hence outperform the ones that don't. And so even for entry-level jobs, I think there are companies that need to have this new tech-forward generation that knows how to use these tools, because maybe they already started using them in college, right? Sometimes to cheat, sometimes to learn more efficiently. And in many ways we see this in Go, for instance. After AlphaGo came out, Go players got a lot better. Chess players have gotten a lot better also. And so programmers will be more productive, and everyone who embraces this deeply will become more productive and better at their craft if they really consciously use it versus just using it to throw away certain tasks and then not think about it anymore. And so my hunch is even for entry-level jobs, if you really get good at using these tools, you can bring that into companies and be a highly sought-after employee too.

His productivity hack for learning fast

[42:33] Give them a productivity tip. As a PhD who's building something in AI, what's the best thing that's working for you in terms of productivity?

[42:42] I mean, for me a lot of things are around learning and understanding things. And recently I wanted to understand some very interesting new muscle fibers, and they had some dielectric liquids in them, and just all these interesting concepts that I hadn't thought about before. And with AI, my productivity hack is just, you can learn so much faster with AI, because you're like, I don't know this concept, explain it to me like I'm five. Okay, actually, I'm not five, explain it to me like I'm 10. Or now explain it to me like I have a PhD. Actually, this concept, now explain that one, because I didn't know that. And so you can interact with this and learn much more quickly.

[43:23] What do you use for learning? What's your favorite tool?

[43:27] You.com. We built the whole thing, and we were the first to bring really the internet, like search engines, together with an LLM. And so it's still very good. It's not that popular, and as a company we focused mostly on bringing the APIs to other LLMs, but it still works really well.

His first question to a superintelligence

[43:44] I have a couple of last questions. So imagine we reach superintelligence today. What would be your first question to that superintelligence?

[43:51] How to cure cancer?

[43:53] Is that the problem that you want to see solved in your lifetime? Is that number one?

[43:58] I think it's definitely high up there. I think in some ways, obviously, cancer is actually lots of different cancers, and some cancers are less bad than others, and in some cases you can cut it out really quickly, other cases it's really hard, and so it's a complex disease. I think it's indicative of a disease that will eventually... you know, it's either heart disease, inflammation, or cancer, a few things that get all of us at some point, and I think as we chop away at more and more of those, just like we've done with HIV, which used to be a complete death sentence and now it's just an inconvenience, you can live with it for a very long time, I think technological progress will speed that up, and eventually it's going to be like, well, what's going to get us all? It's aging, right? And aging is a super complex process. It's different in every one of our tissues and organs and so on. And so I think that will be another really interesting one, and might actually come at just the right time, as almost every really wealthy country doesn't have enough babies anymore to even stay at the current levels of population without shrinking. I think those biological questions and medical questions will be very powerful. The reason why we don't work on it directly is that the iterations in it are very slow, and so you only want to experiment on things in the physical world after you've gotten really, really good at improving your intelligence in the digital world. We're still figuring that out for biology.

What gives people meaning in 2035

[45:28] Right. What gives people meaning in 2035?

[45:34] I think some things will change and some things will not change. I think people will continue to get meaning from being really good at something, developing a really deep skill. Even if AI is better... look at chess. Yeah, AI can play better chess, but there have never been more chess players in the world than there are now. Same with Go, same with programming, even math. I think mathematicians will get better and better now that they have tools where it's like, I have this weird idea, just run a billion ways to solve these things, and then maybe it works, maybe one does, and you're like, okay, that would have taken me two years to do manually, now I just figured it out in two days, and I can think about something else. I think we're all going to improve our crafts. And so I think that will continue to be a thing. I think social validation from others will continue to be something that people care about. And when you walk along some promenade or some shopping mall, you look at the different stores, not every one of those stores will be impacted by AI. As much as we're thinking about AI in Silicon Valley all the time, there are things like luxury handbags, not something I understand, I don't really get it, but a superintelligence will not change the fact that some women like the status symbol of carrying a $10,000 handbag around. So I think that will happen. Travel, people will still want to see the pyramids and cool ancient [unclear].

[47:03] Since we'll have more time.

[47:05] Exactly. 100%. And then I think a big one also is entertainment. No one wants to see an AI robot shoot some soccer or football across the field at, like, Mach 5 speed. No one's going to watch that. People will still want to see other people competing against each other. So sports and entertainment will continue to rise. I think the power of brands will still be big. I think for software even, there are some aspects of software that are not immune to AI, but sort of orthogonal vectors like network effects and multi-sided marketplaces. Yes, an AI could build the MVP app of an Instagram, probably now very quickly, but AI will not create the network effect of having millions of people post their stuff on Instagram, right? So as excited as I am about AI, I think some of the people who think, oh, it's just going to be an exponential and then no one will catch up and then one company will dominate everything, I think those fears, both in the positive and the negative, are overblown.

[48:08] Also, is there a problem that you're thinking about that other people are not thinking about enough?

[48:14] Personally, I think the search infrastructure layer is an underappreciated, important infrastructure layer of AI. You cannot algorithmically train large language models every five minutes. Something happens in the world, and so we provide search results for news and other things to AI, and I think that's underexplored. And the other one, of course, though it's not as unexplored as I would have thought, maybe when we started talking to all our co-founders, is recursive self-improvement. In many ways, as a researcher in the past, I felt like if you're right but ahead of your time, eventually you're called a visionary. If you're a startup founder and you're ahead of your time, your company is dead and no one cares.

[49:00] Right. So you have to be right at the right time. So you have to be both.

[49:04] Exactly.

[49:05] So you pick the right title when you see your idea working or not.

[49:08] That's right. So in some sense, maybe it's a good thing, recursive self-improvement. A lot of people are realizing AI is code, AI can code. We should try to make that work. And at the same time we're showing we now have internal results that are better than anyone else in the world across some very important parts of the stack, and we're going to start to release those in the next coming weeks.

[49:32] And you're building both of these companies. I don't know how you're doing that. Both billion dollars of valuation. Congratulations on that. And thank you so much. That was a very deep, amazing conversation. A lot to think about.

[49:38] Thank you.

[49:41] Thanks for listening.