Existential Risk and the Future of Humanity: Lessons from AI, Pandemics, and Nuclear Threats (Toby Ord, Author of "The Precipice")

[00:00] The human story might just be beginning, but there are various threats to our continued existence. Some of these have been around forever, such as asteroid impacts, but some of them are threats of our own making, such as nuclear war. You put the odds at one in six that in this century we do incur the existential risk and humanity fails to live beyond this century. Have the last few years made you more or less worried about this? So another possibility is what gets called gradual disempowerment. [00:30] maintain sufficient control over them, they don't break the law, but they're more successful than us over time at doing all the kinds of jobs that we do. Even if we get richer through trade with these systems, a higher and higher share of the wealth eventually accumulates in AI hands. [00:49] Hey, I'm Mario, and this is The Generalist Podcast. As the saying goes, the future is already here, [00:56] It's just not evenly distributed. [00:58] Each episode, we sit down with the visionaries, builders, and thinkers. [01:02] who are already living in that future. [01:04] to help you see it earlier, understand it better, and capitalize on it. Today, I'm speaking with Toby Ord, [01:10] a senior researcher at Oxford University, one of the world's leading experts on existential risk, [01:15] and author of the excellent book, The Precipice. [01:18] It's a thoughtful analysis of the greatest dangers to humanity's survival. [01:23] From rogue asteroids to pandemics, [01:25] to unaligned artificial intelligence. These may sound like fanciful sci-fi problems, but there's actually good reason to believe they're extremely pressing.

[01:33] with Toby estimating humanity has a 1 in 6 chance [01:37] of extinction this century. [01:40] In my conversation with Toby, we discuss how AI risk has evolved since LLMs emerged. [01:45] why the scientists working on the Manhattan Project didn't stop developing nuclear bombs after Hitler died. [01:52] and the Cold War lessons the U.S. is ignoring in its current great power conflict with China. [01:58] I walked away from our discussion with a better sense. [02:00] of how we should think about humanity's future [02:03] what we owe coming generations. [02:06] And what steps we might take today around AI safety, [02:09] and pandemic preparedness. [02:12] This is a new podcast. [02:13] So if you like it, [02:14] I hope you'll consider subscribing. [02:16] and leaving us a review. [02:18] Now, [02:18] Here's my conversation. [02:20] with Toby Ord. [02:22] Well, Toby, I'm so excited to chat with you today. I can't tell you how much I enjoyed your book, The Precipice, which I have to hand. And in looking at my sort of marginalia... [02:32] in preparation for this, I was struck by one, that I could be a much more thoughtful reader, and two, how much I enjoyed it, because all of my notes are sort of, wow, exclamation mark, exclamation mark, and no way. So... [02:48] Maybe with that as a little bit of a background, perhaps you could tell us, [02:53] what exactly existential risk is and what does it mean to study it in the way that you do? [02:59] Yeah, so humanity has had a long and illustrious past. Our species has been around for about 300,000 years or more than 10,000 generations. And we might be able to have a future of equal or larger size. Certainly most animals do. They typically live for about a million years. So the human story might just be beginning.

[03:21] But [03:22] There are various threats to our continued existence. Some of these have been around forever, such as asteroid impacts and things like that, these natural risks from the somewhat hostile world that we live in. But some of them are threats of our own making, such as nuclear war. And so existential risks are any risks that could threaten to permanently destroy humanity's long [03:52] go extinct. Or it could be something that, say, causes a permanent collapse of civilization, where if it was unrecoverable and there was no way back, then these things would have a similar kind of role. They would be the types of events that reduce the value of our future to almost nothing, and which we have to make sure that we avoid falling victim to these even once over the hundreds of thousands of years to come. [04:19] Yes. When you sort of started in this field, was it the case that these risks were relatively well known, at least, if not well sort of studied? Or, you know, does the work of an existential risk researcher involve thinking of, you know, all of these possibilities that maybe no one has actually even properly cataloged before? Yeah. [04:44] Yeah, I think the risks, at least the risks that I know of, had been fairly well understood, or at least, sorry, maybe that's going too far. They were all known of. So it's not that I sat around trying to think about new things that could threaten us. It's possible to do that. And there's kind of, you know, any number of things that could pose some kind of risk. But often those risks get smaller and smaller as you go further down the list.

[05:14] you know, that needed in order to kind of keep adding on things that are only tiny compared to the things you've already got on the list. [05:19] And I think that they'd been generally known about for hundreds of years through to kind of decades or something like that. But there could be risks that we're still unaware of. For example, in 1900, we were unaware of the risks of gamma ray bursts or supernova explosions. [05:49] It's actually surprisingly recent. And so there could be other risks that we're ignorant of. But there is at least a helpful thing when it comes to the natural risks, which is that we know we've survived for 300,000 years so far. And we know that typical animal species survive for about a million years. So it can't be the case that the risks are kind of much higher than that. [06:19] million per year or lower. [06:21] fascinating we'll get into some of those risks in greater detail especially some of those that you [06:28] particularly relevant these days. But I'm curious, you know, more on your journey, you studied sort of computer science in Melbourne, how does one go from, you know, perhaps a future software engineer to spending their lives focused on the far future, on risk on these sorts of questions?

[06:47] I guess one thing they have in common is I was probably always going to be an academic. And so in computer science, I was interested in the kind of theoretical side of computer science. Oh, interesting. Including very much in artificial intelligence, which has turned out to be useful for me now. No kidding. I guess I was drawn to these questions about... [07:10] You're really zooming out and looking at the big picture of humanity's future. So thinking about this, when reading good works of science fiction while I was growing up, one of the things I remember being struck by is if we did manage to reach a point where we dealt with all of the negatives in the human condition. So we dealt with poverty and we dealt with various other forms of pain and suffering. As we have been slowly doing over the years, right? [07:40] things. The idea that you could go through surgery without extreme pain would have shocked people. If we kept going down that trajectory, what would happen next? What's the positive story about what we should be doing if we got to a point where we had removed a lot of the injustices and discriminations and pain and suffering and inequality, all the bads, what next? I was interested in these kinds of questions about the future of humanity and trying to understand the [08:10] we face. I guess one of those challenges, you know, that was preventing us from reaching that point was global poverty. And so I was, I was very interested in that, you know,

[08:22] since I was an undergraduate studying computer science in Melbourne, and had just decided since individuals could do so much to make a difference, that I would kind of make that part of my life. And I've continued to do so. And that led me to found this organization, Giving What We Can, where people make a pledge to give at least a tenth of their income over their lives to the most effective charities trying to help others, be they people or animals. Yeah, it's often turned [08:52] that are some of the biggest challenges facing humanity are also some of the ones where we can kind of, [08:57] I get the most leverage on them. So we kind of get the most bang for your buck in terms of, if you're going to make, if you're going to donate say a thousand pounds where you can, [09:05] where you can do some of the most good with it. That wouldn't have to be the case. You could imagine a case where there's some huge challenge we face, but it's really quite intractable. And there's not much or there's not much individuals can do about it. And I think that global poverty is at least more tractable with respect to money, at least money for people in America or the UK or Australia in the richer countries, you know, where people are among the, say, the 5% richest people in the world. It's then not so surprising that money, [09:35] the most of, could be something that we could do to really help those who have much less of it. [09:39] But in the case of, say, risks posed by artificial intelligence, it does become a harder story about what's everyone supposed to do about it. Yes. And when we're talking about sort of these more tractable problems, things on the order of sort of malaria nets, deworming initiatives, that sort of thing where you really can, with small amounts of money, make like a very immediate impact. Is that a fair characterization? Yeah, they're the things that I was particularly drawn to.

[10:09] Yeah. [10:10] I think, you know, there's interesting different opinions on giving. Some people just aren't very interested in helping others. And I'm not going to convince them otherwise. But there's a lot of people who I think would be tempted to do this, that they really feel the pull of it. But there's some kind of blockers or things that stop them. And one of those blockers is a feeling that, do we really know it helps? [10:40] say, well, to the extent to which you really just want to make sure that you make a difference, here's something you can do that is very likely to do so. And so I was particularly interested in those cases, as well as the possibility of more speculative things for people who are willing to go a bit more out on a limb and, for example, fund some new technology that could help people in poor countries, but maybe it will never pan out. Maybe they can do even better than these tried and true things. [11:05] But at least the tried and true things can create a nice kind of baseline for the idea of should you keep the money in your pocket and spend it on yourself? At least if it's some kind of skepticism about whether it will make a difference, then finding those reliable things is very useful. Yes. Also important to... [11:22] get into the sort of mode of talking about these things. I think another blocker is that people feel, you know, they shouldn't mention it or, you know, they're inviting too much attention, but actually by signaling it. [11:32] what you do, you sort of normalize that for a lot of other people. And yeah, so I'm glad we're talking about it up top. I'm certainly not at the 10% rate, but I'm a recurring donator to give well with the idea that I can't count on my empathy to Spike with enough regularity that I remember to do it. So I may as well just sort of put it on autopilot. But anyway, that's a really, yeah, I'm glad we sort of got into that. And that sort of led you into some of these

[12:01] these questions and studying with [12:03] you know, one of the most prominent moral philosophers of the 20th, 21st century in Derek Parfit. Also, you know, in a very crowded field, one of the best sets of hair of any philosopher, I must say. But I'm curious what it was that, you know, you learned from him as a mentor and how that sort of framed your worldview. [12:24] Yeah, interesting. So, I mean, one of probably the biggest effect that Derek has had on me was that I probably came to Oxford in large part on the strength that he was here as when I'd studied computer science and then also philosophy in Melbourne. [12:54] particularly. And [12:56] Yeah, he did amazing foundational thinking on the nature of personal identity. So what it means to be an individual persisting across time. And as someone who was thinking about artificial intelligence, some of his thought experiments seemed to, maybe humans couldn't do these things. For example, a human can't split in two. [13:15] And Derek had a whole lot of fanciful science fiction style thought experiments about this. But if it's a computer program, it's a lot easier to imagine. [13:26] at a certain point and then fork the process into two different processes. And it really does seem like you could have an individual, a kind of Y-shaped individual who has a certain kind of common path and then kind of branches into two different futures and ask a lot of questions. If a program could be a person and a program could be conscious and things like that, then you could ask, what would it anticipate before the moment that it was split into these two different environments? And so I thought a lot of his style actually really fit well with

[13:56] my computer science background, despite the fact that he was, or he saw himself as a very non-mathematical and non-technical person, and he tried to avoid any maths in any of his writing, which I found remarkable because it was, in some sense, it was so mathematical already. It had this real clarity of the logic. Wow. [14:14] I enjoyed some hobbies with him as well. He was a prolific photographer and took these breathtaking photographs, particularly in St. Petersburg in the snow and Venice in the mist. I enjoyed talking photography with him as well. [14:36] Amazing. I'll have to try and find those. I'm sure they're somewhere. One of the things that, you know, Derek Parfit, you know, was so famous for, as well as sort of the concept of identity is sort of the idea of caring about these future persons, the people that that come next. And that's obviously such an important part of thinking about existential risk. Perhaps you can give us a little pricey on [15:00] on that concept and how it factors into this topic. [15:04] Yeah, so kind of a couple of areas, related areas that Derek worked on were to do with what's called population ethics, which is... [15:15] A lot of our ethical questions concern individuals who are all alive at the time when the choice is made. For example, we're trying to work out whether to extend one person's life versus extending a different person's life with scarce medical treatment. Ultimately, though, in almost all standard moral cases, like whether to tell a lie or something like that, the person who would be lied to is someone who already exists.

[15:45] who will ever come to live. And so that could be because, for example, if there is a risk of extinction, then there won't be people in future generations. So those people will never be born, or those lives will never exist at all. And so there's a question about how to value that or think about it. Is that, do we lose the entire value of those lives? [16:05] So kind of taking them from what they would have been down to zero, or is it something different that we do? Maybe it's not as bad to have someone never exist at all compared to if they died when they were an infant. There's a lot of different ways of thinking about this. And so Derek Parfitt really opened up this, helped ask the right kinds of questions and then sketch some of the challenges with trying to answer them. He also, though, somewhat separately, but [16:33] But it goes together well. He was interested in the idea about actions that could have benefits over very long timeframes. [16:42] So when it comes to economists, they often do something called discounting, where they try to [16:49] In order to understand the value of actions that have effects at different times, they use a mathematical technique in order to say that the effects they have at times further into the future matter less. [17:03] And in some cases, this [17:05] totally makes sense. So for example, if there's going to be inflation, then getting a dollar in 20 years time won't be worth as much as a dollar now. Or if you know you're going to be richer in 20 years time than you are now, maybe you're a student and in 20 years time, you'll be kind of high up in some career, then a dollar, even inflation adjusted, will be worth more to you now when you're poorer. So obviously you want to adjust for those things. But most people,

[17:35] once you've adjusted for all of that, is just worth less if it happens later in time. [17:41] And Derek Parfitt, and in fact, most philosophers ultimately, rejected this. And so Derek Parfitt has nice thought experiments on this. So there's one where if you imagine hiking on a trail and accidentally breaking a glass bottle when you're having a picnic, and then the shards of glass, you're deciding whether to pick them up and take them with you or to just leave them there. [18:04] And suppose if you left them there, you knew that they would injure a child's foot. So this young child would be kind of stabbed through their shoes and be bleeding on the trail. They would survive, but they would have a very bad time of it. [18:17] Then his idea was, does it matter if you knew that that was going to happen? Does it matter if it was going to happen in one week or a year or 100 years or 10,000 years if there was going to be exactly one child and they were going to have the same amount of pain and suffering from this and so on? This is kind of thought experiment to suggest actually that the time of these things, if there's no way that it compounds or does something else like that, then the time seems to be irrelevant. [18:47] arms or intergenerational benefits. And then famously, he also wrote one of the first descriptions of existential risk as a major moral issue. So definitely, in terms of, I guess, getting back to your earlier question about the type of influence that he had on me, certainly what I've done since then has been very shaped by these ideas. Amazing. Yeah. I think in the book you have, I can't remember the exact phrasing of it, but the idea that if you do apply this discounted rate to future lives, you can get into these

[19:17] you are waiting the fact of someone currently having a headache more than a million people dying and suffering in however many hundreds of thousands of years in the future. And so it can get really wonky, but fundamentally, [19:34] I think the glass example was so clear that these are questions about what we owe the future and what we owe to the people that come next. And that's really what you outline, I think, so thoughtfully in The Precipice, which... [19:51] came out a few years ago, but it strikes me that it was [19:55] incredibly prescient immediately, which is not something that usually happens, and remains that way. And I'm saying that because it came out before COVID, and you talk a huge amount about pandemic risk. And you also talk a lot about AI alignment, which has obviously become more and more critical. But one of the sort of headlines, so to speak, is that [20:17] You put the odds at... [20:19] One in six that's in this century. We do incur the existential risk and humanity fails to live beyond this century [20:30] What do you put our odds at now? Have the last few years made you more or less sort of worried about this? [20:37] Yeah, I'm not actually sure. I think that if you look risk by risk, so some of them say nuclear war, I think the risk has gone up. Ultimately, when I wrote it, it was before the invasion of Ukraine. And subsequent to the book coming out, the UK has been actively threatened with nuclear war by Russia. And there are also a whole lot of questions about the possibility of

[21:05] getting into either a conventional or nuclear war directly between Western powers, you know, as part of NATO and with Russia. So, [21:16] That one had seemed forgotten and distant when I wrote the book, and I'm afraid it's just gotten worse. In addition to that, there's only one treaty that's still kind of bilateral treaty protecting, well, keeping the number of warheads low. [21:33] between the US and Russia. And that treaty has been extended. It was extended in the previous administration, but it can't be extended again. A new treaty has to be negotiated. Oh, wow. [21:48] But it looks like this is going to lapse. And the long history of nuclear warhead numbers going down and kind of slowly but surely creeping down further and further. Where we started to become [22:03] I think that probably actually that number is going to go up. [22:08] rather than down. And then the new challenge will be trying to keep it low rather than trying to get it lower. Reproliferation, yeah. Yeah. But when it comes to pandemics or when it comes to AI, I think that there's been a number of changes that have made it more risky and a number of changes that have made it less risky. And I think that they're quite hard actually to tease out the overall effect. Yeah. [22:32] Wow, that's really interesting. I would have for sure anticipated that you would feel more worried at this point. So I'm excited to dive into these a little bit. Before we...

[22:42] jump to AI risk. This is such a silly question on one dimension, but on another, it's one that I think of myself. [22:50] Why does it necessarily matter if humans go extinct? Obviously, that would matter a great deal to me and to my, you know, uh, [22:59] my children and maybe my future grandchildren. But should we really see ourselves as quite so important, especially when we can imagine that given the, you know, the, the, [23:11] magnitude of the universe, there is probably, you know, it seems more likely than not that there's other life out there. What makes us so unique? [23:19] Yeah, interesting question. So it may not be important that we're unique. So suppose you have a child and they live their life and they do a whole lot of activities that are kind of standard and kind of celebrated cultural activities in your milieu, dance and song and spending good time with friends and things like that. [23:49] the envelope of what's been done before or something. Or suppose there's someone who has a life like that and someone else who's doing more unique things and you ask which life should we save if we can only save one of them. It's not clear that we should save the more unique one or something. It seems like there's value in a life well lived, whether or not it's unique. But the ultimate answer to this question is unknown at the moment. But there are some different kind of leads we have on it, different intuitions people have, and I think that there's something

[24:19] So one way to see it is it would obviously be what's bad in the present if there was some extinction event, say. So all of the lives, the 8 billion lives that would be lost in that event. And I think we could all see that that would be terrible. We know that it's terrible when someone that we know is to die. And presumably it would be about 8 billion times more terrible if that happened to everyone, which is really very bad. [24:49] But I set that aside because that bit's fairly obvious. Then there's also everything that we would lose. So if you think about the future and all of the generations who could have come to pass, say that the next 700,000 years of humanity, if we get our kind of average amount of time, and all of the hopes and joys of all of the people who would have lived across 20,000 more generations of people. [25:18] art and achievements that they would have made over that time, the kind of fruition of the potential of humanity as we kind of evolve over that time. [25:28] So all of that would be lost. So that's a kind of view based on the future. So now we've got the past, sorry, we've got the present, what happens in the present and what happens in the future. And that one gets into population ethics, the future bit. There's this question about exactly how should we conceive of that kind of loss of something that never was.

[25:58] passing down their improved knowledge over time until almost everything I can see in this room around me is not a natural item, whether it's technological in the sense of what we say technological today, or something even like a drinking glass, which is still a technological artifact. It took us many thousands of years before we could develop glassblowing and so forth, [26:28] intergenerational cooperation where our society would be impossible without this partnership of generations. And then if our generation [26:37] were to drop the baton and be the one who [26:41] who kind of failed to pass that on and to kind of, you know, continue to improve this for the future. It seems like, you know, in some sense, we might owe it to the past to be part of this cooperation. You know, it's one of these pay it forward type situations where there's nothing we can do to help our ancient ancestors. But maybe the right role that we're trying to play in this cooperation, you know, is this paying it forward. So it might be this real dereliction of our responsibilities. [27:11] mentioning, which is that it may be that as well as the present, the future, and the past, it may be that there's some additional cosmic significance to humanity. It may be that the Earth is the only place in the universe where there is life, or the only place where there is consciousness, or the only place where there are beings that are moral agents.

[27:33] So beings that... [27:35] to act systematically to steer the world towards what is good or what is right in a way that maybe even if there were no humans but there were other mammals, maybe they would morally matter. [27:47] But they wouldn't be trying to move the world towards better situations. If they turned out to be destroying the ecosystem, such as through some kind of byproducts of their living, they wouldn't be able to kind of realize that that was a bad thing and to act so to correct that kind of side effect. And so maybe we're the only kinds of beings who can do things like that. So that's focusing on what makes us cosmically unique. [28:17] parts of the universe can come to know the laws of the universe itself. But I think that even if we're not unique in that sense, there's still a whole lot of importance which come from these other senses. There's still all of the deaths that would happen during the event. There's still all of the future lives and the meaning in those lives that would be lost. And there's still all of this failure to live up to the kind of hopes and dreams and expectations [28:47] Well, thank you for humoring my unlettered nihilism. But that was certainly a part of the book that got a wow exclamation mark with some of that Carl Sagan piece. And then, you know, I think you give such a beautiful personal example around this generational compact of when you had your daughter and you realize that.

[29:08] what it is that your parents gave you and ask how, you know, I'm never going to be able to repay this of you. And, you know, they sort of tell you, [29:16] You know, that's just how this goes. You can only pay it forward. It's not possible to, you know, pay it back. [29:46] back like 1% of what they did for me, not realizing that it was only this tiny fraction. If I'd realized the full magnitude of the thing earlier, I think I would have been a bit better as a child. So, I appreciated their answer. It was certainly a very helpful answer as far as I was concerned. Another kind of answer would be to say, well, we're getting old and you could visit us every [30:16] I don't know if they'd asked for that, I would have certainly had to think about it because, yeah, no, it's a... [30:23] I really take seriously this magnitude. And I also take seriously this magnitude of what we, you know, this kind of gratitude, yeah, to all of these other generations, that if you just look at kind of any little thing around you, like I'm looking at the computer keyboard, and as well as the materials it's made of, of like, you know, like aluminium, that is extremely difficult to work out how to refine that, you need high energy and so on.

[30:53] but that the letter forms you know that they're who you know and the development of the language and the electronics inside it and you know the rubber of the the cable that you know and just how this has been kind of come together from all different kinds of plants and minerals across the world and different processes even looking at at simple things like you know things made of brick or a wood and and the the improvements in carpentry and so on to get a surface as smooth as this and [31:23] planning and [31:25] It is bewildering that you could just look at kind of any item and even just a very simple thing and [31:32] and realize that in a lifetime of work, you wouldn't be able to create this thing from the ground up. In terms of standing on the shoulders of giants, I think it's a very useful humility [31:46] lesson to look around to you and really appreciate it. Yeah, that's a good exercise. It's almost a gratitude exercise, to put it glibly, but profound. With all of that said, perhaps we should talk a little bit about AI. And I was interested that you said it's not clear if that risk of an unaligned AI is higher or lower. What are some of the sort of different factors that [32:12] muddy that up a little bit. [32:15] Yeah, so back in 2019, when I was putting the finishing touches on the manuscript, the [32:23] The cutting edge AI systems were reinforcement learning game playing systems. So think of things like AlphaGo by DeepMind or the StarCraft and Dota playing action game type systems. So these are systems that

[32:46] Through this process of reinforcing them, giving them kind of rewards if they do something right and kind of punishments if they fail, and letting them kind of explore the space of actions in a game, you can often move very quickly through the human level, you know, from kind of very weak play, you know, quite rapidly through the human level. [33:16] a human amateur, just random moves through the human level and to superhuman play. I don't know if it was minutes during the human zone or something like that. And so that can be [33:27] a very rapid ascent. And then also these systems are very inhuman. They don't understand language. It's extremely unclear how you would explain anything to do with morality or what people value. So if you had some industrial system, suppose you use these techniques to train something to run a company or to produce a factory that produced a lot of widgets and trying to maximize the number of widgets it could produce or something. It might just do all kinds [33:57] actions. [33:57] because it wouldn't understand that the world around it was important in some intrinsic way. But now we've got these systems that are very different to that. Here are four key differences. One is that these large language models are trained based on human data, on a huge amount of human data, so about 10 trillion words of human data.

[34:21] And [34:23] They have read pretty much every book and every paper so that they know a lot of moral philosophy. If you ask them about the details of population ethics or various abstruse things, they can give pretty good answers to them all. And as Stuart Russell pointed out, pretty much every work of fiction is just in every page. There's some human judging some other human's behavior and whether it was found inadequate or not and why. [34:53] of these kinds of statements so that they would understand if the AI were to take certain kinds of actions, whether humans would judge it as good or bad, or perhaps whether it would be ambiguous. And there's just this vast wealth of information to have access to about that so that the information is in there, which is good. Also, if you train on human data and the pre-training phase of these AI systems, the first phase is this next token prediction. So you look at a whole lot of words [35:23] a form of imitation learning, where it's learning to kind of pick the kinds of words a human would pick next. And that fundamentally pulls you towards the human level, the human range of behaviors. [35:34] rather than doing some kind of bizarre set of behaviors that are totally different to us. So unlike things that might kind of just zip through the human level and then overpower us very suddenly, this is something that has had very rapid improvements, but pulling towards the human level, which is nice. So it might at least spend some time there before we apply additional techniques and allow it to kind of take off again beyond the human range.

[36:04] A raw large language model is just one of these next token prediction things. Once we do additional what's called post-training, so additional stages of training, such as reinforcement learning from human feedback, where we get them to produce some different answers, and then we mark the answers and tell them which one we liked and which one we didn't. That helps teach them how to follow instructions and things like that. [36:34] because they're not just pure imitators at that point. But there's still much more simple one-step things that don't do a whole lot of planning. They're not planning 18 moves ahead or something like that. So... [36:51] You've got these systems that are more like oracles than they are like agents. They're more like just things that you ask at a question, it gives you the answer. And so they could be much safer. They can avoid these types of cases where there's a subtle misalignment and then it goes off by its own and amasses a lot of power and potentially works out how to get humans out of the way. And then even then also the AI systems are generally very inscrutable. It's very hard to understand what they're thinking. [37:21] of the neural net weights. [37:24] And on top of that, when we build reasoning systems out of LLMs, by default, they reason in natural language, like in English. So you can kind of see what they're thinking.

[37:36] So they're just like, you know, a whole lot of ways in which the technology is less scary. It doesn't fundamentally start with goals. It doesn't start with long-term planning. It knows a lot about human morality, even if it's not necessarily influenced by it. And so the step is then to how to turn that knowledge into something that guides it. [37:57] But that's a way in which the technology that we've been given that was somewhat difficult for anyone to anticipate five or 10 years ago has turned out to make alignment a bit easier as a problem compared to what it could have been. [38:27] clear. You know, it's this sort of reinforcement learning style where you, uh, you have really unpredictable behavior that doesn't have the context of the world around it and isn't born in a human mind in some way. And then this, you know, [38:41] The rise of LLMs has more of this context and more of that human style of thinking. [38:47] This episode is brought to you by Brex. Fred Adler, the influential venture capitalist of the 1970s, was known for displaying decorative pillows in his office that featured a signature business philosophy. [39:00] Corporate happiness is positive cash flow. [39:03] In today's post-SERP environment, Adler's wisdom feels particularly relevant.

[39:08] as founders need to make every dollar work harder. [39:11] That's exactly what Brex delivers. [39:13] Their modern finance platform was built specifically [39:16] for startups like yours. [39:18] and designed to help extend your runway when capital efficiency matters most. [39:23] With Brex, you get global corporate cards with up to 20x higher credit limits [39:28] and no personal guarantee required. [39:30] Their banking solution has no minimums and no transaction fees. [39:34] while letting you earn high yield from day one. [39:37] with same-day liquidity. [39:38] Best of all, Brex knows you were born to build, not juggle spreadsheets and finance tools. [39:44] Their AI-powered platform brings cards, banking, [39:47] expense management, and travel all in one place. [39:51] It's simple, scalable, [39:53] and designed to get you back to what you do best. [39:56] building. [39:57] More than 30,000 companies, [39:59] including one in three U.S. venture-backed startups, trust Brex to help make every dollar count toward their mission. [40:06] Join them at brex.com slash mario. [40:11] Given that sort of... [40:13] evolution over the past few years, where do you see the sort of biggest risks remaining? Is it that there's some new paradigm that shifts things back in a frightening direction? I could also imagine that simply extrapolating from this point could create significant issues, maybe not existential, I don't know. But yeah, what should we be paying most attention to? [40:37] Yeah, so I think this is a good question. And

[40:41] Even in the case of [40:43] alignment risk, it's not clear that it's gone down. So I said the technology is in some sense better, but maybe the world around the technology is worse. So we'd had a certain amount of racing between different AI labs, even back five to 10 years ago. But that really went up a notch once Microsoft and Google got involved. And now we're in a situation where it's not just [41:13] racing between small labs of specialists. [41:17] but rather between trillion dollar companies, the largest companies in the world. And nation states. And nation states as well, yeah. So this geopolitical race between the US and China is also very worrying. And so the pressures from these races and also the general commercial pressure that comes from, I guess, the capitalist system applied to this race are very fierce. [41:47] AI. They were both created, I think, by idealists, or at least by people in their more idealistic moments. And there was a bit more of a chance that there'd be room for that idealism, rather than these additional kind of pressures on them to race. [42:02] So that's an example of where that one's going. [42:07] has also gotten worse in some way, even though it's got better in other ways. But then, yeah, back when I was writing the book, I really felt that there were a number of different kinds of risk from AI, but the chief one was this risk of AI takeover or alignment risk. So you could think of that as one AI system or a small number of AI systems that are fairly similar to each other, maybe different copies of the same model. Realizing that to achieve their

[42:37] that they needed to wrest power away from humans who would stop them if they saw them attempting to achieve this goal. And that they would try to accumulate power and eventually overpower us and take control of our future from us. In a fairly abrupt way that would involve violating our rights. Okay. But... [42:58] Another possibility that's arisen, I was aware of these possibilities already, but they seem smaller. And now I feel that these are all roughly equal in... [43:09] in size or threat. So another possibility is what gets called gradual disempowerment. And I think of this as a case of, suppose the AI systems never violate our rights, that we maintain sufficient control over them, that they don't break the law, but they're more successful than us over time at doing all the kinds of jobs that we do, which often uses the definition of artificial general intelligence. If that's the case, we should expect that over time, [43:39] trade with these systems, that a higher and higher share of the wealth eventually accumulates in AI hands. So ultimately, you know, [43:51] They're more effective at doing what we do and they out-compete us and their wealth grows faster than ours. And then at some point in time, they have half the wealth. And then a generation later, they have 99% of the wealth. And with wealth goes power. They could use that wealth to influence political elections, even if they can't themselves run for election. And we just might expect, just as if you're a minority in a country where some other subset

[44:21] Even if your wealth is increasing, if others are increasing much faster and the share of control of resources is all flowing into their hands, you might expect it to be a bad outcome for you and a bad and permanent outcome perhaps. So that's a gradual disempowerment story. [44:51] say [44:52] who's not the head of the country, say someone else in the White House or in the AI lab who tries to seize control of the AI system and have it kind of secretly uses it to try to plot, you know, to take over. Ultimately, if these AI systems are capable, eventually do become capable enough that they could seize control of the future by themselves, for themselves, then they'd also be capable enough that if they were what's called intent aligned, so that they could do what humans actually want them to do if asked. [45:19] then that a human could ask them to take over on their behalf. That's another kind of concern. [45:25] Normally there would be safeguards against an AI system doing these things, but maybe the human, if they run the AI lab or they work there, maybe they're in a position to remove those safeguards. [45:34] or if they're the leader of a country, they could command that they be removed. So that's another kind of concern. And then a fourth kind of concern is... [45:44] the building of various forms of weapons of mass destruction using AI assisted help. So in particular, the concern there is that AI systems could kind of democratize newer and kind of more powerful destructive abilities. So to kind of empower, you know,

[46:02] Many individuals, perhaps people just with an undergraduate biology degree to create some world-destroying virus or something like that. It doesn't have to be biology. It could be in other disciplines if new capabilities appear. So I think that those four different areas are all concerning to me. And I'm also troubled by the fact that I don't know which is the most concerning. And I'm troubled by the fact that a lot of the things you could do to help with one of them make the others worse. Yeah. [46:31] So some of them, for example, the issue is concentration of power. And other ones, like the weapons of mass destruction, the issue is the dispersal of this power. So open weights, for example, helps with one of these issues and hurts others. And so I feel really at sea for a lot of these levers that people argue about. I really don't know which way to pull the lever, whether it's actually helping or not for the overall situation. [47:01] very small sense to me and disconcerting on the broader scale because you clearly think about this at a much deeper level and much more than I do. But I do find myself really struggling with figuring out what level levers can we actually pull here that... [47:17] that unilaterally help. And on one dimension, the race dynamic is something I keep struggling with figuring out how we get around that because [47:28] If we sort of assume that [47:30] the major actors of these AI labs are

[47:33] Morally normal, which. [47:35] We could probably debate that, and there's probably actually significant variance, I would suggest. But just the sheer incentive structure and mindset that you get into when you are in these race dynamics really feels like it leads to... [47:51] uh, [47:52] a lack of circumspection, lack of sort of caution. Are there good examples historically of taking moral action in a race dynamically? You know, there's some perhaps from, uh, [48:04] from the nuclear race, but that was quite incomplete as you as you share in the book. [48:10] Yeah, I mean, I would say mainly bad lessons. Lessons about bad behavior rather than good behavior. I mean, one example there was it. [48:20] the Manhattan Project that I think is particularly salient is that the [48:25] And if you ask, why were all of these people, these academics who studied physics, working on building a superweapon that would be the most horrible weapon humans have ever built? Yes. And the answer for most of them was that they were concerned about Hitler's Germany having unrivaled access to those weapons and holding the world to nuclear blackmail. So they worked as quickly as they could to develop such weapons. [48:55] when he killed himself and then shortly after the war in Europe was over and Germany surrendered, they didn't stop. Only one of the atomic scientists actually stopped at that point.

[49:08] And you'd think that if there was one reason that was so powerful that it led you to make the worst thing that humans have ever made, and you hadn't yet finished making that thing, and then that reason disappears, you might hope that you would have used time for re-evaluation. And there were some reasons why America used the nuclear weapons against the Japanese, but they were certainly much weaker than those other reasons. [49:38] project, with the idea of shortening the war in the Pacific being the objective. [49:44] at least not many of them. So one of the things we learned there was a group of people doing something that had serious problems for reasons that they'd convinced themselves were important, but then just no longer being sensitive to those reasons disappearing. And there's a whole lot of cases like this. I mean, another example of, I guess, a smart person talking themselves into a bad argument, I can't remember who it was, but the chemical weapons for the Germans in World War [50:14] written documents saying that [50:18] He thought it could shorten the war and that even though it could kill a lot of people, the bullets were also killing a lot of people. And that if the war was shortened, then ultimately fewer people would be harmed by it, both soldiers and also the civilian populations. [50:34] And so he thought that ultimately, while the weapon itself would be brutal, that this would fundamentally be a good thing.

[50:48] And if you were helping the side that lost the war, that means that your technology was lengthening the war. [50:53] not shortening it. And so the idea that [50:56] He was smart enough to develop all of these chemical techniques and things and to run this argument, but not smart enough to realize the argument only works if your side wins the war and that there was like a 50-50 chance that your side would lose the war, in which case it doesn't work. Yes. I think there's a lot of these things where we see smart people, in that case, maybe not someone who is particularly idealistic, but it shows that it's quite easy to talk yourself into various things. [51:23] And I think that there's a lot of that going on at the moment, particularly when it comes to racing. Yeah. You talk about the unilateralist curse in the book, which is sort of the idea that... [51:36] All of us may judge these different actions that we could potentially take to, you know, let's say there are 20 people who might release the next AI model. Whoever is essentially most optimistic and sort of has the most Panglossian view of what might happen, they're the ones who release it. And then, you know, if it actually causes damage, everyone sort of reaps the damage of that or, you know, incurs the damage of that. [52:06] So just like how in an auction, it goes to the highest bidder. And so if all the bidders, if there are many different bidders and they all have somewhat noisy and inaccurate estimations of the value of some object, the person with the rosiest impression of the value of the object will be the one who wins the auction. So this is called the winner's curse that sometimes you can systematically end up...

[52:30] The very fact that you won probably means you overpaid. There's a very similar issue here where, yeah, if a whole lot of people could all unilaterally make some action happen, maybe it released the genome of smallpox or something like that. And a lot of different people are trying to think about the benefits and the drawbacks. And there are serious benefits of doing that kind of research. [53:00] in order to understand what mutations are needed so that you can be on the lookout for those mutations occurring naturally. There is an argument, a rationale there, but it's not obvious if that rationale wins out versus the harms of doing that. But if you just allow anyone to do it, then it will always be the people who end up with the most inaccurately positive estimation of this action who will be the ones who will make it happen. And so this is an [53:30] problem. [53:30] And there are ways around it. For example, you could get together with the other group of people who could all unilaterally make something happen. So say all the other people who could be releasing a model like this. And you could say, let's have a vote. [53:43] And if you just take the vote, it ends up reflecting the median estimate, the middle estimate, if you do a majority vote. And so that actually almost completely resolves the problem if you can do it. [53:56] In the case of a model release, it could run into antitrust issues if there's an issue of collusion to slow down a technology. So unfortunately, I think antitrust law could well be quite bad for us when it comes to these situations. There are appropriate mechanisms whereby companies can get together if they create a third mechanism.

[54:15] body, a standards body, which is open to new members and not just a fixed list of members. Then you get that body to recommend a course of action, and then you follow that action or something. There is an official way to do it. I worry that the extra effort and steps involved in doing that slows everything down, as in it slows these remedial actions down while the other actions are being sped up by the pace of investment, need to make a profit, and so on. [54:45] antitrust is do you think like uh if we were able to resolve [54:51] some of this race dynamic within the US, would that be sufficient? Because we are, you know, the sort of thinking there would be that [55:00] the most advanced models and most advanced capabilities are US centric right now. And so, by sort of removing this dynamic, we're reducing the existential risk from an aligned AI. Or is the fact that China is clearly taking this very seriously, even if they do seem to lag in some considerable places and have been cut out of the semiconductor supply chain in really [55:30] a different race dynamic where we're sort of actually hamstringing [55:34] Western powers and leaving the possibility of unaligned AI in a much less democratic environment. Yeah. I mean, this is another thing that's...

[55:46] related to the unilateralist curse, that if there's some issue that seems [55:52] at some level problematic, let's say taking some new step with a technology is at least at some level troubling, then if all of the people who are concerned about such things all stand aside and say they're not going to do it, then the person who has the fewest ethical qualms will be the person to step across the threshold and maybe we'll get the first version of something will be a more dangerous version of it. So that is a general concern. I think that it's also one of those, it's exactly the kind of thing that people in the position of the atomic scientists [56:22] use to rationalize racing. Totally. So while it is a [56:27] not a bad idea. It convinces different people different amounts and then you worry that the type of people who are erroneously convinced by it will be the people who take the actions. [56:39] I think it's a real challenge. As you say, there's kind of racing, roughly speaking, at two levels at the moment. There's this kind of company racing between different companies, most of whom are at least headquartered in the US. Then there's a kind of racing between the US and China. The US is clearly the one that triggered the race between the US and China. [56:59] And partly it triggered it because on the rationale that eventually the race was going to happen and we want to win it if it does happen. And so if we start the race, we've got more chance we'll be winning the race. [57:09] And there's a logic to that. [57:12] It still is a case of, you know, if you think races are bad and the racing behavior is kind of like defecting in a prisoner's dilemma, and you think that unfortunately the structure of the world is such that probably both parties are going to defect, so therefore we should defect first.

[57:25] maybe that's the right thing to do, but it's still kind of, [57:29] you know, still jerk behavior at some level. And so I think, you know, one has to admit that. And if, you know, if ultimately we succumb to some existential risk with AI, if you kind of imagine, you know, the different powers that be or all the different people involved kind of being called up to the pearly gates, you know, meeting St. Peter and trying to explain their behaviors, it's a little kind of thought experiment. You'd have to get into some Gabe theory. It wouldn't sound great, I think, if you said fundamentally, we knew it was really risky, [57:59] People would do it. Humans would do it even though they shouldn't. [58:03] And we thought there could be a race. So we started the race. And did you attempt to have a treaty to not race? No, we didn't attempt that. We thought it wouldn't work. So we started the race and then we raced and we got there first and we built the thing and the thing killed us all. I just think it'd be pretty difficult to really... [58:20] If you add a kind of imagined your honor at the end of these statements, they don't sound great to me. And I think that as it happens... [58:29] Given that China is behind in this technology, [58:33] I think that they have additional reasons to not want to race. So if the US were to think, as I think they should, that we would rather there be no race, and China would prefer that even more because the race that there would be is a race that I'd likely lose as well. [58:49] then I think China could be in quite a position to be happy with some kind of verification conditions or things around kind of avoiding a race. So I feel that there's been very little...

[59:04] thought going into not racing. [59:07] And I think that if you pretty much just had those two countries agree, they could potentially police their own spheres of influence to say, okay, no one here is building things towards superintelligence. No one over there is doing it. And then have verification on each other's AI industries. Such that no one cheats it, essentially. Yeah. For example, by saying, okay, well, we'll send 30 of our people over to be part of your AI companies, and you can send 30 of yours to be part of ours or something to make sure that [59:37] this. In the Cold War, I think there's a lot of lessons from the Cold War. Ultimately, [59:44] There was a situation where the US and the USSR were very much adversaries of each other, if anyone is. And yet, they actually treated each other as adversaries in what I think is a somewhat grown up way that we're not doing these days. I think that the US is treating China as an enemy, not an adversary. And they just kind of want to get China. Whereas with the USSR, I think there really was a realization that there are a bunch of things that were in their common interest. [1:00:14] the non-proliferation treaty was in their common interest. If the US has nuclear missiles and the USSR does as well, they don't really want all the other countries to have nuclear missiles. It's in their common interest that this would be an elite kind of exclusive club. And it was also, I think, in most of the world's interest. And so they helped to broker and police this non-proliferation treaty. And similarly, their arms reduction treaties that they had, because they both would

[1:00:44] And they managed to do that. And there were very smart people working out very novel verification techniques. There were these ideas of once spy planes became possible, of allowing the Russians to fly spy planes over the US airfields while they sword their bombers in half. [1:01:01] And they pulled the halves apart from each other with tractors. Wow. And those bombers are still there on the airfields, sawn in half as a costly demonstration of destroying their military capacity. Wow. Even without inviting inspectors over, they worked out ways to credibly show that they were reducing their arms. [1:01:21] And so, [1:01:23] It's a... [1:01:25] There were really people thinking outside the box is what I'm saying. And there's a lot of very smart people today thinking about these issues. A lot of them are going to work at AI companies and build these technologies. But there's also a lot of kind of complaining that we have of like, oh, verification's too hard. There's no way we can verify a kind of deal to not race. [1:01:55] three full-time equivalent people working on that or something in the whole world. If that. And there's probably almost nothing has been explored. So I think that there's a lot of just giving up, oh, that'd be way too hard without really trying to work out other really clever technological or sociological techniques. Yeah.

[1:02:14] That's really interesting. I want to touch on one of the other areas briefly that you talk about in your book, which was the extremely prescient and immediately prescient part, which was pandemic risk. You mentioned gain of function research and sort of thought through the trade offs there. [1:02:34] What have been the lessons of COVID that you take from that kind of research in general and how risky it is? And, you know, more broadly, [1:02:42] Did COVID make us more prepared in some ways for what comes next? Or have we really not... [1:02:48] taken almost anything from it. So one thing it did, [1:02:51] was that when I was writing the book, I was certainly concerned that people were just not going to take threats to humanity's entire future seriously. That there was a feeling that, I don't know, that there was a time when these things were possible and that that time was in the past. Maybe in the case of pandemics, a feeling that, oh, sure, there was this 1918 flu that I've heard of. And there was the Black Death, I guess, but that was medieval. And technology's advanced so much since then. And we do so much in our hospitals. We understand the germ theory of [1:03:21] things. And so we're doing much better at this. And so it's not a risk. And COVID certainly reminded us that we're still at risk and that for all of our technologies, in some ways, we're just as vulnerable as we were before. So when it comes to dealing with individual infections and trying to save someone's life in the hospital, we're much better at that.

[1:03:51] disease because the airplanes that we've invented and things, the cities that we've built and the public transport systems and so on are so efficient at just spreading this disease around. A lot of our technologies make it worse at a similar pace to the way that our protective technologies make it better. And it's not obvious at all that we're actually more protected overall when it comes to pandemics. So that's... [1:04:17] It's helped give us some of that kind of reminder and also [1:04:21] It also was this feeling, I think that, you know, when it started, right, there was this thing where you're like, hang on, what, you know, is it going to be the case that everyone in my country is going to be at home in their house? Yes. And the streets are just empty for days, you know, days. I mean. [1:04:37] Months? Yeah. Is it a school's going to shut down? And there was this feeling of like, is that, should that even happen? And, [1:04:43] In general, I think, you know, for people who'd lived through, say, the Blitz in London, you know, that really shaped their view and they felt, oh, well, we're not even allowed to have lights on at night or something. And they've been aware that the space of ways of living, you know, could be quite different in the space of kind of big things that could happen could be quite big. Ultimately, COVID, you know, was probably this biggest world event since World War II. Yes.

[1:05:13] is set by the biggest one you've lived through so far. And so at least it's helped raise the ceiling on what we can imagine could be required of us and how different life could be. So that's been somewhat useful for kind of letting people imagine more. I mean, also, I guess, in terms of political trends and political populism as a rise of this movement and so on. [1:05:36] If you lived through the 90s and the 2000s, it had really felt like we were in some really stable era of some sort. Yes. And then we've come out the end of that. And that took quite a while to learn those lessons and to kind of realize that, at least for someone of my age, that the kind of world that I'd, the only world I'd known as an adult could be kind of coming to an end as a geopolitical order or something like that. It's helpful to take that wider view. And COVID has helped us do that. [1:06:06] So in some ways, it's helped us kind of realize that for other threats as well. It was thought by almost everyone that there would be what's called a panic neglect cycle, where having been hit hard with something, that you overreact and that the governments would kind of would do more than they should. Either they'd spend more money or they would kind of like inhibit freedoms too much or something like that to react against this thing. And then that would slowly fade away until it was too little. But that hasn't happened. We actually didn't get the panic phase of that. [1:06:36] time everyone had reopened after COVID. [1:06:40] I think people were just so sick of it that they didn't even want to talk about funding these bills, you know, to spend even a tiny fraction of the amount of money that we would cost or the amount of damage that was caused by COVID to prevent future such things happening. So that was a really missed opportunity. And I was surprised. I would have thought that you'd get this panic. And then the question was how to soften the panic into the appropriate level of response, but then how to maintain that appropriate level as well.

[1:07:10] if we got the appropriate level. [1:07:11] And I also feel that it had some good features like the mRNA vaccines, getting a test out and flexing our muscles on this new technique of very fast vaccine development was good. [1:07:26] And I guess I don't know overall, you know, whether how it's really helped our preparedness. I think that, again, you can kind of tease out like, you know, four or so of the biggest factors kind of that either made it better or worse, or perhaps in some cases have not made it better or worse, but revealed underlying structural weaknesses. [1:07:56] vaccines and various things like this. [1:07:58] I guess we learned that we can band together for a while. [1:08:01] a bit of while might be a six-month period. If the crisis carries on for years, that that really does start to fragment. Yeah, but I think that's true that your point on... [1:08:17] the way in which this totally shifted our frame of reference from this can never happen, that would be as fanciful as a zombie apocalypse to actually we all lived through it. And also Operation Warp Speed with the development of vaccines is maybe a good example of a beneficial race dynamic and where those things can work favorably. Yeah. I mean, it's even, I think the

[1:08:47] cooperation and so forth. Yeah. You mentioned the political landscape and the institutions that [1:08:56] played their parts in COVID. One of the things that I've been [1:09:01] Thinking about and reflecting on the book and some of these questions is how compatible are democratic institutions with addressing with existential risk? You know, there's sort of this time span of discretion where your senators are thinking on, you know, I think they got six years. I actually can't remember now. Six year cycles. Your presidents are thinking on maybe two and a half because then they start thinking about, you know, the next election and so on and so forth. [1:09:31] incentivized to think, [1:09:32] about [1:09:33] maybe even 10 years, that would actually be a really farsighted person. How can we sort of [1:09:39] address these existential problems with such a [1:09:44] short-term mindset from the institutional level? Yeah. So it's a real challenge. It's not necessarily, you could say, a challenge of democracy in the sense that if you didn't have democracy, if you had some kind of autocracy, it's not clear that makes it better at all. If the autocrat happened to care about this issue, maybe it would make it better. But if they don't happen to care about it, then they wouldn't really be able to get the signal coming from the people. [1:10:08] I'm not sure that it's... [1:10:10] democracy itself that's to blame for it. But some things happened to the democracies that we have in Western countries, at least, to really shorten the timescales that they focus on.

[1:10:26] I'm not exactly sure what's going on there or why that happened, because it does seem like the democracies, if you go back 50 or 100 years, it does seem that they were able to achieve larger projects and projects that operate on longer timescales than they currently can conceptualize. [1:10:56] they asked on Monday or something. And so everyone's thinking in terms of the news cycle, not even the election cycle. I don't know what it is, but it does seem like the time span, the time horizon for politicians has gotten shorter. [1:11:09] And there's also, though, separately from whether it would be better to not be a democracy, there's a question of, is there a kind of failing still of the current democratic system to perhaps live up to the principles behind democracy when it comes to this issue? [1:11:39] say, technologies like AI, for example, deciding actively to not regulate and prevent regulation seems to be an active decision to do something negative there. But these are decisions that, if you make them in one way versus another way, may well have implications that just echo through the generations, making people's lives significantly worse for many generations to come,

[1:12:09] that there's this issue that most of the stakeholders or the people who are affected by it don't have representation in that decision because they're people of future generations. And in some cases, we can also predictably work out which way they would vote. They would vote for doing more action against climate change, for example. And so it's not even just that there's [1:12:39] constituency or stakeholder group that is [1:12:43] just not getting the franchise. Yeah, that's really interesting. It's somewhat similar to questions about people of different races or sexes or landholders versus non-landholders and other types of issues in democracy of extending the franchise. And so I think that there are potentially things that a democracy could do about that. For example, in the UK system where we have the House of Commons [1:13:06] which is a bit like the House of Representatives in the US. We also have the House of Lords, which is an appointed body, some of whom are appointed by the Prime Minister, some of whom are appointed as the kind of good and great of society, former heads of the Royal Society and things like that. And it would be possible to appoint a group of lords to represent future generations, or you could do it in other ways. You could have a Citizens Assembly, for example, [1:13:36] that experimenting with tweaks to the current understanding of democracy to deal with the fact that now so many of our choices, our most important choices, could be affecting all of these stakeholders who don't have representation. The idea of no taxation without representation, foundational to the American ideas of democracy. And yet, maybe we need to take it seriously in this case of future generations and to work out, are there ways to appropriately implement?

[1:14:02] understand their interests and then also to force some kind of representatives to represent those interests. Because obviously the people themselves who live in the future can't vote in this thing. It doesn't mean they can't be represented. [1:14:14] That's a really fascinating idea. Yeah. Having these sort of stewards of the future as part of the governance process. Or even if it seems to be too impractical to at least set it as an ideal to say, oh, if only we could do this, that would be great. Unfortunately, we can't or something. But it seems at the moment, people don't even think about it or don't realize that maybe there are things we can do about it. Yeah. There's sort of the implication that people suspect, sort of hope that politicians are thinking about the future. But I think the reality is that [1:14:44] That is, as you said, has sort of long since gone. Yeah, so that's a nice way to at least consider it. [1:14:50] And we like to sort of end episodes with a few sort of more abstract thought experiment type of questions. So with that in mind, if you had unlimited resources and no operational constraints, what experiment would you like to run? [1:15:04] I think I have no idea. Oh, wow. Great. Okay. That's a... Apologies. No, no. This is a sign that you wouldn't want to go off on a wild experiment without thinking it through, which makes sense given your background. I guess that's part of it. There's a famous comment from Carl Sagan when he was asked about, are you really sure about nuclear winter? [1:15:31] You know, how could you know this is experiments never happened? You know, and he said, you know, it's unfortunate, you know, that we have only one Earth. We have no spare Earths to destroy in some kind of experimental laboratory in order to determine the exact reliability of this theory. So if we had unlimited resources, you know, we could have those spare Earths to destroy in order to find out what these risks actually are. But unfortunately, that sounds like the kind of experiment that's probably ethically a no-go regardless. Yes, that's right. That makes sense.

[1:16:01] Well, let's try this one. If you had the power to assign a book to everyone on earth to read and understand, what book might you choose? I might choose Practical Ethics by Peter Singer as a book that's... [1:16:13] It's a really good kind of no-nonsense approach to thinking about ethics. It really helps show how to kind of reason about all kinds of issues with an open mind, not assuming that our current kind of ethical intuitions are the be all and end all. And, you know, I think it's a good example of that kind of thought. Is there a sort of a particular framework or sort of lesson that you particularly reflect on from that book from Singer? I imagine he's a major influence of yours. [1:16:42] Yeah, he is. Probably just really what I said then of the fact that he just kind of models this kind of really open inquiry into moral matters. Derek Parfitt mentions that ultimately there's been only really a couple of generations in which people have made what he calls non-religious ethics their life's work. [1:17:12] been within a religious context where the set of possible answers always had to kind of align to what had already been thought thousands of years earlier. But people trying to, separately from that, to try to understand these questions, there'd only been a couple of people in the whole history of philosophy. And then there'd only been a very short time since it became

[1:17:42] that will make some substantial moral progress. Huh, fascinating. What tradition or practice from either another culture or time period do you think we should widely adopt? [1:17:52] I think I don't know enough about particular time periods. This is the problem about being an academic and wanting to, wanting to not just make something up. Uh, but I, I do feel that, uh, [1:18:01] Something to... [1:18:03] something that involves more gratitude to our ancestors. [1:18:07] I think would be particularly important. I know that there are many cultures that have more gratitude to our ancestors than Western cultures, which have almost none. But I'm not sure exactly who we should be recopying it from. But I would love to see more of that. Yeah, I agree. That's a great one. Well, Toby, it's been such a pleasure. And yeah, I can't thank you enough for your work and your writing and for your time today. [1:18:30] Oh, thank you. It was a wonderful conversation. [1:18:44] Ratings and reviews help others discover these discussions, so if you enjoyed the conversation, [1:18:49] I'd be grateful if you could take a moment to leave one. [1:18:51] For all past episodes and more, [1:18:53] Visit us at thegeneralist.substack.com. [1:18:57] dot com. [1:18:58] See you next time as we continue to explore. [1:19:00] the future. [1:19:02] .