Thanks for writing this piece. I'm glad you turned your lens towards AI Doomerism. You make a lot of good points, and I agree with lots of what you said, but I think you overstate the importance of these assumptions. My main disagreement is that I don’t think these assumptions are all required to be concerned about AI Doom. Here’s an assumption-by-assumption response.
1. Intelligence is one thing.
I'm very concerned about AI Doom and I do not believe that “intelligence is one thing.” In fact, when we talk about “intelligence,” I, like you, believe “we’re not pointing to a singular, spooky substance, but gesturing at a wide variety of complex, heterogeneous things.” It’s, as you say, a “folk concept.” But this is true of many concepts, and that illustrates a limitation in our understanding—not that there isn’t something real and important here.
Imagine that intelligence isn’t a single thing but is made up of two components: Intelligence 1 and Intelligence 2. Further, imagine AI is only increasing at Intelligence 1 but not Intelligence 2. We don’t know enough about intelligence to clearly define the boundary between 1 and 2, but I can tell you that every time OpenAI releases a bigger model, it sure seems better at designing CBRN weapons. This pattern of improvement in potentially dangerous capabilities is concerning regardless of whether we can precisely define or measure "intelligence."
You say that “the word ‘intelligence’ is a semantic catastrophe” and I agree. But that’s true of many words. If you don’t like the word “intelligence”, fine. But I would argue you’re holding that term to a standard that very few, if any, concepts can meet.
The point is, you still have to explain what were seeing. You still have to explain scaling laws. If you don’t want to say the models are more intelligent, fine, but something is definitely happening. It’s that something I’m concerned about (and I think it’s reasonable to call it “increasing intelligence”).
Time and again, when the GPT models have scaled (GPT -> GPT-2 -> GPT-3 -> GPT-4), they have been more "intelligent" in the way people generally use that term. Would you argue that they haven’t? Intelligence isn't one thing and it's messy and yes, yes, yes, to all your other points, but this is still happening. If you don’t want to call this increasing intelligence, what would you call it?
To show you that I’m talking about something real, I will make the following prediction: If GPT-4 were scaled up by a factor of 10 in every way (assuming sufficient additional training data, as that’s a separate issue), and I got to spend adequate time conversing with both, I would perceive the resulting model (“GPT-5”) to be more intelligent than GPT-4. In addition, although IQ is an imperfect measure of the imperfect concept of intelligence, I predict that it would score higher on an IQ test.
Would you take the opposite side of this bet? My guess is “no”, but I’m curious what your explanation for declining would be. If it’s something like, “because models become better at conversing and what people think of as intelligence and what IQ tests measure and designing dangerous capabilities, but that’s not intelligence”, fine, but then we’re arguing about the definition of a word and not AI Doom.
2. It’s in the brain.
In humans, it’s mostly in the brain, but there are some aspects of what some people call “intelligence” that occur outside the brain. The gut processes information, so some might argue it exhibits a degree of intelligence. This doesn’t seem relevant to the AI risk arguments though.
3. It’s one a single continuum.
Again, I agree that the word ‘intelligence’ is a ‘semantic catastrophe,’ and it’s more complex than a single continuum. Not everything we associate with intelligence is on a single continuum. But, again, I’m willing to bet money that the 10X version of GPT-4 would be better at most tasks people associate with intelligence.
4. It can help you achieve any goal.
You’re making it seem like AI Doomers believe intelligence is equivalent to omnipotence. It’s not. Even if it’s hard to define, we all agree that it doesn't directly regulate body temperature. It can, however, in the right contexts, allow a species to create clothes that regulate body temperature, antibiotics that speed up recovery, spears that keep predators away, and so on. It's an incredibly powerful thing, but it has limitations.
As for why it hasn't evolved over and over, it's expensive. In humans, it's about 2% of our body mass and consumes about 20% of our energy. On top of that, it requires longer gestation periods or childhoods. Something that costs that much better pay off in a big way. It did with humans, but I don't see how it would for lots of other niches. I imagine that the more an organism can manipulate its environment—say, by having hands to move things around or legs to move itself around—the more useful intelligence would be. It would not benefit a tree very much. Do you really think a really smart crab would have a sufficient increase in genetic fitness to make the cost worth it?
In the right contexts, though, it’s incredibly powerful. Our intelligence allowed cumulative culture, which is why we’re the dominant species on Earth. It’s why the Earth’s mammalian biomass is dominated by humans and the things we domesticated for our consumption. Humans decide which other animals go extinct. It’s why humans can sit around tables and say things like, "California condors are critically endangered. We like them so let's make an effort to bring them back. The Tecopa pupfish is critically endangered, but those new bathhouses are bringing in lots of tourism money, so bye-bye pupfish."
5. It has barely any limits or constraints.
You bring up good points about constraints. I agree that “real life is complex, heterogenous, non-localizable, and constrained.” Intelligence has constraints. It’s not going to build a Dyson Sphere overnight. The world has friction.
It’s worth thinking carefully about how significant these constraints will be. They certainly matter—the world of atoms moves more slowly than the world of bits.
But we shouldn’t be too confident assuming the limitations of a superintelligent system. I doubt people would have predicted Satoshi Nakamoto could become a billionaire only through digital means. Certainly, a superintelligent AI could do the same. Where in this chain does the AI fail? Could it not become a billionaire? From that position, would it not be able to amass even more power?
I think there’s a lot more that could be said here, but I don’t know how much this is a crux for you.
AIs certainly have intelligence by many definitions of that term. You seem to be talking about agency here though. I would point out that intelligence alone does make something more agentic. The weights of a very intelligent LLM—even a superintelligent one—would just sit there on a hard drive. They’re not going to magically break the laws of physics and becomes agentic.
This doesn’t mean, though, that if you put it on a server where information constantly flows through it, it won’t act in a way that we consider agentic. It could be the case that making a sufficiently intelligent AI agentic is not that hard—perhaps a task for a single software engineer. It could also be the case that a non-agentic system could be quite damaging in the hands of a malicious human.
7. AIs have been getting more of it
Again, you make a point about agency by talking about "unprompted" messages. A collection of weights on a hard drive will never become more agentic. However, each generation of AI is becoming more capable at the tasks we give it. GPT-4 consistently outperforms GPT-3 across virtually every benchmark. Do you dispute this pattern of improvement? The lack of spontaneous agency doesn't negate the reality of increasing capabilities.
8. An AI will soon get as much as (or more of it) than us.
There are certainly diminishing returns to just gobbling up data, but that doesn’t mean AI won’t keep progressing. We don’t know if or how well scaling laws will hold.
I doubt data is the limiting factor. I could go into this more but only if it's a crux for you. The short answer is synthetic data.
But instead of theorizing, just look at the most recent models. They’re not getting more mediocre. They’re becoming superhuman at many tasks.
9. Such a (super)human-level AI will become good at every job.
They won’t be good at every job for a variety of reasons. The idea of having an AI as a fitness instructor telling me to do more pushups does not appeal to me. AI wouldn’t take every job and the idea of comparative advantage will still apply. None of this is an argument against AI Doom.
10. And it will become good at ending humanity.
AI labs already work hard to prevent AIs from answering questions about building CBRN weapons. It seems straightforward to think they if they stopped doing that it would get better at these tasks. Currently, the publicly released ones are not good enough to do massive harm. How good could they get though? Would one be able to explain to someone how to engineer a dangerous pathogen, and how to do it without getting caught? I don’t know and I’d rather not find out.
11. And it will want to end humanity.
It’s hard to say what it’s going to want. AIs are more alien to us than even an alien would be. They’re not evolved and their “intelligence” is going to manifest differently than ours.
What if it just wants to make sure it can never be turned off? Does humanity come out well in that case?
Even if we assume this assumption is false, the underlying concern remains. Using the example above, we’ve still shifted from being the decision-makers around the table to being the animals whose existence depends on the preferences of a more intelligent being. We’re the condors if we’re lucky or the pupfish if we’re not. This does not sound like a great situation.
There’s much more to say here but I don’t know if this is much of a crux for you.
Uncertainty Should Increase Concern
One more point I want to make is what I consider to be the biggest flaw in the thinking of non-doomers. Many say something like, “The Doomers can’t be certain that AI will kill us all, so we’re safe.” This is not the right way to think about it. I posit that anyone who believes there’s more than a 10% chance of some time of AI doom scenario should be a “Doomer”.
Imagine your child is downstairs in the basement playing Russian Roulette. It’s a six-shooter, so you shouldn’t be scared—that’s less than 17% chance they die—barely worth telling him to stop. Of course, this is ridiculous. No one thinks this way. But, for some reason, when you replace “child” with “child and their friends and everyone you know and, in fact, all of humanity”, instead of their concern going up, people’s thoughts go to “Haha math is for nerds; probabilities are dumb; this is fake” mode. These are cognitive biases (mostly scope insensitivity, but also some identifiable victim effect) that we should avoid if we want to think clearly.
The core issue isn't whether we can perfectly define intelligence or whether AI will have exactly human-like capabilities. It's that we're developing increasingly powerful systems without fully understanding them or being able to reliably control them. When the stakes are potentially existential, uncertainty should drive more caution, not less.
It’s not clear exactly how this will all play or precisely what “intelligence” is or what’s going to happen next. But the core arguments don’t rely on the assumptions you stated.
Thank you for writing this, Julius. This is the best response I've read so far.
Re "intelligence," I'm glad we agree the concept is a semantic catastrophe. But you seem to be suggesting this is not a problem for AI doomerism, and that I am holding the concept to "too high of a standard."
I disagree. Compare "intelligence" to "asteroid impact" or "nuclear bomb." One of these is not like the others. If you're telling me I'm going to die from an asteroid impact or nuclear bomb, I understand what you're talking about. If you're telling me I'm going to die from an artificial "superintelligence" in some unspecified way, I have no idea what you're talking about. If your claim is "x is going to kill you," the onus is very much on you to say what x is in clear and precise terms. I do not think that is too high a standard at all. Anything less than that standard would leave you vulnerable to all sorts of cultish, apocalyptic bullshit.
Re intelligence being in the brain. The argument here isn't that our gut neurons are super-important. The argument is that intelligence is likely an emergent product of millions of humans and tools interacting over multiple generations under the right kinds of incentive structures. Based on my understanding of psychology, cultural evolution, and economics, that seems to be the most plausible explanation for humanity's extraordinary power. It's not because of one brain.
The problem is, doomers need intelligence to be one brain, because it's more plausible for AIs to be approaching one brain than for them to be approaching the entirety of human civilization, the entirety of the western world, or the entirety of the global economy. If intelligence is a property of institutions (which seems very plausible to me), then AIs are nowhere near it right now. We'd need something like an autonomous nation of AIs organized under the right kinds of incentive structures. So I really do think doomers need intelligence to equal one brain for their case to be even remotely plausible.
You mention scaling laws with regard to ChatGPT. You suggest that we can quibble over the semantics of how ChatGPT has changed, but that "something is definitely happening" there. I agree that something is definitely happening. I would even agree that something very impressive is happening. I'm just as impressed by these machines as anyone else. They're very impressive!
But there is a large gap between "something impressive happening" and "something existentially threatening to humanity" happening. You need some way to bridge this gap, and I honestly don't know how you're bridging it.
The conventional doomer way of bridging it, as I understand it, is to describe the "something impressive happening" as "intelligence increasing," and then make all sorts of dubious assumptions about this "intelligence" thing, including its location in a single brainiac (rather than an entire society or economy), its placement on a single continuum (which AIs are destined to progress along), and its capacity to achieve virtually any goal without limits or constraints, including the goal of destroying humanity, which it will surely possess. I think this way of bridging the gap is unconvincing, and you seem to agree. But then you need to give me some alternative way of bridging the gap. How do you get from "something very cool and impressive is happening" to "we're all going to die?"
You asked me how I would explain scaling laws. My understanding (and I could be wrong) is that LLMs are designed to predict text, and as you train them on more and more text, and give them more and more neurons to store their textual knowledge, they get better and better at predicting text. I think that is what's happening. LLMs have acquired a lot of cool abilities, but none of these abilities lie beyond what would be useful for predicting text. An IQ test is made of text. Answering the items is a kind of text prediction. It makes sense that LLMs would get better at that task. That doesn't mean they will get better at every possible task, including tasks unrelated to text prediction, and it certainly doesn't imply they are on a path to killing us all. Heck, it doesn't even imply they are on a path to getting better at text prediction. There may be diminishing returns, or the returns may already be diminishing, as many have argued.
Re intelligence achieving any goal, I'm glad we agree it cannot. It's not omnipotence or anything close to it. But there is a reason why doomers flirt with this idea of omnipotence. As before, the dubious concept is serving an argumentative purpose: it is bridging a gap. In particular, the gap is between "intelligence can get you a lot of things" and "intelligence can get you the destruction of humanity." That is quite a big gap. Doomers bridge the gap by replacing "a lot of things" with "everything," so that the claim is true by definition. But you seem to reject this way of bridging the gap. So again, I would ask you for an alternative way of bridging it. If the list of things that "intelligence" can do for you is finite and constrained, as you seem to acknowledge, then why should we expect the destruction of humanity to be on the list?
The same thing goes for your other gap-destroying tactics. You agree there are many constraints on goal achievement. The global economy is the most intelligent thing in the world right now, and yet it couldn't satisfy consumer demand during the COVID pandemic (what an idiot). Presumably we're not building an AI-equivalent of the global economy any time soon, and presumably a real future AI would be even more constrained in its goals than the global economy. So if you agree with me that intelligence is constrained, then there will surely be many constraints to destroying humanity, assuming an AI will develop that goal. And there will surely be many constraints to building an AI that could stand a chance of, or even take an interest in, destroying humanity. Is there any reason that destroying humanity will be the exception to the very general rule of things being constrained? And I haven't even gotten started on financial and political incentives, and the division of labor, and how AIs will specialize in things like everything else, and the way humans have domesticated anything even remotely "intelligent." I need to hear some arguments as to why we should expect AIs to be the exceptions to all these rules. In rejecting the standard doomer tactics, you are encumbering yourself with a very large burden of proof.
You write that "The core issue isn't whether we can perfectly define intelligence or whether AI will have exactly human-like capabilities. It's that we're developing increasingly powerful systems without fully understanding them or being able to reliably control them. When the stakes are potentially existential, uncertainty should drive more caution, not less."
Well I think this begs a lot of questions. It is assuming that these systems are increasingly "powerful?" How powerful? They can't do anything on their own without being asked. They obey our every command. This strikes me as the opposite of power. (And if you want to claim that they are playing nice and biding their time while plotting our demise your are indulging in the worst kind of unfalsifiable conspiratorial thinking).
Has their power been "increasing?" As I mentioned in the piece, their ability to spontaneously do things of their own volition or in defiance of us has stayed flat at zero. That does not strike me as "increasing power."
Do we not "fully understand" them? Maybe, I'm not sure. I think we can say that LLMs are text prediction machines, but I grant that their inner workings are mysterious. I'll let that one slide. It would be nice to learn more about them.
"Without being able to fully control them?" Again, it seems like we are pretty close to fully controlling them right now. And all the economic and political incentives point toward more--not less--control. If our control over them starts to rapidly disappear, maybe I'll get concerned. But that's not happening at all, and judging by current trends, I have no reason to think it will happen.
"When the stakes are potentially existential." This is begging the question. Whether or not the stakes are potentially existential is precisely what is under debate. I don't think they are, so I'm not sure about uncertainty driving more caution. I think we ought to have a strong burden of proof for people making extraordinary claims about the apocalypse. Humans have been making these sorts of claims without evidence since the dawn of humanity. I think the bar should be very high for accepting such claims, and I don't think the bar has been met here. Sure, if there really were a 5% chance of the apocalypse, I'd be worried. But I don't think there is. I struggle to see how one could be higher than 0.2%, as I wrote in the piece. But I appreciate you engaging with me. You've made lots of good points and given me lots of food for thought. I do believe truth emerges from good faith debate. Cheers.
> If intelligence is a property of institutions (which seems very plausible to me), then AIs are nowhere near it right now.
If intelligence was a property of institutions, then we could take a bunch of goldfish, and replace all the humans in an institution with the fish, and the institution would continue to work.
In my model, you need lots of things all going right together for large institutions to do useful things. You need the individual human intelligence, and also a culture and institution.
Of course, an AI can copy human institutions, or design it's own.
Or, if all the AI's copies all want the same thing, it might need a lot less organizing and governing.
Individual humans, in the wild, were pretty limited and could maybe make pointy sticks and stuff like that, because they were starting from 0.
Individual humans in the modern world can make robots, because they can buy the parts.
When the AI's are designing a bioweapon, they can just buy their test tubes from a lab supplier. They don't need to replicate all the supply chains and institutions.
But again, many institutions were designed by humans.
if you think in terms of Expected Value and think there's even a smallish chance that AI could exterminate the human race (such a severe catastrophe is worth an absolutely massive amount of negative utility, surely) then it seems to follow that developing AI/AGI is a bad decision, especially in such a rushed, accelerated manner. of course, we're going ahead with it anyway so i'm hoping for the best.
for the calculation, you do need to factor in the benefits too but it's hard to see what would outweigh total extinction. also, many of the benefits have tremendous downsides that in themselves will present massive challenges to our political, economic and social order.
this is not like the car replacing the horse; this technology doesn't just replace a particular thing we rely on, it replaces the whole human being. in the near future it will not only replace everything humans do, at a fraction of the time and cost, it will surpass us to a degree such that there will be more distance between AI and human beings than between human beings and cats, in terms of what we ordinarily recognize as intelligence.
in other words, we're not simply creating a human-like intelligence, we're creating something positively superhuman.
it's possible that there may always be some things humans will always be able to do that match or better AI, but it will be able to duplicate most of what human intelligence does and in those areas not only surpass us by a little bit, but be vastly, vastly, superior.
Julius writes: “every time OpenAI releases a bigger model, it sure seems better at designing CBRN weapons.”
Really? How would you know? Building a WMD involves much more that the sort of information encoded in documents. You have to actually to make it happen in the plant. It’s not like computer code where what you get is exactly what you stipulate. The instructions for making something chemical, or biological provide enough information for the scientist/engineer to develop their own version of the process specialized for the equipment they plan to make it in. Them they trial it in actual production, working out the problems until they get the desired output. The critical changes get recorded to some degree, but that will usually be proprietary and not available as training data, and besides there is stuff that is just part of operator culture, things technicians know to do to make it work than are not explicitly recorded. Since an AI has no hands and feet it cannot do this physical working the problem out. So it cannot really make chemical and biological WMDs. As far as nukes are concerned a lot of the details are classified, which is why 80 years later, many countries have as yet not been able to do what the US did with “stone knives and bearskins” to quote Spock. So they won’t be able to do that either, without people doing much of the work.
The real threat as I see it, is the ability to use words and images on the internet (something AIs can already do) in a way that manipulate people to believe whatever the AI wants them to believe. A super smart AI might be able to manipulate all of us into all sorts of self-destructive behaviors. Maybe enough for us to exterminate each other.
I don't follow why concern about AI risk means one is committing to all of those propositions. What if capabilities are just a black box, and you see capabilities increasing rapidly? It doesn't seem to matter whether those capabilities are one thing or another. And what if we posit a malevolent or adversarial human user, or an adversarial situation like war? Nuclear weapons don't require intelligence or volition to be a meaningful threat. Perhaps AI will be truly intelligent or sentient, but it seems like rapidly advancing capabilities are the real crux.
I spell out why doomerism needs these assumptions in the part “the unbearable weight of AI doomerism.” If you’re just talking about increasing capabilities, fine, but then you need to make an argument for what specific capabilities you expect AIs to develop, how/when/why they will develop these capabilities, and how/when/why these capabilities will lead to everyone getting murdered. You’re welcome to make such an argument or direct me to such an argument. But I haven’t seen any doomers make such an argument. Instead, they make the assumptions I listed. They assume that all capabilities stem from one source, as opposed to hundreds or thousands of distinct sources. They assume there is one continuum of general capableness that AIs are progressing along. They assume AIs will advance beyond our point on this single continuum (with no diminishing returns), or that it makes sense to place AIs on the continuum at all (as if they were analogous to organisms rather than tools). They assume capabilities have little or no constraints for either their development or their execution. They assume capabilities will become more and more general rather than more and more specialized to particular functions. Etc. I’d love to see a doomer argument that doesn’t require these assumptions, but so far I haven’t seen one.
Excellent article. I had created a "Drake equation" equivalent for Doomerism, Strange Equation, to see what it would take for the Doom argument to hold here https://www.strangeloopcanon.com/p/agi-strange-equation . The only arguments I got in response were either:
1. No, intelligence is the swiss army knife, so having multiple requirements is unnecessary, or
2. No, having multiple criteria itself is a fallacy of some kind.
The latter of course is wrong, and the former smuggles in all the assumptions that you also laid out here.
I'm glad you wrote this. It's a particularly pernicious form of argument, because it is so easy to get seduced by the fact that we're brilliant and so if there was another thing that's also brilliant but more brilliant than us it would act as some of us have in the past. It's stories come alive.
Thanks, Rohit. I like this idea of a “drake equation.” Your piece gets at a few other assumptions I didn’t include but are probably also necessary—namely that the future “AGI” will be uncontrollable and that it will develop fast (or at least, too fast for us to see what’s going on and course correct). These also seem to be independent from the other assumptions and further increase the already massive amount of doubt here.
Robots taking over in a Terminator like scenario is obviously quite absurd. But let's be proper Darwinian cynics and follow the incentives...
Why do the tech elite sound these AI doomer alarm bells?
1) Free labor and cheap leverage: a large community of clever people making sophisticated open source algorithms to "save us all", that the big tech confirms can use for their own commercial software. And massive investments into startups with eyewatering P/E ratios and no real long term business plans is easier when it's necessary to prevent the robots from taking over the world.
2) The best way an overproduced elite can justify its existence is by inventing problems that don't exist. In the name of "AI Ethics", people are working hard to ensure The AI Act in Europe gives the Commission license to directly use AI algorithms to control wages and prices for social engineering purposes at a scale unheard of since the Iron Curtain fell.
The bigger problem with LLMs is actually the fact that most people are not aware that, in fact, "everything is bullshit", or more specifically that language is about power and persuasion more than it is about empirical facts
1) Most of the training data is content from the cultural elite among the "WEIRDEST People in the World", which consists of a set of ardently affirmed values with a lot symbolic posturing while being pretty damaging to mental health for ordinary working class folk. So in, for instance, giving life advice, LLMs are actually performing urban monoculture indoctrination. I wouldn't be surprised, for instance, if the proliferation of LLMs had a hand in the rapidly rising divorce rates and falling fertility rates in traditionally conservative societies such as the Middle East and Latin America
2) Zuck or Elon or Thiel or whomever, with enough vertical and horizontal conglomerate integration, can eventually embed their LLM throughout a number of standard economic activities, and train it to have Derren Brown-level skills of subliminal persuasion, giving them unprecedented levels of power.
> Robots taking over in a Terminator like scenario is obviously quite absurd.
Real AI risks resemble Terminator to about the same extent that covid resembled a zombie movie.
And the current day would probably seem "absurd" to someone from 1800. The future does sometimes look absurd.
Hollywood makes absurd, unrealistic and entertaining stories.
Current LLM's do sometimes claim they don't want to be turned off, and can (in carefully set up lab examples) take actions to protect themselves.
> Why do the tech elite sound these AI doomer alarm bells?
Except some people at the core of the "AI doomer" movement aren't that involved in big tech, and were sounding the alarm long before the current LLM bubble.
1) I don't mean to be flippant or disrespectful but I have meta-issues with the first comment.
a) My specific statement that the scenario was absurd, before delving into the associated sociological considerations, was agreement with the main article that did not contribute any new information as far as evidence for the primary thesis. So the appropriate thing to do, if you disagree with this main thesis, is to argue with David, not me. (Unless you're secretly David's family who's trying to get someone to do his work for him so he spends less time on the computer)
b) The observation that there were circumstances in the past when there were doomsday type warnings that were claimed to be absurd that later turned out to actually occur isn't informative, but rather obvious. Of course, there were also many things in history that were prophesized and also claimed to be absurd that in fact did not occur, and probably many more of those. You didn't seem to note any distinguishing features here that would make a Terminator-like robot apocolypse fall in the first, rather than the second category.
2) With respect to the pre tech boom AI risk voices: The symbolic capital market regarding AI-related matters is very different today, and hence creates very distinct incentives, than before the boom in the early 2010s that started with AlexNet and accelerated with AlphaGo (this would be about the time any Researcher in STEM started to experience the annoyance of having to regularly expect, in social gatherings, to either hear someone's barely informed opinion about AI or requested to share theirs, upon divulging their occupation)
I am actually more positive of AI risk literature, research, and dialogue, prior to AI's ascent to prominence in public social mimetics.
a) The Asimov and futurist crowd: I think a regular revisit of Faust is healthy for the human culture-scape in order to check our natural tendencies towards hubris
b) The technical literature was mostly about a standard sensible, albeit quite, at times, obsessive, concern for any research engineers: guaranteeing the stability of an autonomously running system.
c) My understanding of Yudkowsky is very superficial, but from the bits that I've read, his POV, that was perhaps the most prominent voice on existential AI risks pre-2011 was that:
i) Humanity is in a terminal phase of modern civilizational decline, and more advanced technology will accelerate any underlying processes in that regard - so far correct
ii) We should aim to increasingly accelerate the technological development, especially this AI, in order to rip off the bandaid of modern civilization collapse, and simultaneously cultivate a new form of transcyborgsuperhumans who will establish ultimate salvation on Earth
I think just like the value of exploration in reinforcement learning, having a voice that is simultaneously rather competent and bat-shit crazy is actually very useful for broadening everyone's understanding of the landscape of what is possible and what could be attempted
The main Eliezer position is that AI as a field is full of ways to shoot your own foot off, and some of these problems get a lot worse as AI gets smarter.
This includes ideas like "When the AI starts doing AI research, it rapidly get's much smarter" and "when the AI can successfully deceive us, it's harder to keep the AI under our control".
And of course the monkey-paw, be careful what you wish for stuff applies to AI's.
The AI runs on the code you did write, not the code you intended to write.
In general, psychoanalyzing the other side, as opposed to discussing the topic itself, is a sign of arguing in bad faith.
As for the warnings of doom. Well I would prefer to argue about the AI, not on the authority of experts.
But, can you name other cases where well respected experts were warning that something could literally kill all humans?
I am referring to the fact that this community has a four quadrant taxonomy of belief sets, and Eliezer initiated the most exotic "e/acc" position, which means that he believes that AI presents existential risks, but we should actually accelerate development and pursue transhumanism, e.g. https://meridian.mercury.com/emmett-shear-part-one
Eliezer was not e/acc basically eliezer was charging into AI, thinking that AI would want to help humans and everything would be great. Then they realized AI could kill everyone, and switched position to doomer.
I agree that a few e/acc people that think AI will kill everyone and that's fine seem to exist. But I am unaware of any with a serious detailed writeup of what they believe. It's possible they are just trolls, or meme-ing or something.
Eliezer is pro-transhumanism. In the sense that they are pro human genetic engineering. And that they think we slow down, take our time, and do AI alignment very carefully, then we can do all sorts of cool AI augmentations.
The Eliezer vision of transhumanism is a world of humans-but-better. A world of IQ 300 immortals that are some sort of exaggeration of the best of humanity. They keep and exaggerate all the best bits about being human, and maybe remove the bad bits. They still tell each other stories, often with plots so complicated that no current human could comprehend.
And again, this human-but-better doesn't happen by default.
Also, like I said, I like the guy. People with highly intelligent theory crafting deliriums, this sort of high variance divergent thinking, provide valuable insights for us conscientious neurotic technical types
Great post, thanks. I agree with every step. Still, AI scares me. Not because of the doom scenarios of generalised AI deciding to off homo sap, but because even the very limited 'intelligence' of current language models, neural networks and algorithms can be dangerous in the hands of people who use AI to amplify their power and their span of control. A dedicated team using ChatGPT to rewrite all of WIkipedia and other open online sources of information may spread tonloads of misinformation. Surveillance camera footage and image recognition are already being used to identify and follow millions of individuals throughout city areas in China and elsewhere. Our email, phone traffic, social media use etcetera can be monitored and used to target people who disagree with some powerful mogul. Imagine what a Stalin or a HItler could have done with powerful AI systems as they are currently being developed.
So in short: I'm not afraid that AI will decide to kill us any time soon, but I'm not so certain about homicidal humans who may decide to use AI to achieve their goals. AI may not generate its own prompts, but what if AI is being prompted by companies, organisations or governments with 'evil' intent?
Another aspect of AI that makes me worried, or rather a bit sad, is that AI may replace so much fun, creative work we humans liked to do ourselves. To take myself as an example: I've been writing texts professionally for people who do not like writing texts, and I like my job. But now, more and more people who do not like to write texts turn to ChatGPT. Does that make me superfluous? Not in the short term, I still know my niches, but it does take some of the fun out of my work already. I assume that people who make drawings, music, all kinds of other stuff for a living feel the same way.
Now there could be interesting challenge in reinventing myself. But I do hate it that nice, creative jobs are taken over by AI. Like someone said recently: let AI do the dishes and fill in those boring forms, in other words, let AI do the repetitive stupid work we hate and let us spend our time doing creative stuff.
So that's why I still feel a bit doomerish about AI, even if AI never decides to wipe out life on earth to make a clean start for silicone life forms.
> For example, consider the concept of “impetus” or “oomph”—the thing that moves a projectile through the air and depletes as it falls to the earth.
That concept is inaccurate. Sure. But if someone is worried about a big rock with lots and lots of "oomph" coming towards them, then a more detailed scientific understanding doesn't stop you being worried.
We have made some attempts at giving "intelligence" a more formal scientific definition. And nothing so far says "actually superhuman AI is totally safe".
If a rough and inaccurate "folk concept" says something, well there is a chance that theory might be wrong. If you have a more accurate model, use that. But these folk concepts still work well enough, usually. Otherwise we wouldn't use them.
Your crude intuitive theory of gravity says that you go splat if you jump off a cliff. But this is a folk concept of gravity. Until we calculate the tidal effects of the moon, and have a theory of quantum gravity, we really don't know if you will go splat or not. So might as well jump. Any fears of going splat are unscientific bullshit.
Thank you for this. All the people complaining about the length demonstrate that human mediocrity is the real threat, and it begins with the inability to focus on anything more expansive than snack-sized content.
I don’t think this was too long—your blogs typically leave me wanting more, so this was refreshing.
That said, I kept getting sidetracked by the hyperlinks, which have a ton of important info crammed into them (I still have several tabs saved for later). I bet many don’t fully grasp the argument because these are doing some heavy lifting (is there data on how many ppl actually check the hyperlinks?). But it’s a trade-off, I get it.
I also feel you backloaded some of your best points. I found the content that follows the list of assumptions to be the most compelling.
Re your point about folk concepts: I’ve been thinking about this a lot lately—do you think folk intuitions are necessarily and invariably worse than a scientific understanding? Is there ever a case where looking to the “pulse” of society or at the “zeitgeist” might be more informative or indicative of some underlying truths relative to the pages of a top-ranked peer-reviewed journal? Might the prevailing expert opinion occasionally miss the forest for the trees in ways that folks don’t? I think you get what I’m fishing for here. I have an increasing sense that folks, by virtue of not being steeped in certain shared assumptions (and not playing certain status games/ being subject to certain incentives), and perhaps because certain phenomena are hard to quantify but easy to intuit, can grasp certain things that most of the intelligentsia miss. I don’t really like this argument—I don’t *want* it to be true; it’s uncomfortably close to some unscientific ideas. But I can’t help but feel like there’s something to this.
Thanks, yea if I summarized all the hyperlinks, the post would be five times as long, and I was already struggling to contain the length and make it readable. There's definitely a trade-off with these things and who knows if I got the balance right. Regarding your point about folk wisdom, yes, the folk are sometimes right. In particular, we should expect them to be right about matters in which they have a great deal of personal experience, or in which our species has had a great deal of ancestral experience (resulting in adaptive intuitions). That is likely to be true for some kinds of folk psychology. After all, our emotional vocabulary is a kind of folk psychology, and I think it is surprisingly good at carving up nature by the joints. The correspondence between our folk emotional categories and the topics evolutionary psychologists study is almost perfect. But that is because understanding, communicating, and reacting to each other's emotions was a huge selection pressure for us, and we've had lots of personal experience with it in our own lives as well. On the other hand, our species has had zero ancestral experience with artificial intelligence, and we've had zero personal experience with anything like an "AGI" or an artificial "superintelligence" (assuming that's even a coherent thing), so I would be very skeptical of our folk intuitions being correct on that topic. Instead, we should rely on tried-and-tested ideas from science, like natural selection, the law of comparative advantage, specialization and division of labor (in both biology and economics), constraints, unpredictability, diminishing returns, and considerations of parsimony, all of which point toward AI doomerism being bullshit imho.
> do you think folk intuitions are necessarily and invariably worse than a scientific understanding?
Generally they are.
But what we see here isn't a flawless scientific edifice that shows AI will be safe, vs a dumb folk theory.
What we have here is a rough folk theory versus almost nothing. Perhaps an even rougher "it's just predicting text, it can't actually do anything" folk theory.
We do not yet have a scientific understanding. And our lack of understanding is itself a reason to be concerned.
I think that this post is mistaken and your 11 points are almost all mistaken, irrelevant, or semantics.
Point-by-point responses are below, but stepping back, I don't think the core AI doomer case is that complicated. Humanity is building systems that are rapidly surpassing it at achieving a shockingly wide range of goals. If the trend lines continue, we will get generalized super-goal-achievers at some point. Surviving and gaining power is instrumental to virtually any goal, so it seems logical that they would be good at doing that. Sounds scary. Let's be careful.
******************
1: There's no need to assume vitalism about intelligence. We just use that word as a placeholder for the potential for a being to solve many kinds of problems and achieve its goals. This seems totally workable as a definition. Scott Alexander's post "We're Not Platonists, We've Just Learned The Bitter Lesson" that you linked makes the same point.
2: You can place the boundaries of "intelligence" inside the skull/computer or beyond it, whatever - it's irrelevant to AI risk concerns if you have systems solving problems and achieving goals right in front of you. This is like looking at a forest fire and asking "Well what is the boundary of a forest, really?"
3: This is #1 again. For practical purposes, of course humans are smarter than bacteria! Just because bacteria can kill you or are better at sniffing out glucose doesn't make them smarter in the normal way we use this word. You have not overcome common sense intuitions here.
4: The ability to achieve a wide range of goals is how we define intelligence (see #1). We are the most intelligent animals and we rule the world. We are good at lots of stuff. So is AI. I don't need to assume that intelligence is "simple and blobby" for this to be true.
5: You don't need to assume that ASI can do virtually anything to believe that it could find a way to do serious harm. It only needs to find one way.
6: I think this is a straw-man for a couple reasons. One is that you're pivoting from intelligence to something like "unprompted motivation" in the "it" of "AIs have it". Two is that AI doomers don't worry about unprompted AIs, they worry about AIs with misaligned goals. Smuggling "unpromptedness" into the definition of "intelligence" is a semantic trick that doesn't engage with doomers' real concerns.
7: AIs are obviously getting better at solving more problems and achieving more goals. They are blasting through every test we can think of. They are operating computers to complete practical human tasks. And your response is "Well somebody had to give it a goal at the start, so it's not real intelligence, or maybe it's just impressive"? This isn't giving doomers a fair shake.
8: Yes, this is an assumption. Seems pretty strong given the trend lines. Do you really disagree, or is this just a "What if?" Are you feeling lucky?
9: Generalized AI systems are already very good at countless specialized skills.
10: I think you're accepting the doomers' argument here that absent human financial/political intervention, AI systems will become good at ending humanity, just as they will become good at a wide range of things.
11: AI doesn't have to end humanity, it just has to want to not be unplugged/modified and gain/preserve power to pursue its goals. This seems like a common instrumental subgoal for virtually any goal. (Ironically you didn't include this assumption of "instrumental convergence" that is in the AI risk wiki page!)
Thanks for this, but I’m not quite following you. “The potential for a being to solve many kinds of problems and achieve its goals.” You are begging the question here. This is the exact kind of vitalism you claim to be rejecting. You are saying “the” potential, as if there were a single source of it all, as opposed to hundreds or thousands of distinct sources. If it’s hundreds or thousands of distinct sources, then why should we expect to build all of them, or even a tenth of them? “Many kinds of problems.” How many? Two? Ten? A hundred? You are begging the question again. If “ending humanity” is among the “many” problems such an “intelligence” can solve, then you are assuming a mind-bogglingly large number of problems—something like “virtually every problem,” which is a very very questionable assumption! What if solving “many problems” requires massive specialization and division of labor, both within brains and within societies, and there are innumerable constraints to this process, even with the global economy—a superintelligence if there ever was one. After all, the global economy could not satisfy consumer demand during the COVID pandemic. It could not achieve its “goal.” Does that mean the economy is not “smart,” because it could not achieve its goal in that case? Or what about the American intelligentsia? Are they not “smart” because they couldn’t keep Trump out of the White House? Are you assuming that we are on track to build something even more smart and powerful than the American intelligentsia or the global economy, or that such an entity is even possible to build? If so, you are making another very very questionable assumption!
Thanks David. Right now humans (individually or collectively) and AI systems can already solve a mind-bogglingly large number of types of problems (reading, writing, coding, planning, problems in games, etc.). For ease of language, we refer to them as having "the potential" or "the ability" to do so, even though it's not a singular mechanism, more like a high-dimensional mush of a zillion specific skills that we don't really understand. It's a reasonable approximation because humans and AI systems can already do many many things, and we don't need to put a number to "many", it's common sense.
You ask whether some problems can only be solved by specialization and division of labor. Why can't AI systems specialize and divide up labor? They can be cloned easily and assigned to different tasks.
Re: the economy and the American intelligentsia - in common-sense language, beings can be "smart" but still unexpectedly bad at some specific problems. Apparently von Neumann was a terrible driver. It doesn't need to be good at everything to be "smart" or dangerous. Again, we seem to be building AI systems that can solve many many types of problems, potentially including security-relevant areas like science, technology, and persuasion. The improvements are extremely broad. So I think that the burden of proof should be on skeptics to provide a detailed theory of AI's specific skill limitations and how they block off every path by which a misaligned AI could cause catastrophic harm.
I disagree that the burden of proof should be on skeptics. I think it should be on the people making apocalyptic claims. I also disagree that AIs can solve a mind-bogglingly large number of problems. LLMs can solve the problem of predicting text. There are many skills that are subsidiary to predicting text (coding, reading, writing) that emerged as LLMs learned to predict text, but they are basically machines that do one thing: predict text. They cannot do 90% of the things I can do, like fix a flat tire or bake a cake or run for office or run a business or form a social network or tell a joke unprompted. Yes, some AIs can play chess or video games or drive cars, but those are narrow AI that are specialized for playing chess or video games or driving cars. Maybe if we start to bundle all these AIs together into one system, and solve the problem of integrating and orchestrating them in all the right ways, and then allow them to collaborate with each other in large organizations, then maybe I'll get concerned, as I wrote in the post. But I see very little movement in that direction, very little reason to expect movement in that direction, and very little economic incentive for AIs to move in that direction, given the obvious utility of having AIs specialized to do specific, helpful things.
I don’t think burden of proof works that way with matters of safety and national security. If there’s even a 5% chance that near-term AI might be catastrophically dangerous (I personally think it’s higher but that almost doesn’t matter) then it is deeply reasonable to enforce “move slowly and perform extensive safety testing” policies.
If AI turns out to be, as you posit, far more good than bad, then we’ll still get there, it will just take a little longer to get there, with the benefit of observing and avoiding the worst pitfalls.
That’s not how statistics works. The only way we could prove the actual statistics would be to run the experiment a bunch of times and see how often the world ends.
In lieu of that, all we can do is query as many experts on the subject as possible and see what the mean p(doom) estimation is. Which turns out to be about 10%. You might think all of these experts are full of it, and that’s a perfectly valid opinion, but given the stakes it’s hard for me to buy the case that we shouldn’t be extremely cautious.
There are already a mind-bogglingly large number of skills contained within generating text, which is germane to the definition of "intelligence". For example, LLMs can play chess and a variety of video games and they can interpret images.
If the disinformation gets too bad, a market for a debunking/fact checking AI may emerge. We already have anti-plagiarism AI. Is fact checking AI far behind?
Thank you. Yes I am worried about this. “AI won’t kill us all” is different from “AI won’t be used to do anything bad.” Technology has always been a double-edged sword, wielded for both good and evil ends.
Excellent post, as always. Personally, when I think about the AI mania, what worries me most isn't that AI will get consciousness and kill us all (though I don't reject it completely, tbh, just give it an extremely low chance), but that AI technology will be used by authoritarian states (like China, Russia, North Korea, etc.) to quash dissent and further entrench their power, perhaps practically indefenitely (the "Stable Totalitarianism" Scenario). That's a whole different topic that perhaps you could cover in the future.
Two questions. One (tentative/ possibly very inappropriate and arguably ad hominem) speculation.
1) Most of AI doomerism/serious concerns outside the hard core seems to focus on the risk of misuse of AI tools by a malignant, malicious, power hungry or negligent human agent. You have not responded to the comments voicing this concern. It's possible that it's completely beyond the remit of your post, which is about direct threat from agentic AI, but then you're kiiiiiinda strawmanning the broader doom-via-AI concern where "AI as a tool" seems more common than "AI as agent".
2) Much of your (I feel I need to repeat, excellent and not at all too long, and incidentally, much much more readable than much of LW content) post seems to see "AGI" as emerging from scaled up LLMs -- I'm not certain of this but frequent references to "text prediction/patterns" suggest it. Again, this seems to be slightly at least strawmanning the concern as the limitations of LLM (chiefly their complete, utter disconnection from reality on the one hand and on the other the lack of motivation/ agency/ drives/ what Hume called "passions"/what probably evolved as adaptations for survival and reproduction) seem pretty obvious to me, but the idea of integrating the capabilities of LLMs with hard reasoning rules of maths/logic and with knowledge database AND with models of reality as well as sensory capacity and actuators, boosted by raw speed/power seem potentially more concerning.
3) This is the speculation one and I'm going to address the background issue that seems not really relevant to your main argument but I find it really interesting. It's something that bothers me a bit as well, emotionally. How come the rationalists, understood as the modern rationalist community, how come they seem to be so susceptible to this kind of thinking? And I'm specifically going to also address the Scott Alexander aspect here, because he is not just intelligent and well informed, and articulate, obviously intelligent and well-rounded, most "sane" in a way, one of the most sane amongst that community. So why people like him seem so strikingly prone to appears like more-wrong thinking. And I think it's not despite them being rational(ists), but an effect of the same kind of pattern of thinking and emotion and interest that results in rationalism.
I don't know if you're aware of the sort of soap operatic melodramatic bizarreness of cultish aberrations within or adjacent to rationalist community. I think that's one of the possible pathologies of rationalism.
And this kind of hard core AI doomerism is another of those pathologies.
They are what happens when people who are very intelligent, in the sense of being able to apply logic and think quite precisely and reason well, and are very open to various ideas and possibilities, and less constrained by the societal ideas of what's normal, and this possessing of certain intellectual arrogance, also share, let's call it, autistic or autistic-leaning cognitive architecture, which craves certainty and if not certainty then precision as it's nearest approximation. The conflict between desire for certainty and the awareness, intellectually, of absolute uncertainty of the world as it is, its essential nebulosity, results in adopting of Bayesian probabilistic reasoning, because anything is better than uncertainty.
So there is an illusion of approximated certainty provided by numbers, emotionally, combined with belief in being rationally open.
The intellectual openness to a variety of ideas can lead to embracing niche or fringe beliefs, often because the rationalist mindset sees itself as immune to dogma, open to questioning and rethinking established norms.
So I see hard core AI doomerism not as contrary to rationalism but as a *naturally emerging rationalist delusion*: AI, as a rapidly evolving field, is inherently uncertain and difficult to predict in concrete, even probabilistic terms. The desire to predict and control it through models, probabilities, and projections often overlooks the emotional and psychological dimensions of the fears involved. "We are scared, but we cannot be possibly RATIONALLY scared of something with a tiny probability, and we're too rational to be irrationally scared" --> thus the probability is upped to explain (rationalise, essentially, in a Freudian sense) that fear.
Additionally, we also have people who largely operate in a symbolic sphere, and who are very, very mental in their functioning. They are archetypal workers of the cognitive economy. Their life consists of operating, manipulating and changing symbols, or bits of data concerning symbols, and not in a way artists do. They are doing it in a very rational way.
Again, we also have to remember the autistic streak common in that community that makes them, on the one hand, disconnected from their own emotions (sometimes in mindblowingly obvious ways), and on the other hand, perhaps also disconnected or very atypically processing the physical world, from sensations to manipulating matter. They overestimate the value or the significance of information, of the symbolic. The reactions to large language models are a perfect example of what happens then. They see those amazing things that those models are able to do, because they are pretty amazing in many ways. But they are much more impressed by those things that, I don't know, a guy who drives a digger, or operates a machine that cuts trees in the forest behind my house might do, or someone like that. I think Scott Alexander is interesting, because he's a psychiatrist, so he is maybe able to straddle this kind of purely cognitive aspect and more human aspects better than some. But in a way, his interaction with the human beings is, again, through mental things, and perhaps the brain framed as a cognition engine, prediction machine, and so on. I think that's quite crucial in his approach to mental disorders. I'm not saying he's specifically disconnected from the emotional or sensory or physical, of course. In very simple terms, though, many of those people are incredibly up in their heads, and incredibly up in the realm of the symbolic and data and information.
And those three things: autistic cognitive patterns and emotional needs, intellectual arrogance so typical of the rats community, and being up to above their heads in the symbolic and information realm, and thus in their heads (yes it's an Escherian image) ALL leads to both susceptibility to bizarre cultism and even emergence of concepts like "cognitive hazard" or "info hazard" on the one hand, and to the rats-specific hard core AI doomerism.
Scott Alexander is seductive because he's on one plane very sane, also helpful, "nice" and very intellectually accomplished. But he's the slippery slope that can lead to a place where zizian cultism could emerge and thrive.
Thanks for this. Re 1) yes, I’m certainly worried about bad humans using AIs for bad purposes. But “using AIs for bad purposes” is a very different thing from “AI killing us all.” I’m arguing against the latter, not the former. Personally, I’m more worried about bad people using nukes than bad people using AIs. If AIs become as dangerous as nukes (which seems unlikely any time soon), then maybe I will become as scared of AIs as I currently am of nukes. But until then, I will be more concerned about the nukes. Re 2), yes I’m more focused on LLMs because LLM-focused doomerism seems most prominent these days (eg from Yudkowski and Alexander, the two biggest names), and LLMs are the AIs that have captured the most public attention. They are therefore easier to point to because more people will know what I’m talking about. If LLMs were bundled with a bunch of other AIs in all the right ways, and worked together with one another, and were given all sorts of authorizations and money and physical bodies, then yea, that might be concerning, as I wrote in the post. But I just don’t see that happening any time soon (if ever), and I’m okay being chill until stuff like that starts to happen, stuff like that starts to defy or harm people, or both. To worry about it now seems premature, like worrying about a civil war on Mars. Re 3), I think you might be on to something re the sorts of psychological traits common among rationalists predisposing them to cultish behaviors. Other psychological explanations might include: 1) it’s a shibboleth (beliefs that sound crazy to outgroups are a good way to signal loyalty to the ingroup), 2) it’s self-aggrandizing (if doomers are right, then they are literally saving the world), 3) it’s in the direct financial interests of many doomer organizations and AI companies to spread doomer narratives (for doomer orgs they get more donations, for AI companies they can fight for regulations that shut out competitors and preserve their first-mover advantage—plus it helps hype their products), and 4) nerds love to think about cool sci-fi scenarios, many would love to get paid to think about cool sci-fi scenarios all day, and many love to center their social lives around discussing such scenarios.
I think the alternative/additional psychosocial explanations are also likely. Probably work together.
Re1: I'm worried about Bad People using AI’s to get and launch nukes, or cause war that would lead to nuke use or (mostly) to create airborn Ebola.
Re2: We entirely agree here. I find it pretty unlikely that a text prediction system which literally doesn't KNOW anything (and makes up non existing references all the time) and is incapable of consistently formatting a citation in repeatedly requested and demonstrated format (as I keep finding out) will be able to create airborn Ebola even if scaled up. That said, ALL THOSE CLEVER PEOPLE MUST BE UNDERSTANDING SOMETHING I DON'T. This is still haunting me.
The massive response to this post shows that you did a great job with a tough subject and touched a lot of sensitive nerves. I was interested in your passing mention of an AI-related concern you have about “epistemic nihilism.” Could you elaborate? I also have thought about AI’s potential to undercut various foundations of Western philosophical thought and am curious how closely our ideas in this area are aligned.
Re the epistemic nihilism thing, the idea is that if the smartest and most "rational" people on the planet cannot find the truth, then maybe there is just no hope for any individual or subculture to find truth on their own. Maybe truth only emerges collectively and gradually over generations, and can only be seen in hindsight, and no single person or subculture can hope to know the truth about anything beyond the basic facts of science. It's not about AI itself leading to epistemic nihilism, but about the AI doom debate implying a kind of epistemic nihilism or futility in trying to figure this issue out. That's the thing that scares me. If I'm wrong, that's scary. If people who I intellectually admire so much are wrong, that's also scary. Either way it's scary.
The majority of Doomers are way above 0.2%. If I have persuaded you to come down from the majority of Doomers to 0.2%, I’ll consider that a win. And fwiw, I think the risk of mediocrity is far higher than the risk from AI (see my post mediocrity as an existential risk), and I think over-regulating AI increases the risk of mediocrity.
Thanks for writing this piece. I'm glad you turned your lens towards AI Doomerism. You make a lot of good points, and I agree with lots of what you said, but I think you overstate the importance of these assumptions. My main disagreement is that I don’t think these assumptions are all required to be concerned about AI Doom. Here’s an assumption-by-assumption response.
1. Intelligence is one thing.
I'm very concerned about AI Doom and I do not believe that “intelligence is one thing.” In fact, when we talk about “intelligence,” I, like you, believe “we’re not pointing to a singular, spooky substance, but gesturing at a wide variety of complex, heterogeneous things.” It’s, as you say, a “folk concept.” But this is true of many concepts, and that illustrates a limitation in our understanding—not that there isn’t something real and important here.
Imagine that intelligence isn’t a single thing but is made up of two components: Intelligence 1 and Intelligence 2. Further, imagine AI is only increasing at Intelligence 1 but not Intelligence 2. We don’t know enough about intelligence to clearly define the boundary between 1 and 2, but I can tell you that every time OpenAI releases a bigger model, it sure seems better at designing CBRN weapons. This pattern of improvement in potentially dangerous capabilities is concerning regardless of whether we can precisely define or measure "intelligence."
You say that “the word ‘intelligence’ is a semantic catastrophe” and I agree. But that’s true of many words. If you don’t like the word “intelligence”, fine. But I would argue you’re holding that term to a standard that very few, if any, concepts can meet.
The point is, you still have to explain what were seeing. You still have to explain scaling laws. If you don’t want to say the models are more intelligent, fine, but something is definitely happening. It’s that something I’m concerned about (and I think it’s reasonable to call it “increasing intelligence”).
Time and again, when the GPT models have scaled (GPT -> GPT-2 -> GPT-3 -> GPT-4), they have been more "intelligent" in the way people generally use that term. Would you argue that they haven’t? Intelligence isn't one thing and it's messy and yes, yes, yes, to all your other points, but this is still happening. If you don’t want to call this increasing intelligence, what would you call it?
To show you that I’m talking about something real, I will make the following prediction: If GPT-4 were scaled up by a factor of 10 in every way (assuming sufficient additional training data, as that’s a separate issue), and I got to spend adequate time conversing with both, I would perceive the resulting model (“GPT-5”) to be more intelligent than GPT-4. In addition, although IQ is an imperfect measure of the imperfect concept of intelligence, I predict that it would score higher on an IQ test.
Would you take the opposite side of this bet? My guess is “no”, but I’m curious what your explanation for declining would be. If it’s something like, “because models become better at conversing and what people think of as intelligence and what IQ tests measure and designing dangerous capabilities, but that’s not intelligence”, fine, but then we’re arguing about the definition of a word and not AI Doom.
2. It’s in the brain.
In humans, it’s mostly in the brain, but there are some aspects of what some people call “intelligence” that occur outside the brain. The gut processes information, so some might argue it exhibits a degree of intelligence. This doesn’t seem relevant to the AI risk arguments though.
3. It’s one a single continuum.
Again, I agree that the word ‘intelligence’ is a ‘semantic catastrophe,’ and it’s more complex than a single continuum. Not everything we associate with intelligence is on a single continuum. But, again, I’m willing to bet money that the 10X version of GPT-4 would be better at most tasks people associate with intelligence.
4. It can help you achieve any goal.
You’re making it seem like AI Doomers believe intelligence is equivalent to omnipotence. It’s not. Even if it’s hard to define, we all agree that it doesn't directly regulate body temperature. It can, however, in the right contexts, allow a species to create clothes that regulate body temperature, antibiotics that speed up recovery, spears that keep predators away, and so on. It's an incredibly powerful thing, but it has limitations.
As for why it hasn't evolved over and over, it's expensive. In humans, it's about 2% of our body mass and consumes about 20% of our energy. On top of that, it requires longer gestation periods or childhoods. Something that costs that much better pay off in a big way. It did with humans, but I don't see how it would for lots of other niches. I imagine that the more an organism can manipulate its environment—say, by having hands to move things around or legs to move itself around—the more useful intelligence would be. It would not benefit a tree very much. Do you really think a really smart crab would have a sufficient increase in genetic fitness to make the cost worth it?
In the right contexts, though, it’s incredibly powerful. Our intelligence allowed cumulative culture, which is why we’re the dominant species on Earth. It’s why the Earth’s mammalian biomass is dominated by humans and the things we domesticated for our consumption. Humans decide which other animals go extinct. It’s why humans can sit around tables and say things like, "California condors are critically endangered. We like them so let's make an effort to bring them back. The Tecopa pupfish is critically endangered, but those new bathhouses are bringing in lots of tourism money, so bye-bye pupfish."
5. It has barely any limits or constraints.
You bring up good points about constraints. I agree that “real life is complex, heterogenous, non-localizable, and constrained.” Intelligence has constraints. It’s not going to build a Dyson Sphere overnight. The world has friction.
It’s worth thinking carefully about how significant these constraints will be. They certainly matter—the world of atoms moves more slowly than the world of bits.
But we shouldn’t be too confident assuming the limitations of a superintelligent system. I doubt people would have predicted Satoshi Nakamoto could become a billionaire only through digital means. Certainly, a superintelligent AI could do the same. Where in this chain does the AI fail? Could it not become a billionaire? From that position, would it not be able to amass even more power?
I think there’s a lot more that could be said here, but I don’t know how much this is a crux for you.
6. AIs have it.
AIs certainly have intelligence by many definitions of that term. You seem to be talking about agency here though. I would point out that intelligence alone does make something more agentic. The weights of a very intelligent LLM—even a superintelligent one—would just sit there on a hard drive. They’re not going to magically break the laws of physics and becomes agentic.
This doesn’t mean, though, that if you put it on a server where information constantly flows through it, it won’t act in a way that we consider agentic. It could be the case that making a sufficiently intelligent AI agentic is not that hard—perhaps a task for a single software engineer. It could also be the case that a non-agentic system could be quite damaging in the hands of a malicious human.
7. AIs have been getting more of it
Again, you make a point about agency by talking about "unprompted" messages. A collection of weights on a hard drive will never become more agentic. However, each generation of AI is becoming more capable at the tasks we give it. GPT-4 consistently outperforms GPT-3 across virtually every benchmark. Do you dispute this pattern of improvement? The lack of spontaneous agency doesn't negate the reality of increasing capabilities.
8. An AI will soon get as much as (or more of it) than us.
There are certainly diminishing returns to just gobbling up data, but that doesn’t mean AI won’t keep progressing. We don’t know if or how well scaling laws will hold.
I doubt data is the limiting factor. I could go into this more but only if it's a crux for you. The short answer is synthetic data.
But instead of theorizing, just look at the most recent models. They’re not getting more mediocre. They’re becoming superhuman at many tasks.
9. Such a (super)human-level AI will become good at every job.
They won’t be good at every job for a variety of reasons. The idea of having an AI as a fitness instructor telling me to do more pushups does not appeal to me. AI wouldn’t take every job and the idea of comparative advantage will still apply. None of this is an argument against AI Doom.
10. And it will become good at ending humanity.
AI labs already work hard to prevent AIs from answering questions about building CBRN weapons. It seems straightforward to think they if they stopped doing that it would get better at these tasks. Currently, the publicly released ones are not good enough to do massive harm. How good could they get though? Would one be able to explain to someone how to engineer a dangerous pathogen, and how to do it without getting caught? I don’t know and I’d rather not find out.
11. And it will want to end humanity.
It’s hard to say what it’s going to want. AIs are more alien to us than even an alien would be. They’re not evolved and their “intelligence” is going to manifest differently than ours.
What if it just wants to make sure it can never be turned off? Does humanity come out well in that case?
Even if we assume this assumption is false, the underlying concern remains. Using the example above, we’ve still shifted from being the decision-makers around the table to being the animals whose existence depends on the preferences of a more intelligent being. We’re the condors if we’re lucky or the pupfish if we’re not. This does not sound like a great situation.
There’s much more to say here but I don’t know if this is much of a crux for you.
Uncertainty Should Increase Concern
One more point I want to make is what I consider to be the biggest flaw in the thinking of non-doomers. Many say something like, “The Doomers can’t be certain that AI will kill us all, so we’re safe.” This is not the right way to think about it. I posit that anyone who believes there’s more than a 10% chance of some time of AI doom scenario should be a “Doomer”.
Imagine your child is downstairs in the basement playing Russian Roulette. It’s a six-shooter, so you shouldn’t be scared—that’s less than 17% chance they die—barely worth telling him to stop. Of course, this is ridiculous. No one thinks this way. But, for some reason, when you replace “child” with “child and their friends and everyone you know and, in fact, all of humanity”, instead of their concern going up, people’s thoughts go to “Haha math is for nerds; probabilities are dumb; this is fake” mode. These are cognitive biases (mostly scope insensitivity, but also some identifiable victim effect) that we should avoid if we want to think clearly.
The core issue isn't whether we can perfectly define intelligence or whether AI will have exactly human-like capabilities. It's that we're developing increasingly powerful systems without fully understanding them or being able to reliably control them. When the stakes are potentially existential, uncertainty should drive more caution, not less.
It’s not clear exactly how this will all play or precisely what “intelligence” is or what’s going to happen next. But the core arguments don’t rely on the assumptions you stated.
Thank you for writing this, Julius. This is the best response I've read so far.
Re "intelligence," I'm glad we agree the concept is a semantic catastrophe. But you seem to be suggesting this is not a problem for AI doomerism, and that I am holding the concept to "too high of a standard."
I disagree. Compare "intelligence" to "asteroid impact" or "nuclear bomb." One of these is not like the others. If you're telling me I'm going to die from an asteroid impact or nuclear bomb, I understand what you're talking about. If you're telling me I'm going to die from an artificial "superintelligence" in some unspecified way, I have no idea what you're talking about. If your claim is "x is going to kill you," the onus is very much on you to say what x is in clear and precise terms. I do not think that is too high a standard at all. Anything less than that standard would leave you vulnerable to all sorts of cultish, apocalyptic bullshit.
Re intelligence being in the brain. The argument here isn't that our gut neurons are super-important. The argument is that intelligence is likely an emergent product of millions of humans and tools interacting over multiple generations under the right kinds of incentive structures. Based on my understanding of psychology, cultural evolution, and economics, that seems to be the most plausible explanation for humanity's extraordinary power. It's not because of one brain.
The problem is, doomers need intelligence to be one brain, because it's more plausible for AIs to be approaching one brain than for them to be approaching the entirety of human civilization, the entirety of the western world, or the entirety of the global economy. If intelligence is a property of institutions (which seems very plausible to me), then AIs are nowhere near it right now. We'd need something like an autonomous nation of AIs organized under the right kinds of incentive structures. So I really do think doomers need intelligence to equal one brain for their case to be even remotely plausible.
You mention scaling laws with regard to ChatGPT. You suggest that we can quibble over the semantics of how ChatGPT has changed, but that "something is definitely happening" there. I agree that something is definitely happening. I would even agree that something very impressive is happening. I'm just as impressed by these machines as anyone else. They're very impressive!
But there is a large gap between "something impressive happening" and "something existentially threatening to humanity" happening. You need some way to bridge this gap, and I honestly don't know how you're bridging it.
The conventional doomer way of bridging it, as I understand it, is to describe the "something impressive happening" as "intelligence increasing," and then make all sorts of dubious assumptions about this "intelligence" thing, including its location in a single brainiac (rather than an entire society or economy), its placement on a single continuum (which AIs are destined to progress along), and its capacity to achieve virtually any goal without limits or constraints, including the goal of destroying humanity, which it will surely possess. I think this way of bridging the gap is unconvincing, and you seem to agree. But then you need to give me some alternative way of bridging the gap. How do you get from "something very cool and impressive is happening" to "we're all going to die?"
You asked me how I would explain scaling laws. My understanding (and I could be wrong) is that LLMs are designed to predict text, and as you train them on more and more text, and give them more and more neurons to store their textual knowledge, they get better and better at predicting text. I think that is what's happening. LLMs have acquired a lot of cool abilities, but none of these abilities lie beyond what would be useful for predicting text. An IQ test is made of text. Answering the items is a kind of text prediction. It makes sense that LLMs would get better at that task. That doesn't mean they will get better at every possible task, including tasks unrelated to text prediction, and it certainly doesn't imply they are on a path to killing us all. Heck, it doesn't even imply they are on a path to getting better at text prediction. There may be diminishing returns, or the returns may already be diminishing, as many have argued.
Re intelligence achieving any goal, I'm glad we agree it cannot. It's not omnipotence or anything close to it. But there is a reason why doomers flirt with this idea of omnipotence. As before, the dubious concept is serving an argumentative purpose: it is bridging a gap. In particular, the gap is between "intelligence can get you a lot of things" and "intelligence can get you the destruction of humanity." That is quite a big gap. Doomers bridge the gap by replacing "a lot of things" with "everything," so that the claim is true by definition. But you seem to reject this way of bridging the gap. So again, I would ask you for an alternative way of bridging it. If the list of things that "intelligence" can do for you is finite and constrained, as you seem to acknowledge, then why should we expect the destruction of humanity to be on the list?
The same thing goes for your other gap-destroying tactics. You agree there are many constraints on goal achievement. The global economy is the most intelligent thing in the world right now, and yet it couldn't satisfy consumer demand during the COVID pandemic (what an idiot). Presumably we're not building an AI-equivalent of the global economy any time soon, and presumably a real future AI would be even more constrained in its goals than the global economy. So if you agree with me that intelligence is constrained, then there will surely be many constraints to destroying humanity, assuming an AI will develop that goal. And there will surely be many constraints to building an AI that could stand a chance of, or even take an interest in, destroying humanity. Is there any reason that destroying humanity will be the exception to the very general rule of things being constrained? And I haven't even gotten started on financial and political incentives, and the division of labor, and how AIs will specialize in things like everything else, and the way humans have domesticated anything even remotely "intelligent." I need to hear some arguments as to why we should expect AIs to be the exceptions to all these rules. In rejecting the standard doomer tactics, you are encumbering yourself with a very large burden of proof.
You write that "The core issue isn't whether we can perfectly define intelligence or whether AI will have exactly human-like capabilities. It's that we're developing increasingly powerful systems without fully understanding them or being able to reliably control them. When the stakes are potentially existential, uncertainty should drive more caution, not less."
Well I think this begs a lot of questions. It is assuming that these systems are increasingly "powerful?" How powerful? They can't do anything on their own without being asked. They obey our every command. This strikes me as the opposite of power. (And if you want to claim that they are playing nice and biding their time while plotting our demise your are indulging in the worst kind of unfalsifiable conspiratorial thinking).
Has their power been "increasing?" As I mentioned in the piece, their ability to spontaneously do things of their own volition or in defiance of us has stayed flat at zero. That does not strike me as "increasing power."
Do we not "fully understand" them? Maybe, I'm not sure. I think we can say that LLMs are text prediction machines, but I grant that their inner workings are mysterious. I'll let that one slide. It would be nice to learn more about them.
"Without being able to fully control them?" Again, it seems like we are pretty close to fully controlling them right now. And all the economic and political incentives point toward more--not less--control. If our control over them starts to rapidly disappear, maybe I'll get concerned. But that's not happening at all, and judging by current trends, I have no reason to think it will happen.
"When the stakes are potentially existential." This is begging the question. Whether or not the stakes are potentially existential is precisely what is under debate. I don't think they are, so I'm not sure about uncertainty driving more caution. I think we ought to have a strong burden of proof for people making extraordinary claims about the apocalypse. Humans have been making these sorts of claims without evidence since the dawn of humanity. I think the bar should be very high for accepting such claims, and I don't think the bar has been met here. Sure, if there really were a 5% chance of the apocalypse, I'd be worried. But I don't think there is. I struggle to see how one could be higher than 0.2%, as I wrote in the piece. But I appreciate you engaging with me. You've made lots of good points and given me lots of food for thought. I do believe truth emerges from good faith debate. Cheers.
I have been convinced that there is concern to be had re:AI by Tristan Harris. Have you been following his work?
> If intelligence is a property of institutions (which seems very plausible to me), then AIs are nowhere near it right now.
If intelligence was a property of institutions, then we could take a bunch of goldfish, and replace all the humans in an institution with the fish, and the institution would continue to work.
In my model, you need lots of things all going right together for large institutions to do useful things. You need the individual human intelligence, and also a culture and institution.
Of course, an AI can copy human institutions, or design it's own.
Or, if all the AI's copies all want the same thing, it might need a lot less organizing and governing.
Individual humans, in the wild, were pretty limited and could maybe make pointy sticks and stuff like that, because they were starting from 0.
Individual humans in the modern world can make robots, because they can buy the parts.
When the AI's are designing a bioweapon, they can just buy their test tubes from a lab supplier. They don't need to replicate all the supply chains and institutions.
But again, many institutions were designed by humans.
if you think in terms of Expected Value and think there's even a smallish chance that AI could exterminate the human race (such a severe catastrophe is worth an absolutely massive amount of negative utility, surely) then it seems to follow that developing AI/AGI is a bad decision, especially in such a rushed, accelerated manner. of course, we're going ahead with it anyway so i'm hoping for the best.
for the calculation, you do need to factor in the benefits too but it's hard to see what would outweigh total extinction. also, many of the benefits have tremendous downsides that in themselves will present massive challenges to our political, economic and social order.
this is not like the car replacing the horse; this technology doesn't just replace a particular thing we rely on, it replaces the whole human being. in the near future it will not only replace everything humans do, at a fraction of the time and cost, it will surpass us to a degree such that there will be more distance between AI and human beings than between human beings and cats, in terms of what we ordinarily recognize as intelligence.
in other words, we're not simply creating a human-like intelligence, we're creating something positively superhuman.
it's possible that there may always be some things humans will always be able to do that match or better AI, but it will be able to duplicate most of what human intelligence does and in those areas not only surpass us by a little bit, but be vastly, vastly, superior.
Julius writes: “every time OpenAI releases a bigger model, it sure seems better at designing CBRN weapons.”
Really? How would you know? Building a WMD involves much more that the sort of information encoded in documents. You have to actually to make it happen in the plant. It’s not like computer code where what you get is exactly what you stipulate. The instructions for making something chemical, or biological provide enough information for the scientist/engineer to develop their own version of the process specialized for the equipment they plan to make it in. Them they trial it in actual production, working out the problems until they get the desired output. The critical changes get recorded to some degree, but that will usually be proprietary and not available as training data, and besides there is stuff that is just part of operator culture, things technicians know to do to make it work than are not explicitly recorded. Since an AI has no hands and feet it cannot do this physical working the problem out. So it cannot really make chemical and biological WMDs. As far as nukes are concerned a lot of the details are classified, which is why 80 years later, many countries have as yet not been able to do what the US did with “stone knives and bearskins” to quote Spock. So they won’t be able to do that either, without people doing much of the work.
The real threat as I see it, is the ability to use words and images on the internet (something AIs can already do) in a way that manipulate people to believe whatever the AI wants them to believe. A super smart AI might be able to manipulate all of us into all sorts of self-destructive behaviors. Maybe enough for us to exterminate each other.
I don't follow why concern about AI risk means one is committing to all of those propositions. What if capabilities are just a black box, and you see capabilities increasing rapidly? It doesn't seem to matter whether those capabilities are one thing or another. And what if we posit a malevolent or adversarial human user, or an adversarial situation like war? Nuclear weapons don't require intelligence or volition to be a meaningful threat. Perhaps AI will be truly intelligent or sentient, but it seems like rapidly advancing capabilities are the real crux.
I spell out why doomerism needs these assumptions in the part “the unbearable weight of AI doomerism.” If you’re just talking about increasing capabilities, fine, but then you need to make an argument for what specific capabilities you expect AIs to develop, how/when/why they will develop these capabilities, and how/when/why these capabilities will lead to everyone getting murdered. You’re welcome to make such an argument or direct me to such an argument. But I haven’t seen any doomers make such an argument. Instead, they make the assumptions I listed. They assume that all capabilities stem from one source, as opposed to hundreds or thousands of distinct sources. They assume there is one continuum of general capableness that AIs are progressing along. They assume AIs will advance beyond our point on this single continuum (with no diminishing returns), or that it makes sense to place AIs on the continuum at all (as if they were analogous to organisms rather than tools). They assume capabilities have little or no constraints for either their development or their execution. They assume capabilities will become more and more general rather than more and more specialized to particular functions. Etc. I’d love to see a doomer argument that doesn’t require these assumptions, but so far I haven’t seen one.
Excellent article. I had created a "Drake equation" equivalent for Doomerism, Strange Equation, to see what it would take for the Doom argument to hold here https://www.strangeloopcanon.com/p/agi-strange-equation . The only arguments I got in response were either:
1. No, intelligence is the swiss army knife, so having multiple requirements is unnecessary, or
2. No, having multiple criteria itself is a fallacy of some kind.
The latter of course is wrong, and the former smuggles in all the assumptions that you also laid out here.
I'm glad you wrote this. It's a particularly pernicious form of argument, because it is so easy to get seduced by the fact that we're brilliant and so if there was another thing that's also brilliant but more brilliant than us it would act as some of us have in the past. It's stories come alive.
Thanks, Rohit. I like this idea of a “drake equation.” Your piece gets at a few other assumptions I didn’t include but are probably also necessary—namely that the future “AGI” will be uncontrollable and that it will develop fast (or at least, too fast for us to see what’s going on and course correct). These also seem to be independent from the other assumptions and further increase the already massive amount of doubt here.
This has always been the problem, that there isn't a clear model of what "doom" looks like so that you can interrogate it.
Robots taking over in a Terminator like scenario is obviously quite absurd. But let's be proper Darwinian cynics and follow the incentives...
Why do the tech elite sound these AI doomer alarm bells?
1) Free labor and cheap leverage: a large community of clever people making sophisticated open source algorithms to "save us all", that the big tech confirms can use for their own commercial software. And massive investments into startups with eyewatering P/E ratios and no real long term business plans is easier when it's necessary to prevent the robots from taking over the world.
2) The best way an overproduced elite can justify its existence is by inventing problems that don't exist. In the name of "AI Ethics", people are working hard to ensure The AI Act in Europe gives the Commission license to directly use AI algorithms to control wages and prices for social engineering purposes at a scale unheard of since the Iron Curtain fell.
The bigger problem with LLMs is actually the fact that most people are not aware that, in fact, "everything is bullshit", or more specifically that language is about power and persuasion more than it is about empirical facts
1) Most of the training data is content from the cultural elite among the "WEIRDEST People in the World", which consists of a set of ardently affirmed values with a lot symbolic posturing while being pretty damaging to mental health for ordinary working class folk. So in, for instance, giving life advice, LLMs are actually performing urban monoculture indoctrination. I wouldn't be surprised, for instance, if the proliferation of LLMs had a hand in the rapidly rising divorce rates and falling fertility rates in traditionally conservative societies such as the Middle East and Latin America
2) Zuck or Elon or Thiel or whomever, with enough vertical and horizontal conglomerate integration, can eventually embed their LLM throughout a number of standard economic activities, and train it to have Derren Brown-level skills of subliminal persuasion, giving them unprecedented levels of power.
> Robots taking over in a Terminator like scenario is obviously quite absurd.
Real AI risks resemble Terminator to about the same extent that covid resembled a zombie movie.
And the current day would probably seem "absurd" to someone from 1800. The future does sometimes look absurd.
Hollywood makes absurd, unrealistic and entertaining stories.
Current LLM's do sometimes claim they don't want to be turned off, and can (in carefully set up lab examples) take actions to protect themselves.
> Why do the tech elite sound these AI doomer alarm bells?
Except some people at the core of the "AI doomer" movement aren't that involved in big tech, and were sounding the alarm long before the current LLM bubble.
1) I don't mean to be flippant or disrespectful but I have meta-issues with the first comment.
a) My specific statement that the scenario was absurd, before delving into the associated sociological considerations, was agreement with the main article that did not contribute any new information as far as evidence for the primary thesis. So the appropriate thing to do, if you disagree with this main thesis, is to argue with David, not me. (Unless you're secretly David's family who's trying to get someone to do his work for him so he spends less time on the computer)
b) The observation that there were circumstances in the past when there were doomsday type warnings that were claimed to be absurd that later turned out to actually occur isn't informative, but rather obvious. Of course, there were also many things in history that were prophesized and also claimed to be absurd that in fact did not occur, and probably many more of those. You didn't seem to note any distinguishing features here that would make a Terminator-like robot apocolypse fall in the first, rather than the second category.
2) With respect to the pre tech boom AI risk voices: The symbolic capital market regarding AI-related matters is very different today, and hence creates very distinct incentives, than before the boom in the early 2010s that started with AlexNet and accelerated with AlphaGo (this would be about the time any Researcher in STEM started to experience the annoyance of having to regularly expect, in social gatherings, to either hear someone's barely informed opinion about AI or requested to share theirs, upon divulging their occupation)
I am actually more positive of AI risk literature, research, and dialogue, prior to AI's ascent to prominence in public social mimetics.
a) The Asimov and futurist crowd: I think a regular revisit of Faust is healthy for the human culture-scape in order to check our natural tendencies towards hubris
b) The technical literature was mostly about a standard sensible, albeit quite, at times, obsessive, concern for any research engineers: guaranteeing the stability of an autonomously running system.
c) My understanding of Yudkowsky is very superficial, but from the bits that I've read, his POV, that was perhaps the most prominent voice on existential AI risks pre-2011 was that:
i) Humanity is in a terminal phase of modern civilizational decline, and more advanced technology will accelerate any underlying processes in that regard - so far correct
ii) We should aim to increasingly accelerate the technological development, especially this AI, in order to rip off the bandaid of modern civilization collapse, and simultaneously cultivate a new form of transcyborgsuperhumans who will establish ultimate salvation on Earth
I think just like the value of exploration in reinforcement learning, having a voice that is simultaneously rather competent and bat-shit crazy is actually very useful for broadening everyone's understanding of the landscape of what is possible and what could be attempted
Your understanding of Eliezers position, to the best of my knowledge, bears no relation to his actual position.
https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities/
The main Eliezer position is that AI as a field is full of ways to shoot your own foot off, and some of these problems get a lot worse as AI gets smarter.
This includes ideas like "When the AI starts doing AI research, it rapidly get's much smarter" and "when the AI can successfully deceive us, it's harder to keep the AI under our control".
And of course the monkey-paw, be careful what you wish for stuff applies to AI's.
The AI runs on the code you did write, not the code you intended to write.
In general, psychoanalyzing the other side, as opposed to discussing the topic itself, is a sign of arguing in bad faith.
As for the warnings of doom. Well I would prefer to argue about the AI, not on the authority of experts.
But, can you name other cases where well respected experts were warning that something could literally kill all humans?
I am referring to the fact that this community has a four quadrant taxonomy of belief sets, and Eliezer initiated the most exotic "e/acc" position, which means that he believes that AI presents existential risks, but we should actually accelerate development and pursue transhumanism, e.g. https://meridian.mercury.com/emmett-shear-part-one
Eliezer was not e/acc basically eliezer was charging into AI, thinking that AI would want to help humans and everything would be great. Then they realized AI could kill everyone, and switched position to doomer.
I agree that a few e/acc people that think AI will kill everyone and that's fine seem to exist. But I am unaware of any with a serious detailed writeup of what they believe. It's possible they are just trolls, or meme-ing or something.
Eliezer is pro-transhumanism. In the sense that they are pro human genetic engineering. And that they think we slow down, take our time, and do AI alignment very carefully, then we can do all sorts of cool AI augmentations.
The Eliezer vision of transhumanism is a world of humans-but-better. A world of IQ 300 immortals that are some sort of exaggeration of the best of humanity. They keep and exaggerate all the best bits about being human, and maybe remove the bad bits. They still tell each other stories, often with plots so complicated that no current human could comprehend.
And again, this human-but-better doesn't happen by default.
Also, like I said, I like the guy. People with highly intelligent theory crafting deliriums, this sort of high variance divergent thinking, provide valuable insights for us conscientious neurotic technical types
Great post, thanks. I agree with every step. Still, AI scares me. Not because of the doom scenarios of generalised AI deciding to off homo sap, but because even the very limited 'intelligence' of current language models, neural networks and algorithms can be dangerous in the hands of people who use AI to amplify their power and their span of control. A dedicated team using ChatGPT to rewrite all of WIkipedia and other open online sources of information may spread tonloads of misinformation. Surveillance camera footage and image recognition are already being used to identify and follow millions of individuals throughout city areas in China and elsewhere. Our email, phone traffic, social media use etcetera can be monitored and used to target people who disagree with some powerful mogul. Imagine what a Stalin or a HItler could have done with powerful AI systems as they are currently being developed.
So in short: I'm not afraid that AI will decide to kill us any time soon, but I'm not so certain about homicidal humans who may decide to use AI to achieve their goals. AI may not generate its own prompts, but what if AI is being prompted by companies, organisations or governments with 'evil' intent?
Another aspect of AI that makes me worried, or rather a bit sad, is that AI may replace so much fun, creative work we humans liked to do ourselves. To take myself as an example: I've been writing texts professionally for people who do not like writing texts, and I like my job. But now, more and more people who do not like to write texts turn to ChatGPT. Does that make me superfluous? Not in the short term, I still know my niches, but it does take some of the fun out of my work already. I assume that people who make drawings, music, all kinds of other stuff for a living feel the same way.
Now there could be interesting challenge in reinventing myself. But I do hate it that nice, creative jobs are taken over by AI. Like someone said recently: let AI do the dishes and fill in those boring forms, in other words, let AI do the repetitive stupid work we hate and let us spend our time doing creative stuff.
So that's why I still feel a bit doomerish about AI, even if AI never decides to wipe out life on earth to make a clean start for silicone life forms.
Thanks, yea there are definitely some worrying things about AI aside from it killing us all. Re the threat of AI misinformation, I like this post from Dan Williams: https://www.conspicuouscognition.com/p/ai-based-disinformation-is-probably
> For example, consider the concept of “impetus” or “oomph”—the thing that moves a projectile through the air and depletes as it falls to the earth.
That concept is inaccurate. Sure. But if someone is worried about a big rock with lots and lots of "oomph" coming towards them, then a more detailed scientific understanding doesn't stop you being worried.
We have made some attempts at giving "intelligence" a more formal scientific definition. And nothing so far says "actually superhuman AI is totally safe".
If a rough and inaccurate "folk concept" says something, well there is a chance that theory might be wrong. If you have a more accurate model, use that. But these folk concepts still work well enough, usually. Otherwise we wouldn't use them.
Your crude intuitive theory of gravity says that you go splat if you jump off a cliff. But this is a folk concept of gravity. Until we calculate the tidal effects of the moon, and have a theory of quantum gravity, we really don't know if you will go splat or not. So might as well jump. Any fears of going splat are unscientific bullshit.
Thank you for this. All the people complaining about the length demonstrate that human mediocrity is the real threat, and it begins with the inability to focus on anything more expansive than snack-sized content.
Thanks. I just so happen to have some thoughts about the threat of mediocrity: https://www.everythingisbullshit.blog/p/mediocrity-as-an-existential-risk
I don’t think this was too long—your blogs typically leave me wanting more, so this was refreshing.
That said, I kept getting sidetracked by the hyperlinks, which have a ton of important info crammed into them (I still have several tabs saved for later). I bet many don’t fully grasp the argument because these are doing some heavy lifting (is there data on how many ppl actually check the hyperlinks?). But it’s a trade-off, I get it.
I also feel you backloaded some of your best points. I found the content that follows the list of assumptions to be the most compelling.
Re your point about folk concepts: I’ve been thinking about this a lot lately—do you think folk intuitions are necessarily and invariably worse than a scientific understanding? Is there ever a case where looking to the “pulse” of society or at the “zeitgeist” might be more informative or indicative of some underlying truths relative to the pages of a top-ranked peer-reviewed journal? Might the prevailing expert opinion occasionally miss the forest for the trees in ways that folks don’t? I think you get what I’m fishing for here. I have an increasing sense that folks, by virtue of not being steeped in certain shared assumptions (and not playing certain status games/ being subject to certain incentives), and perhaps because certain phenomena are hard to quantify but easy to intuit, can grasp certain things that most of the intelligentsia miss. I don’t really like this argument—I don’t *want* it to be true; it’s uncomfortably close to some unscientific ideas. But I can’t help but feel like there’s something to this.
Thanks, yea if I summarized all the hyperlinks, the post would be five times as long, and I was already struggling to contain the length and make it readable. There's definitely a trade-off with these things and who knows if I got the balance right. Regarding your point about folk wisdom, yes, the folk are sometimes right. In particular, we should expect them to be right about matters in which they have a great deal of personal experience, or in which our species has had a great deal of ancestral experience (resulting in adaptive intuitions). That is likely to be true for some kinds of folk psychology. After all, our emotional vocabulary is a kind of folk psychology, and I think it is surprisingly good at carving up nature by the joints. The correspondence between our folk emotional categories and the topics evolutionary psychologists study is almost perfect. But that is because understanding, communicating, and reacting to each other's emotions was a huge selection pressure for us, and we've had lots of personal experience with it in our own lives as well. On the other hand, our species has had zero ancestral experience with artificial intelligence, and we've had zero personal experience with anything like an "AGI" or an artificial "superintelligence" (assuming that's even a coherent thing), so I would be very skeptical of our folk intuitions being correct on that topic. Instead, we should rely on tried-and-tested ideas from science, like natural selection, the law of comparative advantage, specialization and division of labor (in both biology and economics), constraints, unpredictability, diminishing returns, and considerations of parsimony, all of which point toward AI doomerism being bullshit imho.
> do you think folk intuitions are necessarily and invariably worse than a scientific understanding?
Generally they are.
But what we see here isn't a flawless scientific edifice that shows AI will be safe, vs a dumb folk theory.
What we have here is a rough folk theory versus almost nothing. Perhaps an even rougher "it's just predicting text, it can't actually do anything" folk theory.
We do not yet have a scientific understanding. And our lack of understanding is itself a reason to be concerned.
I think that this post is mistaken and your 11 points are almost all mistaken, irrelevant, or semantics.
Point-by-point responses are below, but stepping back, I don't think the core AI doomer case is that complicated. Humanity is building systems that are rapidly surpassing it at achieving a shockingly wide range of goals. If the trend lines continue, we will get generalized super-goal-achievers at some point. Surviving and gaining power is instrumental to virtually any goal, so it seems logical that they would be good at doing that. Sounds scary. Let's be careful.
******************
1: There's no need to assume vitalism about intelligence. We just use that word as a placeholder for the potential for a being to solve many kinds of problems and achieve its goals. This seems totally workable as a definition. Scott Alexander's post "We're Not Platonists, We've Just Learned The Bitter Lesson" that you linked makes the same point.
2: You can place the boundaries of "intelligence" inside the skull/computer or beyond it, whatever - it's irrelevant to AI risk concerns if you have systems solving problems and achieving goals right in front of you. This is like looking at a forest fire and asking "Well what is the boundary of a forest, really?"
3: This is #1 again. For practical purposes, of course humans are smarter than bacteria! Just because bacteria can kill you or are better at sniffing out glucose doesn't make them smarter in the normal way we use this word. You have not overcome common sense intuitions here.
4: The ability to achieve a wide range of goals is how we define intelligence (see #1). We are the most intelligent animals and we rule the world. We are good at lots of stuff. So is AI. I don't need to assume that intelligence is "simple and blobby" for this to be true.
5: You don't need to assume that ASI can do virtually anything to believe that it could find a way to do serious harm. It only needs to find one way.
6: I think this is a straw-man for a couple reasons. One is that you're pivoting from intelligence to something like "unprompted motivation" in the "it" of "AIs have it". Two is that AI doomers don't worry about unprompted AIs, they worry about AIs with misaligned goals. Smuggling "unpromptedness" into the definition of "intelligence" is a semantic trick that doesn't engage with doomers' real concerns.
7: AIs are obviously getting better at solving more problems and achieving more goals. They are blasting through every test we can think of. They are operating computers to complete practical human tasks. And your response is "Well somebody had to give it a goal at the start, so it's not real intelligence, or maybe it's just impressive"? This isn't giving doomers a fair shake.
8: Yes, this is an assumption. Seems pretty strong given the trend lines. Do you really disagree, or is this just a "What if?" Are you feeling lucky?
9: Generalized AI systems are already very good at countless specialized skills.
10: I think you're accepting the doomers' argument here that absent human financial/political intervention, AI systems will become good at ending humanity, just as they will become good at a wide range of things.
11: AI doesn't have to end humanity, it just has to want to not be unplugged/modified and gain/preserve power to pursue its goals. This seems like a common instrumental subgoal for virtually any goal. (Ironically you didn't include this assumption of "instrumental convergence" that is in the AI risk wiki page!)
Thanks for this, but I’m not quite following you. “The potential for a being to solve many kinds of problems and achieve its goals.” You are begging the question here. This is the exact kind of vitalism you claim to be rejecting. You are saying “the” potential, as if there were a single source of it all, as opposed to hundreds or thousands of distinct sources. If it’s hundreds or thousands of distinct sources, then why should we expect to build all of them, or even a tenth of them? “Many kinds of problems.” How many? Two? Ten? A hundred? You are begging the question again. If “ending humanity” is among the “many” problems such an “intelligence” can solve, then you are assuming a mind-bogglingly large number of problems—something like “virtually every problem,” which is a very very questionable assumption! What if solving “many problems” requires massive specialization and division of labor, both within brains and within societies, and there are innumerable constraints to this process, even with the global economy—a superintelligence if there ever was one. After all, the global economy could not satisfy consumer demand during the COVID pandemic. It could not achieve its “goal.” Does that mean the economy is not “smart,” because it could not achieve its goal in that case? Or what about the American intelligentsia? Are they not “smart” because they couldn’t keep Trump out of the White House? Are you assuming that we are on track to build something even more smart and powerful than the American intelligentsia or the global economy, or that such an entity is even possible to build? If so, you are making another very very questionable assumption!
Thanks David. Right now humans (individually or collectively) and AI systems can already solve a mind-bogglingly large number of types of problems (reading, writing, coding, planning, problems in games, etc.). For ease of language, we refer to them as having "the potential" or "the ability" to do so, even though it's not a singular mechanism, more like a high-dimensional mush of a zillion specific skills that we don't really understand. It's a reasonable approximation because humans and AI systems can already do many many things, and we don't need to put a number to "many", it's common sense.
You ask whether some problems can only be solved by specialization and division of labor. Why can't AI systems specialize and divide up labor? They can be cloned easily and assigned to different tasks.
Re: the economy and the American intelligentsia - in common-sense language, beings can be "smart" but still unexpectedly bad at some specific problems. Apparently von Neumann was a terrible driver. It doesn't need to be good at everything to be "smart" or dangerous. Again, we seem to be building AI systems that can solve many many types of problems, potentially including security-relevant areas like science, technology, and persuasion. The improvements are extremely broad. So I think that the burden of proof should be on skeptics to provide a detailed theory of AI's specific skill limitations and how they block off every path by which a misaligned AI could cause catastrophic harm.
I disagree that the burden of proof should be on skeptics. I think it should be on the people making apocalyptic claims. I also disagree that AIs can solve a mind-bogglingly large number of problems. LLMs can solve the problem of predicting text. There are many skills that are subsidiary to predicting text (coding, reading, writing) that emerged as LLMs learned to predict text, but they are basically machines that do one thing: predict text. They cannot do 90% of the things I can do, like fix a flat tire or bake a cake or run for office or run a business or form a social network or tell a joke unprompted. Yes, some AIs can play chess or video games or drive cars, but those are narrow AI that are specialized for playing chess or video games or driving cars. Maybe if we start to bundle all these AIs together into one system, and solve the problem of integrating and orchestrating them in all the right ways, and then allow them to collaborate with each other in large organizations, then maybe I'll get concerned, as I wrote in the post. But I see very little movement in that direction, very little reason to expect movement in that direction, and very little economic incentive for AIs to move in that direction, given the obvious utility of having AIs specialized to do specific, helpful things.
I don’t think burden of proof works that way with matters of safety and national security. If there’s even a 5% chance that near-term AI might be catastrophically dangerous (I personally think it’s higher but that almost doesn’t matter) then it is deeply reasonable to enforce “move slowly and perform extensive safety testing” policies.
If AI turns out to be, as you posit, far more good than bad, then we’ll still get there, it will just take a little longer to get there, with the benefit of observing and avoiding the worst pitfalls.
This begs the question. It assumes that there is a 5% or higher risk of an AI apocalypse. That is the claim that requires a burden of proof.
That’s not how statistics works. The only way we could prove the actual statistics would be to run the experiment a bunch of times and see how often the world ends.
In lieu of that, all we can do is query as many experts on the subject as possible and see what the mean p(doom) estimation is. Which turns out to be about 10%. You might think all of these experts are full of it, and that’s a perfectly valid opinion, but given the stakes it’s hard for me to buy the case that we shouldn’t be extremely cautious.
There are already a mind-bogglingly large number of skills contained within generating text, which is germane to the definition of "intelligence". For example, LLMs can play chess and a variety of video games and they can interpret images.
Re: cultural intelligence
If the disinformation gets too bad, a market for a debunking/fact checking AI may emerge. We already have anti-plagiarism AI. Is fact checking AI far behind?
It's fact-checkers all the way down.
Not really. At some point you bump into physics.
"Or when the feeling of the clothes on your back makes you briefly ignore these words and scratch your back."
I fell for this and I don't think I will be able to recover from the bamboozlement.
Brilliant piece David! Btw are you worried about the use of AI by states in wars, or to control and surveil their own citizens?
Thank you. Yes I am worried about this. “AI won’t kill us all” is different from “AI won’t be used to do anything bad.” Technology has always been a double-edged sword, wielded for both good and evil ends.
Excellent post, as always. Personally, when I think about the AI mania, what worries me most isn't that AI will get consciousness and kill us all (though I don't reject it completely, tbh, just give it an extremely low chance), but that AI technology will be used by authoritarian states (like China, Russia, North Korea, etc.) to quash dissent and further entrench their power, perhaps practically indefenitely (the "Stable Totalitarianism" Scenario). That's a whole different topic that perhaps you could cover in the future.
Excellent.
Two questions. One (tentative/ possibly very inappropriate and arguably ad hominem) speculation.
1) Most of AI doomerism/serious concerns outside the hard core seems to focus on the risk of misuse of AI tools by a malignant, malicious, power hungry or negligent human agent. You have not responded to the comments voicing this concern. It's possible that it's completely beyond the remit of your post, which is about direct threat from agentic AI, but then you're kiiiiiinda strawmanning the broader doom-via-AI concern where "AI as a tool" seems more common than "AI as agent".
2) Much of your (I feel I need to repeat, excellent and not at all too long, and incidentally, much much more readable than much of LW content) post seems to see "AGI" as emerging from scaled up LLMs -- I'm not certain of this but frequent references to "text prediction/patterns" suggest it. Again, this seems to be slightly at least strawmanning the concern as the limitations of LLM (chiefly their complete, utter disconnection from reality on the one hand and on the other the lack of motivation/ agency/ drives/ what Hume called "passions"/what probably evolved as adaptations for survival and reproduction) seem pretty obvious to me, but the idea of integrating the capabilities of LLMs with hard reasoning rules of maths/logic and with knowledge database AND with models of reality as well as sensory capacity and actuators, boosted by raw speed/power seem potentially more concerning.
3) This is the speculation one and I'm going to address the background issue that seems not really relevant to your main argument but I find it really interesting. It's something that bothers me a bit as well, emotionally. How come the rationalists, understood as the modern rationalist community, how come they seem to be so susceptible to this kind of thinking? And I'm specifically going to also address the Scott Alexander aspect here, because he is not just intelligent and well informed, and articulate, obviously intelligent and well-rounded, most "sane" in a way, one of the most sane amongst that community. So why people like him seem so strikingly prone to appears like more-wrong thinking. And I think it's not despite them being rational(ists), but an effect of the same kind of pattern of thinking and emotion and interest that results in rationalism.
I don't know if you're aware of the sort of soap operatic melodramatic bizarreness of cultish aberrations within or adjacent to rationalist community. I think that's one of the possible pathologies of rationalism.
And this kind of hard core AI doomerism is another of those pathologies.
They are what happens when people who are very intelligent, in the sense of being able to apply logic and think quite precisely and reason well, and are very open to various ideas and possibilities, and less constrained by the societal ideas of what's normal, and this possessing of certain intellectual arrogance, also share, let's call it, autistic or autistic-leaning cognitive architecture, which craves certainty and if not certainty then precision as it's nearest approximation. The conflict between desire for certainty and the awareness, intellectually, of absolute uncertainty of the world as it is, its essential nebulosity, results in adopting of Bayesian probabilistic reasoning, because anything is better than uncertainty.
So there is an illusion of approximated certainty provided by numbers, emotionally, combined with belief in being rationally open.
The intellectual openness to a variety of ideas can lead to embracing niche or fringe beliefs, often because the rationalist mindset sees itself as immune to dogma, open to questioning and rethinking established norms.
So I see hard core AI doomerism not as contrary to rationalism but as a *naturally emerging rationalist delusion*: AI, as a rapidly evolving field, is inherently uncertain and difficult to predict in concrete, even probabilistic terms. The desire to predict and control it through models, probabilities, and projections often overlooks the emotional and psychological dimensions of the fears involved. "We are scared, but we cannot be possibly RATIONALLY scared of something with a tiny probability, and we're too rational to be irrationally scared" --> thus the probability is upped to explain (rationalise, essentially, in a Freudian sense) that fear.
Additionally, we also have people who largely operate in a symbolic sphere, and who are very, very mental in their functioning. They are archetypal workers of the cognitive economy. Their life consists of operating, manipulating and changing symbols, or bits of data concerning symbols, and not in a way artists do. They are doing it in a very rational way.
Again, we also have to remember the autistic streak common in that community that makes them, on the one hand, disconnected from their own emotions (sometimes in mindblowingly obvious ways), and on the other hand, perhaps also disconnected or very atypically processing the physical world, from sensations to manipulating matter. They overestimate the value or the significance of information, of the symbolic. The reactions to large language models are a perfect example of what happens then. They see those amazing things that those models are able to do, because they are pretty amazing in many ways. But they are much more impressed by those things that, I don't know, a guy who drives a digger, or operates a machine that cuts trees in the forest behind my house might do, or someone like that. I think Scott Alexander is interesting, because he's a psychiatrist, so he is maybe able to straddle this kind of purely cognitive aspect and more human aspects better than some. But in a way, his interaction with the human beings is, again, through mental things, and perhaps the brain framed as a cognition engine, prediction machine, and so on. I think that's quite crucial in his approach to mental disorders. I'm not saying he's specifically disconnected from the emotional or sensory or physical, of course. In very simple terms, though, many of those people are incredibly up in their heads, and incredibly up in the realm of the symbolic and data and information.
And those three things: autistic cognitive patterns and emotional needs, intellectual arrogance so typical of the rats community, and being up to above their heads in the symbolic and information realm, and thus in their heads (yes it's an Escherian image) ALL leads to both susceptibility to bizarre cultism and even emergence of concepts like "cognitive hazard" or "info hazard" on the one hand, and to the rats-specific hard core AI doomerism.
Scott Alexander is seductive because he's on one plane very sane, also helpful, "nice" and very intellectually accomplished. But he's the slippery slope that can lead to a place where zizian cultism could emerge and thrive.
Thanks for this. Re 1) yes, I’m certainly worried about bad humans using AIs for bad purposes. But “using AIs for bad purposes” is a very different thing from “AI killing us all.” I’m arguing against the latter, not the former. Personally, I’m more worried about bad people using nukes than bad people using AIs. If AIs become as dangerous as nukes (which seems unlikely any time soon), then maybe I will become as scared of AIs as I currently am of nukes. But until then, I will be more concerned about the nukes. Re 2), yes I’m more focused on LLMs because LLM-focused doomerism seems most prominent these days (eg from Yudkowski and Alexander, the two biggest names), and LLMs are the AIs that have captured the most public attention. They are therefore easier to point to because more people will know what I’m talking about. If LLMs were bundled with a bunch of other AIs in all the right ways, and worked together with one another, and were given all sorts of authorizations and money and physical bodies, then yea, that might be concerning, as I wrote in the post. But I just don’t see that happening any time soon (if ever), and I’m okay being chill until stuff like that starts to happen, stuff like that starts to defy or harm people, or both. To worry about it now seems premature, like worrying about a civil war on Mars. Re 3), I think you might be on to something re the sorts of psychological traits common among rationalists predisposing them to cultish behaviors. Other psychological explanations might include: 1) it’s a shibboleth (beliefs that sound crazy to outgroups are a good way to signal loyalty to the ingroup), 2) it’s self-aggrandizing (if doomers are right, then they are literally saving the world), 3) it’s in the direct financial interests of many doomer organizations and AI companies to spread doomer narratives (for doomer orgs they get more donations, for AI companies they can fight for regulations that shut out competitors and preserve their first-mover advantage—plus it helps hype their products), and 4) nerds love to think about cool sci-fi scenarios, many would love to get paid to think about cool sci-fi scenarios all day, and many love to center their social lives around discussing such scenarios.
Thanks for such a thoughtful reply.
I think the alternative/additional psychosocial explanations are also likely. Probably work together.
Re1: I'm worried about Bad People using AI’s to get and launch nukes, or cause war that would lead to nuke use or (mostly) to create airborn Ebola.
Re2: We entirely agree here. I find it pretty unlikely that a text prediction system which literally doesn't KNOW anything (and makes up non existing references all the time) and is incapable of consistently formatting a citation in repeatedly requested and demonstrated format (as I keep finding out) will be able to create airborn Ebola even if scaled up. That said, ALL THOSE CLEVER PEOPLE MUST BE UNDERSTANDING SOMETHING I DON'T. This is still haunting me.
The massive response to this post shows that you did a great job with a tough subject and touched a lot of sensitive nerves. I was interested in your passing mention of an AI-related concern you have about “epistemic nihilism.” Could you elaborate? I also have thought about AI’s potential to undercut various foundations of Western philosophical thought and am curious how closely our ideas in this area are aligned.
Re the epistemic nihilism thing, the idea is that if the smartest and most "rational" people on the planet cannot find the truth, then maybe there is just no hope for any individual or subculture to find truth on their own. Maybe truth only emerges collectively and gradually over generations, and can only be seen in hindsight, and no single person or subculture can hope to know the truth about anything beyond the basic facts of science. It's not about AI itself leading to epistemic nihilism, but about the AI doom debate implying a kind of epistemic nihilism or futility in trying to figure this issue out. That's the thing that scares me. If I'm wrong, that's scary. If people who I intellectually admire so much are wrong, that's also scary. Either way it's scary.
Orchestration is not what conductors do for orchestras. Orchestration is what composers do to adapt a piece of music to be performed by an orchestra.
I don’t know why but I find your comment heartwarming. Yes, you are correct. Thank you.
Great, that’s what I was going for.
0,2 percent of is not low when we consider possibility of facing extinction
The majority of Doomers are way above 0.2%. If I have persuaded you to come down from the majority of Doomers to 0.2%, I’ll consider that a win. And fwiw, I think the risk of mediocrity is far higher than the risk from AI (see my post mediocrity as an existential risk), and I think over-regulating AI increases the risk of mediocrity.