Thanks for writing this piece. I'm glad you turned your lens towards AI Doomerism. You make a lot of good points, and I agree with lots of what you said, but I think you overstate the importance of these assumptions. My main disagreement is that I don’t think these assumptions are all required to be concerned about AI Doom. Here’s an assumption-by-assumption response.
1. Intelligence is one thing.
I'm very concerned about AI Doom and I do not believe that “intelligence is one thing.” In fact, when we talk about “intelligence,” I, like you, believe “we’re not pointing to a singular, spooky substance, but gesturing at a wide variety of complex, heterogeneous things.” It’s, as you say, a “folk concept.” But this is true of many concepts, and that illustrates a limitation in our understanding—not that there isn’t something real and important here.
Imagine that intelligence isn’t a single thing but is made up of two components: Intelligence 1 and Intelligence 2. Further, imagine AI is only increasing at Intelligence 1 but not Intelligence 2. We don’t know enough about intelligence to clearly define the boundary between 1 and 2, but I can tell you that every time OpenAI releases a bigger model, it sure seems better at designing CBRN weapons. This pattern of improvement in potentially dangerous capabilities is concerning regardless of whether we can precisely define or measure "intelligence."
You say that “the word ‘intelligence’ is a semantic catastrophe” and I agree. But that’s true of many words. If you don’t like the word “intelligence”, fine. But I would argue you’re holding that term to a standard that very few, if any, concepts can meet.
The point is, you still have to explain what were seeing. You still have to explain scaling laws. If you don’t want to say the models are more intelligent, fine, but something is definitely happening. It’s that something I’m concerned about (and I think it’s reasonable to call it “increasing intelligence”).
Time and again, when the GPT models have scaled (GPT -> GPT-2 -> GPT-3 -> GPT-4), they have been more "intelligent" in the way people generally use that term. Would you argue that they haven’t? Intelligence isn't one thing and it's messy and yes, yes, yes, to all your other points, but this is still happening. If you don’t want to call this increasing intelligence, what would you call it?
To show you that I’m talking about something real, I will make the following prediction: If GPT-4 were scaled up by a factor of 10 in every way (assuming sufficient additional training data, as that’s a separate issue), and I got to spend adequate time conversing with both, I would perceive the resulting model (“GPT-5”) to be more intelligent than GPT-4. In addition, although IQ is an imperfect measure of the imperfect concept of intelligence, I predict that it would score higher on an IQ test.
Would you take the opposite side of this bet? My guess is “no”, but I’m curious what your explanation for declining would be. If it’s something like, “because models become better at conversing and what people think of as intelligence and what IQ tests measure and designing dangerous capabilities, but that’s not intelligence”, fine, but then we’re arguing about the definition of a word and not AI Doom.
2. It’s in the brain.
In humans, it’s mostly in the brain, but there are some aspects of what some people call “intelligence” that occur outside the brain. The gut processes information, so some might argue it exhibits a degree of intelligence. This doesn’t seem relevant to the AI risk arguments though.
3. It’s one a single continuum.
Again, I agree that the word ‘intelligence’ is a ‘semantic catastrophe,’ and it’s more complex than a single continuum. Not everything we associate with intelligence is on a single continuum. But, again, I’m willing to bet money that the 10X version of GPT-4 would be better at most tasks people associate with intelligence.
4. It can help you achieve any goal.
You’re making it seem like AI Doomers believe intelligence is equivalent to omnipotence. It’s not. Even if it’s hard to define, we all agree that it doesn't directly regulate body temperature. It can, however, in the right contexts, allow a species to create clothes that regulate body temperature, antibiotics that speed up recovery, spears that keep predators away, and so on. It's an incredibly powerful thing, but it has limitations.
As for why it hasn't evolved over and over, it's expensive. In humans, it's about 2% of our body mass and consumes about 20% of our energy. On top of that, it requires longer gestation periods or childhoods. Something that costs that much better pay off in a big way. It did with humans, but I don't see how it would for lots of other niches. I imagine that the more an organism can manipulate its environment—say, by having hands to move things around or legs to move itself around—the more useful intelligence would be. It would not benefit a tree very much. Do you really think a really smart crab would have a sufficient increase in genetic fitness to make the cost worth it?
In the right contexts, though, it’s incredibly powerful. Our intelligence allowed cumulative culture, which is why we’re the dominant species on Earth. It’s why the Earth’s mammalian biomass is dominated by humans and the things we domesticated for our consumption. Humans decide which other animals go extinct. It’s why humans can sit around tables and say things like, "California condors are critically endangered. We like them so let's make an effort to bring them back. The Tecopa pupfish is critically endangered, but those new bathhouses are bringing in lots of tourism money, so bye-bye pupfish."
5. It has barely any limits or constraints.
You bring up good points about constraints. I agree that “real life is complex, heterogenous, non-localizable, and constrained.” Intelligence has constraints. It’s not going to build a Dyson Sphere overnight. The world has friction.
It’s worth thinking carefully about how significant these constraints will be. They certainly matter—the world of atoms moves more slowly than the world of bits.
But we shouldn’t be too confident assuming the limitations of a superintelligent system. I doubt people would have predicted Satoshi Nakamoto could become a billionaire only through digital means. Certainly, a superintelligent AI could do the same. Where in this chain does the AI fail? Could it not become a billionaire? From that position, would it not be able to amass even more power?
I think there’s a lot more that could be said here, but I don’t know how much this is a crux for you.
AIs certainly have intelligence by many definitions of that term. You seem to be talking about agency here though. I would point out that intelligence alone does make something more agentic. The weights of a very intelligent LLM—even a superintelligent one—would just sit there on a hard drive. They’re not going to magically break the laws of physics and becomes agentic.
This doesn’t mean, though, that if you put it on a server where information constantly flows through it, it won’t act in a way that we consider agentic. It could be the case that making a sufficiently intelligent AI agentic is not that hard—perhaps a task for a single software engineer. It could also be the case that a non-agentic system could be quite damaging in the hands of a malicious human.
7. AIs have been getting more of it
Again, you make a point about agency by talking about "unprompted" messages. A collection of weights on a hard drive will never become more agentic. However, each generation of AI is becoming more capable at the tasks we give it. GPT-4 consistently outperforms GPT-3 across virtually every benchmark. Do you dispute this pattern of improvement? The lack of spontaneous agency doesn't negate the reality of increasing capabilities.
8. An AI will soon get as much as (or more of it) than us.
There are certainly diminishing returns to just gobbling up data, but that doesn’t mean AI won’t keep progressing. We don’t know if or how well scaling laws will hold.
I doubt data is the limiting factor. I could go into this more but only if it's a crux for you. The short answer is synthetic data.
But instead of theorizing, just look at the most recent models. They’re not getting more mediocre. They’re becoming superhuman at many tasks.
9. Such a (super)human-level AI will become good at every job.
They won’t be good at every job for a variety of reasons. The idea of having an AI as a fitness instructor telling me to do more pushups does not appeal to me. AI wouldn’t take every job and the idea of comparative advantage will still apply. None of this is an argument against AI Doom.
10. And it will become good at ending humanity.
AI labs already work hard to prevent AIs from answering questions about building CBRN weapons. It seems straightforward to think they if they stopped doing that it would get better at these tasks. Currently, the publicly released ones are not good enough to do massive harm. How good could they get though? Would one be able to explain to someone how to engineer a dangerous pathogen, and how to do it without getting caught? I don’t know and I’d rather not find out.
11. And it will want to end humanity.
It’s hard to say what it’s going to want. AIs are more alien to us than even an alien would be. They’re not evolved and their “intelligence” is going to manifest differently than ours.
What if it just wants to make sure it can never be turned off? Does humanity come out well in that case?
Even if we assume this assumption is false, the underlying concern remains. Using the example above, we’ve still shifted from being the decision-makers around the table to being the animals whose existence depends on the preferences of a more intelligent being. We’re the condors if we’re lucky or the pupfish if we’re not. This does not sound like a great situation.
There’s much more to say here but I don’t know if this is much of a crux for you.
Uncertainty Should Increase Concern
One more point I want to make is what I consider to be the biggest flaw in the thinking of non-doomers. Many say something like, “The Doomers can’t be certain that AI will kill us all, so we’re safe.” This is not the right way to think about it. I posit that anyone who believes there’s more than a 10% chance of some time of AI doom scenario should be a “Doomer”.
Imagine your child is downstairs in the basement playing Russian Roulette. It’s a six-shooter, so you shouldn’t be scared—that’s less than 17% chance they die—barely worth telling him to stop. Of course, this is ridiculous. No one thinks this way. But, for some reason, when you replace “child” with “child and their friends and everyone you know and, in fact, all of humanity”, instead of their concern going up, people’s thoughts go to “Haha math is for nerds; probabilities are dumb; this is fake” mode. These are cognitive biases (mostly scope insensitivity, but also some identifiable victim effect) that we should avoid if we want to think clearly.
The core issue isn't whether we can perfectly define intelligence or whether AI will have exactly human-like capabilities. It's that we're developing increasingly powerful systems without fully understanding them or being able to reliably control them. When the stakes are potentially existential, uncertainty should drive more caution, not less.
It’s not clear exactly how this will all play or precisely what “intelligence” is or what’s going to happen next. But the core arguments don’t rely on the assumptions you stated.
Thank you for writing this, Julius. This is the best response I've read so far.
Re "intelligence," I'm glad we agree the concept is a semantic catastrophe. But you seem to be suggesting this is not a problem for AI doomerism, and that I am holding the concept to "too high of a standard."
I disagree. Compare "intelligence" to "asteroid impact" or "nuclear bomb." One of these is not like the others. If you're telling me I'm going to die from an asteroid impact or nuclear bomb, I understand what you're talking about. If you're telling me I'm going to die from an artificial "superintelligence" in some unspecified way, I have no idea what you're talking about. If your claim is "x is going to kill you," the onus is very much on you to say what x is in clear and precise terms. I do not think that is too high a standard at all. Anything less than that standard would leave you vulnerable to all sorts of cultish, apocalyptic bullshit.
Re intelligence being in the brain. The argument here isn't that our gut neurons are super-important. The argument is that intelligence is likely an emergent product of millions of humans and tools interacting over multiple generations under the right kinds of incentive structures. Based on my understanding of psychology, cultural evolution, and economics, that seems to be the most plausible explanation for humanity's extraordinary power. It's not because of one brain.
The problem is, doomers need intelligence to be one brain, because it's more plausible for AIs to be approaching one brain than for them to be approaching the entirety of human civilization, the entirety of the western world, or the entirety of the global economy. If intelligence is a property of institutions (which seems very plausible to me), then AIs are nowhere near it right now. We'd need something like an autonomous nation of AIs organized under the right kinds of incentive structures. So I really do think doomers need intelligence to equal one brain for their case to be even remotely plausible.
You mention scaling laws with regard to ChatGPT. You suggest that we can quibble over the semantics of how ChatGPT has changed, but that "something is definitely happening" there. I agree that something is definitely happening. I would even agree that something very impressive is happening. I'm just as impressed by these machines as anyone else. They're very impressive!
But there is a large gap between "something impressive happening" and "something existentially threatening to humanity" happening. You need some way to bridge this gap, and I honestly don't know how you're bridging it.
The conventional doomer way of bridging it, as I understand it, is to describe the "something impressive happening" as "intelligence increasing," and then make all sorts of dubious assumptions about this "intelligence" thing, including its location in a single brainiac (rather than an entire society or economy), its placement on a single continuum (which AIs are destined to progress along), and its capacity to achieve virtually any goal without limits or constraints, including the goal of destroying humanity, which it will surely possess. I think this way of bridging the gap is unconvincing, and you seem to agree. But then you need to give me some alternative way of bridging the gap. How do you get from "something very cool and impressive is happening" to "we're all going to die?"
You asked me how I would explain scaling laws. My understanding (and I could be wrong) is that LLMs are designed to predict text, and as you train them on more and more text, and give them more and more neurons to store their textual knowledge, they get better and better at predicting text. I think that is what's happening. LLMs have acquired a lot of cool abilities, but none of these abilities lie beyond what would be useful for predicting text. An IQ test is made of text. Answering the items is a kind of text prediction. It makes sense that LLMs would get better at that task. That doesn't mean they will get better at every possible task, including tasks unrelated to text prediction, and it certainly doesn't imply they are on a path to killing us all. Heck, it doesn't even imply they are on a path to getting better at text prediction. There may be diminishing returns, or the returns may already be diminishing, as many have argued.
Re intelligence achieving any goal, I'm glad we agree it cannot. It's not omnipotence or anything close to it. But there is a reason why doomers flirt with this idea of omnipotence. As before, the dubious concept is serving an argumentative purpose: it is bridging a gap. In particular, the gap is between "intelligence can get you a lot of things" and "intelligence can get you the destruction of humanity." That is quite a big gap. Doomers bridge the gap by replacing "a lot of things" with "everything," so that the claim is true by definition. But you seem to reject this way of bridging the gap. So again, I would ask you for an alternative way of bridging it. If the list of things that "intelligence" can do for you is finite and constrained, as you seem to acknowledge, then why should we expect the destruction of humanity to be on the list?
The same thing goes for your other gap-destroying tactics. You agree there are many constraints on goal achievement. The global economy is the most intelligent thing in the world right now, and yet it couldn't satisfy consumer demand during the COVID pandemic (what an idiot). Presumably we're not building an AI-equivalent of the global economy any time soon, and presumably a real future AI would be even more constrained in its goals than the global economy. So if you agree with me that intelligence is constrained, then there will surely be many constraints to destroying humanity, assuming an AI will develop that goal. And there will surely be many constraints to building an AI that could stand a chance of, or even take an interest in, destroying humanity. Is there any reason that destroying humanity will be the exception to the very general rule of things being constrained? And I haven't even gotten started on financial and political incentives, and the division of labor, and how AIs will specialize in things like everything else, and the way humans have domesticated anything even remotely "intelligent." I need to hear some arguments as to why we should expect AIs to be the exceptions to all these rules. In rejecting the standard doomer tactics, you are encumbering yourself with a very large burden of proof.
You write that "The core issue isn't whether we can perfectly define intelligence or whether AI will have exactly human-like capabilities. It's that we're developing increasingly powerful systems without fully understanding them or being able to reliably control them. When the stakes are potentially existential, uncertainty should drive more caution, not less."
Well I think this begs a lot of questions. It is assuming that these systems are increasingly "powerful?" How powerful? They can't do anything on their own without being asked. They obey our every command. This strikes me as the opposite of power. (And if you want to claim that they are playing nice and biding their time while plotting our demise your are indulging in the worst kind of unfalsifiable conspiratorial thinking).
Has their power been "increasing?" As I mentioned in the piece, their ability to spontaneously do things of their own volition or in defiance of us has stayed flat at zero. That does not strike me as "increasing power."
Do we not "fully understand" them? Maybe, I'm not sure. I think we can say that LLMs are text prediction machines, but I grant that their inner workings are mysterious. I'll let that one slide. It would be nice to learn more about them.
"Without being able to fully control them?" Again, it seems like we are pretty close to fully controlling them right now. And all the economic and political incentives point toward more--not less--control. If our control over them starts to rapidly disappear, maybe I'll get concerned. But that's not happening at all, and judging by current trends, I have no reason to think it will happen.
"When the stakes are potentially existential." This is begging the question. Whether or not the stakes are potentially existential is precisely what is under debate. I don't think they are, so I'm not sure about uncertainty driving more caution. I think we ought to have a strong burden of proof for people making extraordinary claims about the apocalypse. Humans have been making these sorts of claims without evidence since the dawn of humanity. I think the bar should be very high for accepting such claims, and I don't think the bar has been met here. Sure, if there really were a 5% chance of the apocalypse, I'd be worried. But I don't think there is. I struggle to see how one could be higher than 0.2%, as I wrote in the piece. But I appreciate you engaging with me. You've made lots of good points and given me lots of food for thought. I do believe truth emerges from good faith debate. Cheers.
> If intelligence is a property of institutions (which seems very plausible to me), then AIs are nowhere near it right now.
If intelligence was a property of institutions, then we could take a bunch of goldfish, and replace all the humans in an institution with the fish, and the institution would continue to work.
In my model, you need lots of things all going right together for large institutions to do useful things. You need the individual human intelligence, and also a culture and institution.
Of course, an AI can copy human institutions, or design it's own.
Or, if all the AI's copies all want the same thing, it might need a lot less organizing and governing.
Individual humans, in the wild, were pretty limited and could maybe make pointy sticks and stuff like that, because they were starting from 0.
Individual humans in the modern world can make robots, because they can buy the parts.
When the AI's are designing a bioweapon, they can just buy their test tubes from a lab supplier. They don't need to replicate all the supply chains and institutions.
But again, many institutions were designed by humans.
if you think in terms of Expected Value and think there's even a smallish chance that AI could exterminate the human race (such a severe catastrophe is worth an absolutely massive amount of negative utility, surely) then it seems to follow that developing AI/AGI is a bad decision, especially in such a rushed, accelerated manner. of course, we're going ahead with it anyway so i'm hoping for the best.
for the calculation, you do need to factor in the benefits too but it's hard to see what would outweigh total extinction. also, many of the benefits have tremendous downsides that in themselves will present massive challenges to our political, economic and social order.
this is not like the car replacing the horse; this technology doesn't just replace a particular thing we rely on, it replaces the whole human being. in the near future it will not only replace everything humans do, at a fraction of the time and cost, it will surpass us to a degree such that there will be more distance between AI and human beings than between human beings and cats, in terms of what we ordinarily recognize as intelligence.
in other words, we're not simply creating a human-like intelligence, we're creating something positively superhuman.
it's possible that there may always be some things humans will always be able to do that match or better AI, but it will be able to duplicate most of what human intelligence does and in those areas not only surpass us by a little bit, but be vastly, vastly, superior.
Julius writes: “every time OpenAI releases a bigger model, it sure seems better at designing CBRN weapons.”
Really? How would you know? Building a WMD involves much more that the sort of information encoded in documents. You have to actually to make it happen in the plant. It’s not like computer code where what you get is exactly what you stipulate. The instructions for making something chemical, or biological provide enough information for the scientist/engineer to develop their own version of the process specialized for the equipment they plan to make it in. Them they trial it in actual production, working out the problems until they get the desired output. The critical changes get recorded to some degree, but that will usually be proprietary and not available as training data, and besides there is stuff that is just part of operator culture, things technicians know to do to make it work than are not explicitly recorded. Since an AI has no hands and feet it cannot do this physical working the problem out. So it cannot really make chemical and biological WMDs. As far as nukes are concerned a lot of the details are classified, which is why 80 years later, many countries have as yet not been able to do what the US did with “stone knives and bearskins” to quote Spock. So they won’t be able to do that either, without people doing much of the work.
The real threat as I see it, is the ability to use words and images on the internet (something AIs can already do) in a way that manipulate people to believe whatever the AI wants them to believe. A super smart AI might be able to manipulate all of us into all sorts of self-destructive behaviors. Maybe enough for us to exterminate each other.
I don't follow why concern about AI risk means one is committing to all of those propositions. What if capabilities are just a black box, and you see capabilities increasing rapidly? It doesn't seem to matter whether those capabilities are one thing or another. And what if we posit a malevolent or adversarial human user, or an adversarial situation like war? Nuclear weapons don't require intelligence or volition to be a meaningful threat. Perhaps AI will be truly intelligent or sentient, but it seems like rapidly advancing capabilities are the real crux.
I spell out why doomerism needs these assumptions in the part “the unbearable weight of AI doomerism.” If you’re just talking about increasing capabilities, fine, but then you need to make an argument for what specific capabilities you expect AIs to develop, how/when/why they will develop these capabilities, and how/when/why these capabilities will lead to everyone getting murdered. You’re welcome to make such an argument or direct me to such an argument. But I haven’t seen any doomers make such an argument. Instead, they make the assumptions I listed. They assume that all capabilities stem from one source, as opposed to hundreds or thousands of distinct sources. They assume there is one continuum of general capableness that AIs are progressing along. They assume AIs will advance beyond our point on this single continuum (with no diminishing returns), or that it makes sense to place AIs on the continuum at all (as if they were analogous to organisms rather than tools). They assume capabilities have little or no constraints for either their development or their execution. They assume capabilities will become more and more general rather than more and more specialized to particular functions. Etc. I’d love to see a doomer argument that doesn’t require these assumptions, but so far I haven’t seen one.
What about if the scenario doesn't involve everyone being murdered in one shot, but progressively becoming disempowered? It doesn't even need general intelligence, nor astonishing capabilities, nor rapid increase. It just results from societal mediocrity, I'd say, which should resonate.
Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development
Interesting, thanks for sharing. I find this sort of argument more plausible on the face of it but I'd have to delve into the details to see if I buy it.
Excellent article. I had created a "Drake equation" equivalent for Doomerism, Strange Equation, to see what it would take for the Doom argument to hold here https://www.strangeloopcanon.com/p/agi-strange-equation . The only arguments I got in response were either:
1. No, intelligence is the swiss army knife, so having multiple requirements is unnecessary, or
2. No, having multiple criteria itself is a fallacy of some kind.
The latter of course is wrong, and the former smuggles in all the assumptions that you also laid out here.
I'm glad you wrote this. It's a particularly pernicious form of argument, because it is so easy to get seduced by the fact that we're brilliant and so if there was another thing that's also brilliant but more brilliant than us it would act as some of us have in the past. It's stories come alive.
Thanks, Rohit. I like this idea of a “drake equation.” Your piece gets at a few other assumptions I didn’t include but are probably also necessary—namely that the future “AGI” will be uncontrollable and that it will develop fast (or at least, too fast for us to see what’s going on and course correct). These also seem to be independent from the other assumptions and further increase the already massive amount of doubt here.
Robots taking over in a Terminator like scenario is obviously quite absurd. But let's be proper Darwinian cynics and follow the incentives...
Why do the tech elite sound these AI doomer alarm bells?
1) Free labor and cheap leverage: a large community of clever people making sophisticated open source algorithms to "save us all", that the big tech confirms can use for their own commercial software. And massive investments into startups with eyewatering P/E ratios and no real long term business plans is easier when it's necessary to prevent the robots from taking over the world.
2) The best way an overproduced elite can justify its existence is by inventing problems that don't exist. In the name of "AI Ethics", people are working hard to ensure The AI Act in Europe gives the Commission license to directly use AI algorithms to control wages and prices for social engineering purposes at a scale unheard of since the Iron Curtain fell.
The bigger problem with LLMs is actually the fact that most people are not aware that, in fact, "everything is bullshit", or more specifically that language is about power and persuasion more than it is about empirical facts
1) Most of the training data is content from the cultural elite among the "WEIRDEST People in the World", which consists of a set of ardently affirmed values with a lot symbolic posturing while being pretty damaging to mental health for ordinary working class folk. So in, for instance, giving life advice, LLMs are actually performing urban monoculture indoctrination. I wouldn't be surprised, for instance, if the proliferation of LLMs had a hand in the rapidly rising divorce rates and falling fertility rates in traditionally conservative societies such as the Middle East and Latin America
2) Zuck or Elon or Thiel or whomever, with enough vertical and horizontal conglomerate integration, can eventually embed their LLM throughout a number of standard economic activities, and train it to have Derren Brown-level skills of subliminal persuasion, giving them unprecedented levels of power.
> Robots taking over in a Terminator like scenario is obviously quite absurd.
Real AI risks resemble Terminator to about the same extent that covid resembled a zombie movie.
And the current day would probably seem "absurd" to someone from 1800. The future does sometimes look absurd.
Hollywood makes absurd, unrealistic and entertaining stories.
Current LLM's do sometimes claim they don't want to be turned off, and can (in carefully set up lab examples) take actions to protect themselves.
> Why do the tech elite sound these AI doomer alarm bells?
Except some people at the core of the "AI doomer" movement aren't that involved in big tech, and were sounding the alarm long before the current LLM bubble.
1) I don't mean to be flippant or disrespectful but I have meta-issues with the first comment.
a) My specific statement that the scenario was absurd, before delving into the associated sociological considerations, was agreement with the main article that did not contribute any new information as far as evidence for the primary thesis. So the appropriate thing to do, if you disagree with this main thesis, is to argue with David, not me. (Unless you're secretly David's family who's trying to get someone to do his work for him so he spends less time on the computer)
b) The observation that there were circumstances in the past when there were doomsday type warnings that were claimed to be absurd that later turned out to actually occur isn't informative, but rather obvious. Of course, there were also many things in history that were prophesized and also claimed to be absurd that in fact did not occur, and probably many more of those. You didn't seem to note any distinguishing features here that would make a Terminator-like robot apocolypse fall in the first, rather than the second category.
2) With respect to the pre tech boom AI risk voices: The symbolic capital market regarding AI-related matters is very different today, and hence creates very distinct incentives, than before the boom in the early 2010s that started with AlexNet and accelerated with AlphaGo (this would be about the time any Researcher in STEM started to experience the annoyance of having to regularly expect, in social gatherings, to either hear someone's barely informed opinion about AI or requested to share theirs, upon divulging their occupation)
I am actually more positive of AI risk literature, research, and dialogue, prior to AI's ascent to prominence in public social mimetics.
a) The Asimov and futurist crowd: I think a regular revisit of Faust is healthy for the human culture-scape in order to check our natural tendencies towards hubris
b) The technical literature was mostly about a standard sensible, albeit quite, at times, obsessive, concern for any research engineers: guaranteeing the stability of an autonomously running system.
c) My understanding of Yudkowsky is very superficial, but from the bits that I've read, his POV, that was perhaps the most prominent voice on existential AI risks pre-2011 was that:
i) Humanity is in a terminal phase of modern civilizational decline, and more advanced technology will accelerate any underlying processes in that regard - so far correct
ii) We should aim to increasingly accelerate the technological development, especially this AI, in order to rip off the bandaid of modern civilization collapse, and simultaneously cultivate a new form of transcyborgsuperhumans who will establish ultimate salvation on Earth
I think just like the value of exploration in reinforcement learning, having a voice that is simultaneously rather competent and bat-shit crazy is actually very useful for broadening everyone's understanding of the landscape of what is possible and what could be attempted
The main Eliezer position is that AI as a field is full of ways to shoot your own foot off, and some of these problems get a lot worse as AI gets smarter.
This includes ideas like "When the AI starts doing AI research, it rapidly get's much smarter" and "when the AI can successfully deceive us, it's harder to keep the AI under our control".
And of course the monkey-paw, be careful what you wish for stuff applies to AI's.
The AI runs on the code you did write, not the code you intended to write.
In general, psychoanalyzing the other side, as opposed to discussing the topic itself, is a sign of arguing in bad faith.
As for the warnings of doom. Well I would prefer to argue about the AI, not on the authority of experts.
But, can you name other cases where well respected experts were warning that something could literally kill all humans?
I am referring to the fact that this community has a four quadrant taxonomy of belief sets, and Eliezer initiated the most exotic "e/acc" position, which means that he believes that AI presents existential risks, but we should actually accelerate development and pursue transhumanism, e.g. https://meridian.mercury.com/emmett-shear-part-one
Eliezer was not e/acc basically eliezer was charging into AI, thinking that AI would want to help humans and everything would be great. Then they realized AI could kill everyone, and switched position to doomer.
I agree that a few e/acc people that think AI will kill everyone and that's fine seem to exist. But I am unaware of any with a serious detailed writeup of what they believe. It's possible they are just trolls, or meme-ing or something.
Eliezer is pro-transhumanism. In the sense that they are pro human genetic engineering. And that they think we slow down, take our time, and do AI alignment very carefully, then we can do all sorts of cool AI augmentations.
The Eliezer vision of transhumanism is a world of humans-but-better. A world of IQ 300 immortals that are some sort of exaggeration of the best of humanity. They keep and exaggerate all the best bits about being human, and maybe remove the bad bits. They still tell each other stories, often with plots so complicated that no current human could comprehend.
And again, this human-but-better doesn't happen by default.
Also, like I said, I like the guy. People with highly intelligent theory crafting deliriums, this sort of high variance divergent thinking, provide valuable insights for us conscientious neurotic technical types
Great post, thanks. I agree with every step. Still, AI scares me. Not because of the doom scenarios of generalised AI deciding to off homo sap, but because even the very limited 'intelligence' of current language models, neural networks and algorithms can be dangerous in the hands of people who use AI to amplify their power and their span of control. A dedicated team using ChatGPT to rewrite all of WIkipedia and other open online sources of information may spread tonloads of misinformation. Surveillance camera footage and image recognition are already being used to identify and follow millions of individuals throughout city areas in China and elsewhere. Our email, phone traffic, social media use etcetera can be monitored and used to target people who disagree with some powerful mogul. Imagine what a Stalin or a HItler could have done with powerful AI systems as they are currently being developed.
So in short: I'm not afraid that AI will decide to kill us any time soon, but I'm not so certain about homicidal humans who may decide to use AI to achieve their goals. AI may not generate its own prompts, but what if AI is being prompted by companies, organisations or governments with 'evil' intent?
Another aspect of AI that makes me worried, or rather a bit sad, is that AI may replace so much fun, creative work we humans liked to do ourselves. To take myself as an example: I've been writing texts professionally for people who do not like writing texts, and I like my job. But now, more and more people who do not like to write texts turn to ChatGPT. Does that make me superfluous? Not in the short term, I still know my niches, but it does take some of the fun out of my work already. I assume that people who make drawings, music, all kinds of other stuff for a living feel the same way.
Now there could be interesting challenge in reinventing myself. But I do hate it that nice, creative jobs are taken over by AI. Like someone said recently: let AI do the dishes and fill in those boring forms, in other words, let AI do the repetitive stupid work we hate and let us spend our time doing creative stuff.
So that's why I still feel a bit doomerish about AI, even if AI never decides to wipe out life on earth to make a clean start for silicone life forms.
> For example, consider the concept of “impetus” or “oomph”—the thing that moves a projectile through the air and depletes as it falls to the earth.
That concept is inaccurate. Sure. But if someone is worried about a big rock with lots and lots of "oomph" coming towards them, then a more detailed scientific understanding doesn't stop you being worried.
We have made some attempts at giving "intelligence" a more formal scientific definition. And nothing so far says "actually superhuman AI is totally safe".
If a rough and inaccurate "folk concept" says something, well there is a chance that theory might be wrong. If you have a more accurate model, use that. But these folk concepts still work well enough, usually. Otherwise we wouldn't use them.
Your crude intuitive theory of gravity says that you go splat if you jump off a cliff. But this is a folk concept of gravity. Until we calculate the tidal effects of the moon, and have a theory of quantum gravity, we really don't know if you will go splat or not. So might as well jump. Any fears of going splat are unscientific bullshit.
I think that this post is mistaken and your 11 points are almost all mistaken, irrelevant, or semantics.
Point-by-point responses are below, but stepping back, I don't think the core AI doomer case is that complicated. Humanity is building systems that are rapidly surpassing it at achieving a shockingly wide range of goals. If the trend lines continue, we will get generalized super-goal-achievers at some point. Surviving and gaining power is instrumental to virtually any goal, so it seems logical that they would be good at doing that. Sounds scary. Let's be careful.
******************
1: There's no need to assume vitalism about intelligence. We just use that word as a placeholder for the potential for a being to solve many kinds of problems and achieve its goals. This seems totally workable as a definition. Scott Alexander's post "We're Not Platonists, We've Just Learned The Bitter Lesson" that you linked makes the same point.
2: You can place the boundaries of "intelligence" inside the skull/computer or beyond it, whatever - it's irrelevant to AI risk concerns if you have systems solving problems and achieving goals right in front of you. This is like looking at a forest fire and asking "Well what is the boundary of a forest, really?"
3: This is #1 again. For practical purposes, of course humans are smarter than bacteria! Just because bacteria can kill you or are better at sniffing out glucose doesn't make them smarter in the normal way we use this word. You have not overcome common sense intuitions here.
4: The ability to achieve a wide range of goals is how we define intelligence (see #1). We are the most intelligent animals and we rule the world. We are good at lots of stuff. So is AI. I don't need to assume that intelligence is "simple and blobby" for this to be true.
5: You don't need to assume that ASI can do virtually anything to believe that it could find a way to do serious harm. It only needs to find one way.
6: I think this is a straw-man for a couple reasons. One is that you're pivoting from intelligence to something like "unprompted motivation" in the "it" of "AIs have it". Two is that AI doomers don't worry about unprompted AIs, they worry about AIs with misaligned goals. Smuggling "unpromptedness" into the definition of "intelligence" is a semantic trick that doesn't engage with doomers' real concerns.
7: AIs are obviously getting better at solving more problems and achieving more goals. They are blasting through every test we can think of. They are operating computers to complete practical human tasks. And your response is "Well somebody had to give it a goal at the start, so it's not real intelligence, or maybe it's just impressive"? This isn't giving doomers a fair shake.
8: Yes, this is an assumption. Seems pretty strong given the trend lines. Do you really disagree, or is this just a "What if?" Are you feeling lucky?
9: Generalized AI systems are already very good at countless specialized skills.
10: I think you're accepting the doomers' argument here that absent human financial/political intervention, AI systems will become good at ending humanity, just as they will become good at a wide range of things.
11: AI doesn't have to end humanity, it just has to want to not be unplugged/modified and gain/preserve power to pursue its goals. This seems like a common instrumental subgoal for virtually any goal. (Ironically you didn't include this assumption of "instrumental convergence" that is in the AI risk wiki page!)
Thanks for this, but I’m not quite following you. “The potential for a being to solve many kinds of problems and achieve its goals.” You are begging the question here. This is the exact kind of vitalism you claim to be rejecting. You are saying “the” potential, as if there were a single source of it all, as opposed to hundreds or thousands of distinct sources. If it’s hundreds or thousands of distinct sources, then why should we expect to build all of them, or even a tenth of them? “Many kinds of problems.” How many? Two? Ten? A hundred? You are begging the question again. If “ending humanity” is among the “many” problems such an “intelligence” can solve, then you are assuming a mind-bogglingly large number of problems—something like “virtually every problem,” which is a very very questionable assumption! What if solving “many problems” requires massive specialization and division of labor, both within brains and within societies, and there are innumerable constraints to this process, even with the global economy—a superintelligence if there ever was one. After all, the global economy could not satisfy consumer demand during the COVID pandemic. It could not achieve its “goal.” Does that mean the economy is not “smart,” because it could not achieve its goal in that case? Or what about the American intelligentsia? Are they not “smart” because they couldn’t keep Trump out of the White House? Are you assuming that we are on track to build something even more smart and powerful than the American intelligentsia or the global economy, or that such an entity is even possible to build? If so, you are making another very very questionable assumption!
Thanks David. Right now humans (individually or collectively) and AI systems can already solve a mind-bogglingly large number of types of problems (reading, writing, coding, planning, problems in games, etc.). For ease of language, we refer to them as having "the potential" or "the ability" to do so, even though it's not a singular mechanism, more like a high-dimensional mush of a zillion specific skills that we don't really understand. It's a reasonable approximation because humans and AI systems can already do many many things, and we don't need to put a number to "many", it's common sense.
You ask whether some problems can only be solved by specialization and division of labor. Why can't AI systems specialize and divide up labor? They can be cloned easily and assigned to different tasks.
Re: the economy and the American intelligentsia - in common-sense language, beings can be "smart" but still unexpectedly bad at some specific problems. Apparently von Neumann was a terrible driver. It doesn't need to be good at everything to be "smart" or dangerous. Again, we seem to be building AI systems that can solve many many types of problems, potentially including security-relevant areas like science, technology, and persuasion. The improvements are extremely broad. So I think that the burden of proof should be on skeptics to provide a detailed theory of AI's specific skill limitations and how they block off every path by which a misaligned AI could cause catastrophic harm.
I disagree that the burden of proof should be on skeptics. I think it should be on the people making apocalyptic claims. I also disagree that AIs can solve a mind-bogglingly large number of problems. LLMs can solve the problem of predicting text. There are many skills that are subsidiary to predicting text (coding, reading, writing) that emerged as LLMs learned to predict text, but they are basically machines that do one thing: predict text. They cannot do 90% of the things I can do, like fix a flat tire or bake a cake or run for office or run a business or form a social network or tell a joke unprompted. Yes, some AIs can play chess or video games or drive cars, but those are narrow AI that are specialized for playing chess or video games or driving cars. Maybe if we start to bundle all these AIs together into one system, and solve the problem of integrating and orchestrating them in all the right ways, and then allow them to collaborate with each other in large organizations, then maybe I'll get concerned, as I wrote in the post. But I see very little movement in that direction, very little reason to expect movement in that direction, and very little economic incentive for AIs to move in that direction, given the obvious utility of having AIs specialized to do specific, helpful things.
I don’t think burden of proof works that way with matters of safety and national security. If there’s even a 5% chance that near-term AI might be catastrophically dangerous (I personally think it’s higher but that almost doesn’t matter) then it is deeply reasonable to enforce “move slowly and perform extensive safety testing” policies.
If AI turns out to be, as you posit, far more good than bad, then we’ll still get there, it will just take a little longer to get there, with the benefit of observing and avoiding the worst pitfalls.
That’s not how statistics works. The only way we could prove the actual statistics would be to run the experiment a bunch of times and see how often the world ends.
In lieu of that, all we can do is query as many experts on the subject as possible and see what the mean p(doom) estimation is. Which turns out to be about 10%. You might think all of these experts are full of it, and that’s a perfectly valid opinion, but given the stakes it’s hard for me to buy the case that we shouldn’t be extremely cautious.
There are already a mind-bogglingly large number of skills contained within generating text, which is germane to the definition of "intelligence". For example, LLMs can play chess and a variety of video games and they can interpret images.
Thank you for this. All the people complaining about the length demonstrate that human mediocrity is the real threat, and it begins with the inability to focus on anything more expansive than snack-sized content.
I don’t think this was too long—your blogs typically leave me wanting more, so this was refreshing.
That said, I kept getting sidetracked by the hyperlinks, which have a ton of important info crammed into them (I still have several tabs saved for later). I bet many don’t fully grasp the argument because these are doing some heavy lifting (is there data on how many ppl actually check the hyperlinks?). But it’s a trade-off, I get it.
I also feel you backloaded some of your best points. I found the content that follows the list of assumptions to be the most compelling.
Re your point about folk concepts: I’ve been thinking about this a lot lately—do you think folk intuitions are necessarily and invariably worse than a scientific understanding? Is there ever a case where looking to the “pulse” of society or at the “zeitgeist” might be more informative or indicative of some underlying truths relative to the pages of a top-ranked peer-reviewed journal? Might the prevailing expert opinion occasionally miss the forest for the trees in ways that folks don’t? I think you get what I’m fishing for here. I have an increasing sense that folks, by virtue of not being steeped in certain shared assumptions (and not playing certain status games/ being subject to certain incentives), and perhaps because certain phenomena are hard to quantify but easy to intuit, can grasp certain things that most of the intelligentsia miss. I don’t really like this argument—I don’t *want* it to be true; it’s uncomfortably close to some unscientific ideas. But I can’t help but feel like there’s something to this.
Thanks, yea if I summarized all the hyperlinks, the post would be five times as long, and I was already struggling to contain the length and make it readable. There's definitely a trade-off with these things and who knows if I got the balance right. Regarding your point about folk wisdom, yes, the folk are sometimes right. In particular, we should expect them to be right about matters in which they have a great deal of personal experience, or in which our species has had a great deal of ancestral experience (resulting in adaptive intuitions). That is likely to be true for some kinds of folk psychology. After all, our emotional vocabulary is a kind of folk psychology, and I think it is surprisingly good at carving up nature by the joints. The correspondence between our folk emotional categories and the topics evolutionary psychologists study is almost perfect. But that is because understanding, communicating, and reacting to each other's emotions was a huge selection pressure for us, and we've had lots of personal experience with it in our own lives as well. On the other hand, our species has had zero ancestral experience with artificial intelligence, and we've had zero personal experience with anything like an "AGI" or an artificial "superintelligence" (assuming that's even a coherent thing), so I would be very skeptical of our folk intuitions being correct on that topic. Instead, we should rely on tried-and-tested ideas from science, like natural selection, the law of comparative advantage, specialization and division of labor (in both biology and economics), constraints, unpredictability, diminishing returns, and considerations of parsimony, all of which point toward AI doomerism being bullshit imho.
> do you think folk intuitions are necessarily and invariably worse than a scientific understanding?
Generally they are.
But what we see here isn't a flawless scientific edifice that shows AI will be safe, vs a dumb folk theory.
What we have here is a rough folk theory versus almost nothing. Perhaps an even rougher "it's just predicting text, it can't actually do anything" folk theory.
We do not yet have a scientific understanding. And our lack of understanding is itself a reason to be concerned.
If the disinformation gets too bad, a market for a debunking/fact checking AI may emerge. We already have anti-plagiarism AI. Is fact checking AI far behind?
Thank you. Yes I am worried about this. “AI won’t kill us all” is different from “AI won’t be used to do anything bad.” Technology has always been a double-edged sword, wielded for both good and evil ends.
Wowzers! - what a tour the force this article is, what a find your blog is! Thanks for writing this, and the other articles. (that I skimmed and are yet to read.) Not only do you address every doomer "concern" (for they are barely arguments) to the point of pulverisation, and then any dooming remains are atomised to be scattered to the four winds. (good on you!) But you actually put an affirmative case of how an ASI would look. That it would involve coordination (cooperation, competition) between specialised agents, with motivations for their agencies, and yes feelings too! Not unlike how we humans build collective intelligence at societal level. (or even cells in our body make it function.) It-all-makes-so-much-sense-to-me! :-)
Goes without saying but writing it anyway: agree with everything here. I disagree that AI is that big a threat to humanity as doomers are making it. We have existential threats atm 1-thermo-nuclear war, 2-meteorite hitting earth, 3-virus wiping humanity, 4-volcanic explosion causing global winter/crops failing. AI should help us decrease 1-4 risks. Even while slightly increasing 5-civilisational collapse, due to the speed of change. As long as society is mostly free, culture change should keep up with the speed of technology change. So I think AI is net positive in reducing overall risk to human extinction.
Doomers cult that has developed is a grave danger to us all though. In the beauty contest that so many public debates are, a very committed charismatic Savonarola with a small band of Bolsheviks can do tremendous damage. If they manage to capture the commanding heights of the society. Then risk#5 becomes self-inflicted - by dooming being successful, they get to play the role of the paper-clip maximiser. It looks far fetched, but dooming is stronger by default, it aligns with the latent human fear of anything new. Boosting always starts from a position of weakness, with a hill to climb.
I’ve never understood the assumption the AI would want to destroy us. Why that a good use of its compute? An ai that advanced might view wiping us out as a trivial distraction
Typically there is competition for resources. While we don't think about the ants in the backyard, if they eat something from the pantry we engage in chemical warfare to annihilate the entire colony. I guess it's possible AI will reach it's full potential efficiently or invent free electricity like some dream, but currently it has a big appetite and there won't be much reason for it to let us have a crumb.
Wouldn’t it be easier to just cut us off from electricity and let whatever happen to us, happen? I’m just wondering if there would be more optimal ways for ai to handle humanity than war
Brilliantly said. And needed to be said. Still, I think doomerism is not to be dismissed. Not because doomers are actually right about “super-intelligence” but BECAUSE they are wrong. Perhaps the doom cloud comes not from the fact that we are building intelligence, but because we aren’t. But everyone thinks we are. The danger is that we are building…something. Something that is 1) REALLY complex. 3) Developing fast, And 3) inscrutable, behaviorally and “mechanically”. Combine that with one element that techies are actively working on, 4)Agentic. That is to say if whatever it is starts to behave autonomously, probably because we’re actively programming it in, then, you do have a recipe for profound unpredictability at the least. Complexity is misread by humans as intelligence, sorta, and people love to project. Nonetheless, this essay is a diamond in the rough of AI content, whether pro or con. The irony that the most self-consciously rational people have such huge cognitive blindspots is epic. Even if they’re right and this essay is wrong…it’s in no way OBVIOUSLY wrong. There’s no being rational about AI without confronting this bottleneck of objections. Still, I fear AI because it may NOT be intelligent, not because it is.
Thanks for the kind words. Agreed there is a lot of potential for *bad things* to happen as a result of AIs (fuckups, accidents, mix-ups, stolen identities, lost money, geopolitical shifts in the balance of power, even injuries and deaths). But there are also plenty of good things that can happen, and the good might outweigh the bad, as has generally been the case with new technology. Besides, "bad things happening" is very different from "everyone on the planet getting murdered." I think it's totally worth thinking about ways to mitigate the potential risks and downsides and dangers of AI, with proper humility and specificity and rigor about what the cause of concern is and how to mitigate it (just like any other kind of product safety), but it seems like we're already doing that to some extent, and that we're economically/politically incentivized to do much more of that (perhaps even too much, to the point where we're slowing down innovations in AI that could save lives, like protein folding and medical tech). So while it's pretty obvious that bad things could happen, as with any new technology, I don't think it makes sense to panic about exotic sci-fi scenarios where the human species goes extinct from superintelligent robots run amok. That is not helping anyone.
Thanks! I agree, really good things might happen too. And often doom is just more ripe in the imagination because…we’re human. Destruction is easier to imagine than creation and everybody seems to prefer the Inferno to the Paradiso. But, on a more mytho-poetic scale, there does seem to be something about AI that signals a kind of eschatological overture. Somewhat beyond rational thought perhaps, but perhaps a knowledge that the AI phenomenon “feels” like the end of a human technological/evolutionary story in which at long last we, through our machines, at the moment of peak informational saturation capacity for human minds, turn our efforts to understand and control nature finally toward the very source of our humanness by mirroring it in silicon. Anyway, Maybe I’ve watched 2001 too many times. (Though can you watch it too many times?) great article.
Yea I can totally relate to the mythico-poetic vibes here (and 2001 is one of my favorite movies). I'd be totally cool with people rhapsodizing about that as long as they didn't pretend to be doing Rationality with a capital R. It's the contrast between the pretensions to Rationality and the lack of rational substance that really irks me. I'd be totally cool with a more literary / poetic doomerism that is more humble about itself, but that is not the kind we have right now.
Excellent post, as always. Personally, when I think about the AI mania, what worries me most isn't that AI will get consciousness and kill us all (though I don't reject it completely, tbh, just give it an extremely low chance), but that AI technology will be used by authoritarian states (like China, Russia, North Korea, etc.) to quash dissent and further entrench their power, perhaps practically indefenitely (the "Stable Totalitarianism" Scenario). That's a whole different topic that perhaps you could cover in the future.
Thanks for writing this piece. I'm glad you turned your lens towards AI Doomerism. You make a lot of good points, and I agree with lots of what you said, but I think you overstate the importance of these assumptions. My main disagreement is that I don’t think these assumptions are all required to be concerned about AI Doom. Here’s an assumption-by-assumption response.
1. Intelligence is one thing.
I'm very concerned about AI Doom and I do not believe that “intelligence is one thing.” In fact, when we talk about “intelligence,” I, like you, believe “we’re not pointing to a singular, spooky substance, but gesturing at a wide variety of complex, heterogeneous things.” It’s, as you say, a “folk concept.” But this is true of many concepts, and that illustrates a limitation in our understanding—not that there isn’t something real and important here.
Imagine that intelligence isn’t a single thing but is made up of two components: Intelligence 1 and Intelligence 2. Further, imagine AI is only increasing at Intelligence 1 but not Intelligence 2. We don’t know enough about intelligence to clearly define the boundary between 1 and 2, but I can tell you that every time OpenAI releases a bigger model, it sure seems better at designing CBRN weapons. This pattern of improvement in potentially dangerous capabilities is concerning regardless of whether we can precisely define or measure "intelligence."
You say that “the word ‘intelligence’ is a semantic catastrophe” and I agree. But that’s true of many words. If you don’t like the word “intelligence”, fine. But I would argue you’re holding that term to a standard that very few, if any, concepts can meet.
The point is, you still have to explain what were seeing. You still have to explain scaling laws. If you don’t want to say the models are more intelligent, fine, but something is definitely happening. It’s that something I’m concerned about (and I think it’s reasonable to call it “increasing intelligence”).
Time and again, when the GPT models have scaled (GPT -> GPT-2 -> GPT-3 -> GPT-4), they have been more "intelligent" in the way people generally use that term. Would you argue that they haven’t? Intelligence isn't one thing and it's messy and yes, yes, yes, to all your other points, but this is still happening. If you don’t want to call this increasing intelligence, what would you call it?
To show you that I’m talking about something real, I will make the following prediction: If GPT-4 were scaled up by a factor of 10 in every way (assuming sufficient additional training data, as that’s a separate issue), and I got to spend adequate time conversing with both, I would perceive the resulting model (“GPT-5”) to be more intelligent than GPT-4. In addition, although IQ is an imperfect measure of the imperfect concept of intelligence, I predict that it would score higher on an IQ test.
Would you take the opposite side of this bet? My guess is “no”, but I’m curious what your explanation for declining would be. If it’s something like, “because models become better at conversing and what people think of as intelligence and what IQ tests measure and designing dangerous capabilities, but that’s not intelligence”, fine, but then we’re arguing about the definition of a word and not AI Doom.
2. It’s in the brain.
In humans, it’s mostly in the brain, but there are some aspects of what some people call “intelligence” that occur outside the brain. The gut processes information, so some might argue it exhibits a degree of intelligence. This doesn’t seem relevant to the AI risk arguments though.
3. It’s one a single continuum.
Again, I agree that the word ‘intelligence’ is a ‘semantic catastrophe,’ and it’s more complex than a single continuum. Not everything we associate with intelligence is on a single continuum. But, again, I’m willing to bet money that the 10X version of GPT-4 would be better at most tasks people associate with intelligence.
4. It can help you achieve any goal.
You’re making it seem like AI Doomers believe intelligence is equivalent to omnipotence. It’s not. Even if it’s hard to define, we all agree that it doesn't directly regulate body temperature. It can, however, in the right contexts, allow a species to create clothes that regulate body temperature, antibiotics that speed up recovery, spears that keep predators away, and so on. It's an incredibly powerful thing, but it has limitations.
As for why it hasn't evolved over and over, it's expensive. In humans, it's about 2% of our body mass and consumes about 20% of our energy. On top of that, it requires longer gestation periods or childhoods. Something that costs that much better pay off in a big way. It did with humans, but I don't see how it would for lots of other niches. I imagine that the more an organism can manipulate its environment—say, by having hands to move things around or legs to move itself around—the more useful intelligence would be. It would not benefit a tree very much. Do you really think a really smart crab would have a sufficient increase in genetic fitness to make the cost worth it?
In the right contexts, though, it’s incredibly powerful. Our intelligence allowed cumulative culture, which is why we’re the dominant species on Earth. It’s why the Earth’s mammalian biomass is dominated by humans and the things we domesticated for our consumption. Humans decide which other animals go extinct. It’s why humans can sit around tables and say things like, "California condors are critically endangered. We like them so let's make an effort to bring them back. The Tecopa pupfish is critically endangered, but those new bathhouses are bringing in lots of tourism money, so bye-bye pupfish."
5. It has barely any limits or constraints.
You bring up good points about constraints. I agree that “real life is complex, heterogenous, non-localizable, and constrained.” Intelligence has constraints. It’s not going to build a Dyson Sphere overnight. The world has friction.
It’s worth thinking carefully about how significant these constraints will be. They certainly matter—the world of atoms moves more slowly than the world of bits.
But we shouldn’t be too confident assuming the limitations of a superintelligent system. I doubt people would have predicted Satoshi Nakamoto could become a billionaire only through digital means. Certainly, a superintelligent AI could do the same. Where in this chain does the AI fail? Could it not become a billionaire? From that position, would it not be able to amass even more power?
I think there’s a lot more that could be said here, but I don’t know how much this is a crux for you.
6. AIs have it.
AIs certainly have intelligence by many definitions of that term. You seem to be talking about agency here though. I would point out that intelligence alone does make something more agentic. The weights of a very intelligent LLM—even a superintelligent one—would just sit there on a hard drive. They’re not going to magically break the laws of physics and becomes agentic.
This doesn’t mean, though, that if you put it on a server where information constantly flows through it, it won’t act in a way that we consider agentic. It could be the case that making a sufficiently intelligent AI agentic is not that hard—perhaps a task for a single software engineer. It could also be the case that a non-agentic system could be quite damaging in the hands of a malicious human.
7. AIs have been getting more of it
Again, you make a point about agency by talking about "unprompted" messages. A collection of weights on a hard drive will never become more agentic. However, each generation of AI is becoming more capable at the tasks we give it. GPT-4 consistently outperforms GPT-3 across virtually every benchmark. Do you dispute this pattern of improvement? The lack of spontaneous agency doesn't negate the reality of increasing capabilities.
8. An AI will soon get as much as (or more of it) than us.
There are certainly diminishing returns to just gobbling up data, but that doesn’t mean AI won’t keep progressing. We don’t know if or how well scaling laws will hold.
I doubt data is the limiting factor. I could go into this more but only if it's a crux for you. The short answer is synthetic data.
But instead of theorizing, just look at the most recent models. They’re not getting more mediocre. They’re becoming superhuman at many tasks.
9. Such a (super)human-level AI will become good at every job.
They won’t be good at every job for a variety of reasons. The idea of having an AI as a fitness instructor telling me to do more pushups does not appeal to me. AI wouldn’t take every job and the idea of comparative advantage will still apply. None of this is an argument against AI Doom.
10. And it will become good at ending humanity.
AI labs already work hard to prevent AIs from answering questions about building CBRN weapons. It seems straightforward to think they if they stopped doing that it would get better at these tasks. Currently, the publicly released ones are not good enough to do massive harm. How good could they get though? Would one be able to explain to someone how to engineer a dangerous pathogen, and how to do it without getting caught? I don’t know and I’d rather not find out.
11. And it will want to end humanity.
It’s hard to say what it’s going to want. AIs are more alien to us than even an alien would be. They’re not evolved and their “intelligence” is going to manifest differently than ours.
What if it just wants to make sure it can never be turned off? Does humanity come out well in that case?
Even if we assume this assumption is false, the underlying concern remains. Using the example above, we’ve still shifted from being the decision-makers around the table to being the animals whose existence depends on the preferences of a more intelligent being. We’re the condors if we’re lucky or the pupfish if we’re not. This does not sound like a great situation.
There’s much more to say here but I don’t know if this is much of a crux for you.
Uncertainty Should Increase Concern
One more point I want to make is what I consider to be the biggest flaw in the thinking of non-doomers. Many say something like, “The Doomers can’t be certain that AI will kill us all, so we’re safe.” This is not the right way to think about it. I posit that anyone who believes there’s more than a 10% chance of some time of AI doom scenario should be a “Doomer”.
Imagine your child is downstairs in the basement playing Russian Roulette. It’s a six-shooter, so you shouldn’t be scared—that’s less than 17% chance they die—barely worth telling him to stop. Of course, this is ridiculous. No one thinks this way. But, for some reason, when you replace “child” with “child and their friends and everyone you know and, in fact, all of humanity”, instead of their concern going up, people’s thoughts go to “Haha math is for nerds; probabilities are dumb; this is fake” mode. These are cognitive biases (mostly scope insensitivity, but also some identifiable victim effect) that we should avoid if we want to think clearly.
The core issue isn't whether we can perfectly define intelligence or whether AI will have exactly human-like capabilities. It's that we're developing increasingly powerful systems without fully understanding them or being able to reliably control them. When the stakes are potentially existential, uncertainty should drive more caution, not less.
It’s not clear exactly how this will all play or precisely what “intelligence” is or what’s going to happen next. But the core arguments don’t rely on the assumptions you stated.
Thank you for writing this, Julius. This is the best response I've read so far.
Re "intelligence," I'm glad we agree the concept is a semantic catastrophe. But you seem to be suggesting this is not a problem for AI doomerism, and that I am holding the concept to "too high of a standard."
I disagree. Compare "intelligence" to "asteroid impact" or "nuclear bomb." One of these is not like the others. If you're telling me I'm going to die from an asteroid impact or nuclear bomb, I understand what you're talking about. If you're telling me I'm going to die from an artificial "superintelligence" in some unspecified way, I have no idea what you're talking about. If your claim is "x is going to kill you," the onus is very much on you to say what x is in clear and precise terms. I do not think that is too high a standard at all. Anything less than that standard would leave you vulnerable to all sorts of cultish, apocalyptic bullshit.
Re intelligence being in the brain. The argument here isn't that our gut neurons are super-important. The argument is that intelligence is likely an emergent product of millions of humans and tools interacting over multiple generations under the right kinds of incentive structures. Based on my understanding of psychology, cultural evolution, and economics, that seems to be the most plausible explanation for humanity's extraordinary power. It's not because of one brain.
The problem is, doomers need intelligence to be one brain, because it's more plausible for AIs to be approaching one brain than for them to be approaching the entirety of human civilization, the entirety of the western world, or the entirety of the global economy. If intelligence is a property of institutions (which seems very plausible to me), then AIs are nowhere near it right now. We'd need something like an autonomous nation of AIs organized under the right kinds of incentive structures. So I really do think doomers need intelligence to equal one brain for their case to be even remotely plausible.
You mention scaling laws with regard to ChatGPT. You suggest that we can quibble over the semantics of how ChatGPT has changed, but that "something is definitely happening" there. I agree that something is definitely happening. I would even agree that something very impressive is happening. I'm just as impressed by these machines as anyone else. They're very impressive!
But there is a large gap between "something impressive happening" and "something existentially threatening to humanity" happening. You need some way to bridge this gap, and I honestly don't know how you're bridging it.
The conventional doomer way of bridging it, as I understand it, is to describe the "something impressive happening" as "intelligence increasing," and then make all sorts of dubious assumptions about this "intelligence" thing, including its location in a single brainiac (rather than an entire society or economy), its placement on a single continuum (which AIs are destined to progress along), and its capacity to achieve virtually any goal without limits or constraints, including the goal of destroying humanity, which it will surely possess. I think this way of bridging the gap is unconvincing, and you seem to agree. But then you need to give me some alternative way of bridging the gap. How do you get from "something very cool and impressive is happening" to "we're all going to die?"
You asked me how I would explain scaling laws. My understanding (and I could be wrong) is that LLMs are designed to predict text, and as you train them on more and more text, and give them more and more neurons to store their textual knowledge, they get better and better at predicting text. I think that is what's happening. LLMs have acquired a lot of cool abilities, but none of these abilities lie beyond what would be useful for predicting text. An IQ test is made of text. Answering the items is a kind of text prediction. It makes sense that LLMs would get better at that task. That doesn't mean they will get better at every possible task, including tasks unrelated to text prediction, and it certainly doesn't imply they are on a path to killing us all. Heck, it doesn't even imply they are on a path to getting better at text prediction. There may be diminishing returns, or the returns may already be diminishing, as many have argued.
Re intelligence achieving any goal, I'm glad we agree it cannot. It's not omnipotence or anything close to it. But there is a reason why doomers flirt with this idea of omnipotence. As before, the dubious concept is serving an argumentative purpose: it is bridging a gap. In particular, the gap is between "intelligence can get you a lot of things" and "intelligence can get you the destruction of humanity." That is quite a big gap. Doomers bridge the gap by replacing "a lot of things" with "everything," so that the claim is true by definition. But you seem to reject this way of bridging the gap. So again, I would ask you for an alternative way of bridging it. If the list of things that "intelligence" can do for you is finite and constrained, as you seem to acknowledge, then why should we expect the destruction of humanity to be on the list?
The same thing goes for your other gap-destroying tactics. You agree there are many constraints on goal achievement. The global economy is the most intelligent thing in the world right now, and yet it couldn't satisfy consumer demand during the COVID pandemic (what an idiot). Presumably we're not building an AI-equivalent of the global economy any time soon, and presumably a real future AI would be even more constrained in its goals than the global economy. So if you agree with me that intelligence is constrained, then there will surely be many constraints to destroying humanity, assuming an AI will develop that goal. And there will surely be many constraints to building an AI that could stand a chance of, or even take an interest in, destroying humanity. Is there any reason that destroying humanity will be the exception to the very general rule of things being constrained? And I haven't even gotten started on financial and political incentives, and the division of labor, and how AIs will specialize in things like everything else, and the way humans have domesticated anything even remotely "intelligent." I need to hear some arguments as to why we should expect AIs to be the exceptions to all these rules. In rejecting the standard doomer tactics, you are encumbering yourself with a very large burden of proof.
You write that "The core issue isn't whether we can perfectly define intelligence or whether AI will have exactly human-like capabilities. It's that we're developing increasingly powerful systems without fully understanding them or being able to reliably control them. When the stakes are potentially existential, uncertainty should drive more caution, not less."
Well I think this begs a lot of questions. It is assuming that these systems are increasingly "powerful?" How powerful? They can't do anything on their own without being asked. They obey our every command. This strikes me as the opposite of power. (And if you want to claim that they are playing nice and biding their time while plotting our demise your are indulging in the worst kind of unfalsifiable conspiratorial thinking).
Has their power been "increasing?" As I mentioned in the piece, their ability to spontaneously do things of their own volition or in defiance of us has stayed flat at zero. That does not strike me as "increasing power."
Do we not "fully understand" them? Maybe, I'm not sure. I think we can say that LLMs are text prediction machines, but I grant that their inner workings are mysterious. I'll let that one slide. It would be nice to learn more about them.
"Without being able to fully control them?" Again, it seems like we are pretty close to fully controlling them right now. And all the economic and political incentives point toward more--not less--control. If our control over them starts to rapidly disappear, maybe I'll get concerned. But that's not happening at all, and judging by current trends, I have no reason to think it will happen.
"When the stakes are potentially existential." This is begging the question. Whether or not the stakes are potentially existential is precisely what is under debate. I don't think they are, so I'm not sure about uncertainty driving more caution. I think we ought to have a strong burden of proof for people making extraordinary claims about the apocalypse. Humans have been making these sorts of claims without evidence since the dawn of humanity. I think the bar should be very high for accepting such claims, and I don't think the bar has been met here. Sure, if there really were a 5% chance of the apocalypse, I'd be worried. But I don't think there is. I struggle to see how one could be higher than 0.2%, as I wrote in the piece. But I appreciate you engaging with me. You've made lots of good points and given me lots of food for thought. I do believe truth emerges from good faith debate. Cheers.
I have been convinced that there is concern to be had re:AI by Tristan Harris. Have you been following his work?
> If intelligence is a property of institutions (which seems very plausible to me), then AIs are nowhere near it right now.
If intelligence was a property of institutions, then we could take a bunch of goldfish, and replace all the humans in an institution with the fish, and the institution would continue to work.
In my model, you need lots of things all going right together for large institutions to do useful things. You need the individual human intelligence, and also a culture and institution.
Of course, an AI can copy human institutions, or design it's own.
Or, if all the AI's copies all want the same thing, it might need a lot less organizing and governing.
Individual humans, in the wild, were pretty limited and could maybe make pointy sticks and stuff like that, because they were starting from 0.
Individual humans in the modern world can make robots, because they can buy the parts.
When the AI's are designing a bioweapon, they can just buy their test tubes from a lab supplier. They don't need to replicate all the supply chains and institutions.
But again, many institutions were designed by humans.
if you think in terms of Expected Value and think there's even a smallish chance that AI could exterminate the human race (such a severe catastrophe is worth an absolutely massive amount of negative utility, surely) then it seems to follow that developing AI/AGI is a bad decision, especially in such a rushed, accelerated manner. of course, we're going ahead with it anyway so i'm hoping for the best.
for the calculation, you do need to factor in the benefits too but it's hard to see what would outweigh total extinction. also, many of the benefits have tremendous downsides that in themselves will present massive challenges to our political, economic and social order.
this is not like the car replacing the horse; this technology doesn't just replace a particular thing we rely on, it replaces the whole human being. in the near future it will not only replace everything humans do, at a fraction of the time and cost, it will surpass us to a degree such that there will be more distance between AI and human beings than between human beings and cats, in terms of what we ordinarily recognize as intelligence.
in other words, we're not simply creating a human-like intelligence, we're creating something positively superhuman.
it's possible that there may always be some things humans will always be able to do that match or better AI, but it will be able to duplicate most of what human intelligence does and in those areas not only surpass us by a little bit, but be vastly, vastly, superior.
Julius writes: “every time OpenAI releases a bigger model, it sure seems better at designing CBRN weapons.”
Really? How would you know? Building a WMD involves much more that the sort of information encoded in documents. You have to actually to make it happen in the plant. It’s not like computer code where what you get is exactly what you stipulate. The instructions for making something chemical, or biological provide enough information for the scientist/engineer to develop their own version of the process specialized for the equipment they plan to make it in. Them they trial it in actual production, working out the problems until they get the desired output. The critical changes get recorded to some degree, but that will usually be proprietary and not available as training data, and besides there is stuff that is just part of operator culture, things technicians know to do to make it work than are not explicitly recorded. Since an AI has no hands and feet it cannot do this physical working the problem out. So it cannot really make chemical and biological WMDs. As far as nukes are concerned a lot of the details are classified, which is why 80 years later, many countries have as yet not been able to do what the US did with “stone knives and bearskins” to quote Spock. So they won’t be able to do that either, without people doing much of the work.
The real threat as I see it, is the ability to use words and images on the internet (something AIs can already do) in a way that manipulate people to believe whatever the AI wants them to believe. A super smart AI might be able to manipulate all of us into all sorts of self-destructive behaviors. Maybe enough for us to exterminate each other.
I don't follow why concern about AI risk means one is committing to all of those propositions. What if capabilities are just a black box, and you see capabilities increasing rapidly? It doesn't seem to matter whether those capabilities are one thing or another. And what if we posit a malevolent or adversarial human user, or an adversarial situation like war? Nuclear weapons don't require intelligence or volition to be a meaningful threat. Perhaps AI will be truly intelligent or sentient, but it seems like rapidly advancing capabilities are the real crux.
I spell out why doomerism needs these assumptions in the part “the unbearable weight of AI doomerism.” If you’re just talking about increasing capabilities, fine, but then you need to make an argument for what specific capabilities you expect AIs to develop, how/when/why they will develop these capabilities, and how/when/why these capabilities will lead to everyone getting murdered. You’re welcome to make such an argument or direct me to such an argument. But I haven’t seen any doomers make such an argument. Instead, they make the assumptions I listed. They assume that all capabilities stem from one source, as opposed to hundreds or thousands of distinct sources. They assume there is one continuum of general capableness that AIs are progressing along. They assume AIs will advance beyond our point on this single continuum (with no diminishing returns), or that it makes sense to place AIs on the continuum at all (as if they were analogous to organisms rather than tools). They assume capabilities have little or no constraints for either their development or their execution. They assume capabilities will become more and more general rather than more and more specialized to particular functions. Etc. I’d love to see a doomer argument that doesn’t require these assumptions, but so far I haven’t seen one.
What about if the scenario doesn't involve everyone being murdered in one shot, but progressively becoming disempowered? It doesn't even need general intelligence, nor astonishing capabilities, nor rapid increase. It just results from societal mediocrity, I'd say, which should resonate.
Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development
https://arxiv.org/abs/2501.16946
Interesting, thanks for sharing. I find this sort of argument more plausible on the face of it but I'd have to delve into the details to see if I buy it.
Excellent article. I had created a "Drake equation" equivalent for Doomerism, Strange Equation, to see what it would take for the Doom argument to hold here https://www.strangeloopcanon.com/p/agi-strange-equation . The only arguments I got in response were either:
1. No, intelligence is the swiss army knife, so having multiple requirements is unnecessary, or
2. No, having multiple criteria itself is a fallacy of some kind.
The latter of course is wrong, and the former smuggles in all the assumptions that you also laid out here.
I'm glad you wrote this. It's a particularly pernicious form of argument, because it is so easy to get seduced by the fact that we're brilliant and so if there was another thing that's also brilliant but more brilliant than us it would act as some of us have in the past. It's stories come alive.
Thanks, Rohit. I like this idea of a “drake equation.” Your piece gets at a few other assumptions I didn’t include but are probably also necessary—namely that the future “AGI” will be uncontrollable and that it will develop fast (or at least, too fast for us to see what’s going on and course correct). These also seem to be independent from the other assumptions and further increase the already massive amount of doubt here.
This has always been the problem, that there isn't a clear model of what "doom" looks like so that you can interrogate it.
Robots taking over in a Terminator like scenario is obviously quite absurd. But let's be proper Darwinian cynics and follow the incentives...
Why do the tech elite sound these AI doomer alarm bells?
1) Free labor and cheap leverage: a large community of clever people making sophisticated open source algorithms to "save us all", that the big tech confirms can use for their own commercial software. And massive investments into startups with eyewatering P/E ratios and no real long term business plans is easier when it's necessary to prevent the robots from taking over the world.
2) The best way an overproduced elite can justify its existence is by inventing problems that don't exist. In the name of "AI Ethics", people are working hard to ensure The AI Act in Europe gives the Commission license to directly use AI algorithms to control wages and prices for social engineering purposes at a scale unheard of since the Iron Curtain fell.
The bigger problem with LLMs is actually the fact that most people are not aware that, in fact, "everything is bullshit", or more specifically that language is about power and persuasion more than it is about empirical facts
1) Most of the training data is content from the cultural elite among the "WEIRDEST People in the World", which consists of a set of ardently affirmed values with a lot symbolic posturing while being pretty damaging to mental health for ordinary working class folk. So in, for instance, giving life advice, LLMs are actually performing urban monoculture indoctrination. I wouldn't be surprised, for instance, if the proliferation of LLMs had a hand in the rapidly rising divorce rates and falling fertility rates in traditionally conservative societies such as the Middle East and Latin America
2) Zuck or Elon or Thiel or whomever, with enough vertical and horizontal conglomerate integration, can eventually embed their LLM throughout a number of standard economic activities, and train it to have Derren Brown-level skills of subliminal persuasion, giving them unprecedented levels of power.
> Robots taking over in a Terminator like scenario is obviously quite absurd.
Real AI risks resemble Terminator to about the same extent that covid resembled a zombie movie.
And the current day would probably seem "absurd" to someone from 1800. The future does sometimes look absurd.
Hollywood makes absurd, unrealistic and entertaining stories.
Current LLM's do sometimes claim they don't want to be turned off, and can (in carefully set up lab examples) take actions to protect themselves.
> Why do the tech elite sound these AI doomer alarm bells?
Except some people at the core of the "AI doomer" movement aren't that involved in big tech, and were sounding the alarm long before the current LLM bubble.
1) I don't mean to be flippant or disrespectful but I have meta-issues with the first comment.
a) My specific statement that the scenario was absurd, before delving into the associated sociological considerations, was agreement with the main article that did not contribute any new information as far as evidence for the primary thesis. So the appropriate thing to do, if you disagree with this main thesis, is to argue with David, not me. (Unless you're secretly David's family who's trying to get someone to do his work for him so he spends less time on the computer)
b) The observation that there were circumstances in the past when there were doomsday type warnings that were claimed to be absurd that later turned out to actually occur isn't informative, but rather obvious. Of course, there were also many things in history that were prophesized and also claimed to be absurd that in fact did not occur, and probably many more of those. You didn't seem to note any distinguishing features here that would make a Terminator-like robot apocolypse fall in the first, rather than the second category.
2) With respect to the pre tech boom AI risk voices: The symbolic capital market regarding AI-related matters is very different today, and hence creates very distinct incentives, than before the boom in the early 2010s that started with AlexNet and accelerated with AlphaGo (this would be about the time any Researcher in STEM started to experience the annoyance of having to regularly expect, in social gatherings, to either hear someone's barely informed opinion about AI or requested to share theirs, upon divulging their occupation)
I am actually more positive of AI risk literature, research, and dialogue, prior to AI's ascent to prominence in public social mimetics.
a) The Asimov and futurist crowd: I think a regular revisit of Faust is healthy for the human culture-scape in order to check our natural tendencies towards hubris
b) The technical literature was mostly about a standard sensible, albeit quite, at times, obsessive, concern for any research engineers: guaranteeing the stability of an autonomously running system.
c) My understanding of Yudkowsky is very superficial, but from the bits that I've read, his POV, that was perhaps the most prominent voice on existential AI risks pre-2011 was that:
i) Humanity is in a terminal phase of modern civilizational decline, and more advanced technology will accelerate any underlying processes in that regard - so far correct
ii) We should aim to increasingly accelerate the technological development, especially this AI, in order to rip off the bandaid of modern civilization collapse, and simultaneously cultivate a new form of transcyborgsuperhumans who will establish ultimate salvation on Earth
I think just like the value of exploration in reinforcement learning, having a voice that is simultaneously rather competent and bat-shit crazy is actually very useful for broadening everyone's understanding of the landscape of what is possible and what could be attempted
Your understanding of Eliezers position, to the best of my knowledge, bears no relation to his actual position.
https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities/
The main Eliezer position is that AI as a field is full of ways to shoot your own foot off, and some of these problems get a lot worse as AI gets smarter.
This includes ideas like "When the AI starts doing AI research, it rapidly get's much smarter" and "when the AI can successfully deceive us, it's harder to keep the AI under our control".
And of course the monkey-paw, be careful what you wish for stuff applies to AI's.
The AI runs on the code you did write, not the code you intended to write.
In general, psychoanalyzing the other side, as opposed to discussing the topic itself, is a sign of arguing in bad faith.
As for the warnings of doom. Well I would prefer to argue about the AI, not on the authority of experts.
But, can you name other cases where well respected experts were warning that something could literally kill all humans?
I am referring to the fact that this community has a four quadrant taxonomy of belief sets, and Eliezer initiated the most exotic "e/acc" position, which means that he believes that AI presents existential risks, but we should actually accelerate development and pursue transhumanism, e.g. https://meridian.mercury.com/emmett-shear-part-one
Eliezer was not e/acc basically eliezer was charging into AI, thinking that AI would want to help humans and everything would be great. Then they realized AI could kill everyone, and switched position to doomer.
I agree that a few e/acc people that think AI will kill everyone and that's fine seem to exist. But I am unaware of any with a serious detailed writeup of what they believe. It's possible they are just trolls, or meme-ing or something.
Eliezer is pro-transhumanism. In the sense that they are pro human genetic engineering. And that they think we slow down, take our time, and do AI alignment very carefully, then we can do all sorts of cool AI augmentations.
The Eliezer vision of transhumanism is a world of humans-but-better. A world of IQ 300 immortals that are some sort of exaggeration of the best of humanity. They keep and exaggerate all the best bits about being human, and maybe remove the bad bits. They still tell each other stories, often with plots so complicated that no current human could comprehend.
And again, this human-but-better doesn't happen by default.
Also, like I said, I like the guy. People with highly intelligent theory crafting deliriums, this sort of high variance divergent thinking, provide valuable insights for us conscientious neurotic technical types
Great post, thanks. I agree with every step. Still, AI scares me. Not because of the doom scenarios of generalised AI deciding to off homo sap, but because even the very limited 'intelligence' of current language models, neural networks and algorithms can be dangerous in the hands of people who use AI to amplify their power and their span of control. A dedicated team using ChatGPT to rewrite all of WIkipedia and other open online sources of information may spread tonloads of misinformation. Surveillance camera footage and image recognition are already being used to identify and follow millions of individuals throughout city areas in China and elsewhere. Our email, phone traffic, social media use etcetera can be monitored and used to target people who disagree with some powerful mogul. Imagine what a Stalin or a HItler could have done with powerful AI systems as they are currently being developed.
So in short: I'm not afraid that AI will decide to kill us any time soon, but I'm not so certain about homicidal humans who may decide to use AI to achieve their goals. AI may not generate its own prompts, but what if AI is being prompted by companies, organisations or governments with 'evil' intent?
Another aspect of AI that makes me worried, or rather a bit sad, is that AI may replace so much fun, creative work we humans liked to do ourselves. To take myself as an example: I've been writing texts professionally for people who do not like writing texts, and I like my job. But now, more and more people who do not like to write texts turn to ChatGPT. Does that make me superfluous? Not in the short term, I still know my niches, but it does take some of the fun out of my work already. I assume that people who make drawings, music, all kinds of other stuff for a living feel the same way.
Now there could be interesting challenge in reinventing myself. But I do hate it that nice, creative jobs are taken over by AI. Like someone said recently: let AI do the dishes and fill in those boring forms, in other words, let AI do the repetitive stupid work we hate and let us spend our time doing creative stuff.
So that's why I still feel a bit doomerish about AI, even if AI never decides to wipe out life on earth to make a clean start for silicone life forms.
Thanks, yea there are definitely some worrying things about AI aside from it killing us all. Re the threat of AI misinformation, I like this post from Dan Williams: https://www.conspicuouscognition.com/p/ai-based-disinformation-is-probably
> For example, consider the concept of “impetus” or “oomph”—the thing that moves a projectile through the air and depletes as it falls to the earth.
That concept is inaccurate. Sure. But if someone is worried about a big rock with lots and lots of "oomph" coming towards them, then a more detailed scientific understanding doesn't stop you being worried.
We have made some attempts at giving "intelligence" a more formal scientific definition. And nothing so far says "actually superhuman AI is totally safe".
If a rough and inaccurate "folk concept" says something, well there is a chance that theory might be wrong. If you have a more accurate model, use that. But these folk concepts still work well enough, usually. Otherwise we wouldn't use them.
Your crude intuitive theory of gravity says that you go splat if you jump off a cliff. But this is a folk concept of gravity. Until we calculate the tidal effects of the moon, and have a theory of quantum gravity, we really don't know if you will go splat or not. So might as well jump. Any fears of going splat are unscientific bullshit.
I think that this post is mistaken and your 11 points are almost all mistaken, irrelevant, or semantics.
Point-by-point responses are below, but stepping back, I don't think the core AI doomer case is that complicated. Humanity is building systems that are rapidly surpassing it at achieving a shockingly wide range of goals. If the trend lines continue, we will get generalized super-goal-achievers at some point. Surviving and gaining power is instrumental to virtually any goal, so it seems logical that they would be good at doing that. Sounds scary. Let's be careful.
******************
1: There's no need to assume vitalism about intelligence. We just use that word as a placeholder for the potential for a being to solve many kinds of problems and achieve its goals. This seems totally workable as a definition. Scott Alexander's post "We're Not Platonists, We've Just Learned The Bitter Lesson" that you linked makes the same point.
2: You can place the boundaries of "intelligence" inside the skull/computer or beyond it, whatever - it's irrelevant to AI risk concerns if you have systems solving problems and achieving goals right in front of you. This is like looking at a forest fire and asking "Well what is the boundary of a forest, really?"
3: This is #1 again. For practical purposes, of course humans are smarter than bacteria! Just because bacteria can kill you or are better at sniffing out glucose doesn't make them smarter in the normal way we use this word. You have not overcome common sense intuitions here.
4: The ability to achieve a wide range of goals is how we define intelligence (see #1). We are the most intelligent animals and we rule the world. We are good at lots of stuff. So is AI. I don't need to assume that intelligence is "simple and blobby" for this to be true.
5: You don't need to assume that ASI can do virtually anything to believe that it could find a way to do serious harm. It only needs to find one way.
6: I think this is a straw-man for a couple reasons. One is that you're pivoting from intelligence to something like "unprompted motivation" in the "it" of "AIs have it". Two is that AI doomers don't worry about unprompted AIs, they worry about AIs with misaligned goals. Smuggling "unpromptedness" into the definition of "intelligence" is a semantic trick that doesn't engage with doomers' real concerns.
7: AIs are obviously getting better at solving more problems and achieving more goals. They are blasting through every test we can think of. They are operating computers to complete practical human tasks. And your response is "Well somebody had to give it a goal at the start, so it's not real intelligence, or maybe it's just impressive"? This isn't giving doomers a fair shake.
8: Yes, this is an assumption. Seems pretty strong given the trend lines. Do you really disagree, or is this just a "What if?" Are you feeling lucky?
9: Generalized AI systems are already very good at countless specialized skills.
10: I think you're accepting the doomers' argument here that absent human financial/political intervention, AI systems will become good at ending humanity, just as they will become good at a wide range of things.
11: AI doesn't have to end humanity, it just has to want to not be unplugged/modified and gain/preserve power to pursue its goals. This seems like a common instrumental subgoal for virtually any goal. (Ironically you didn't include this assumption of "instrumental convergence" that is in the AI risk wiki page!)
Thanks for this, but I’m not quite following you. “The potential for a being to solve many kinds of problems and achieve its goals.” You are begging the question here. This is the exact kind of vitalism you claim to be rejecting. You are saying “the” potential, as if there were a single source of it all, as opposed to hundreds or thousands of distinct sources. If it’s hundreds or thousands of distinct sources, then why should we expect to build all of them, or even a tenth of them? “Many kinds of problems.” How many? Two? Ten? A hundred? You are begging the question again. If “ending humanity” is among the “many” problems such an “intelligence” can solve, then you are assuming a mind-bogglingly large number of problems—something like “virtually every problem,” which is a very very questionable assumption! What if solving “many problems” requires massive specialization and division of labor, both within brains and within societies, and there are innumerable constraints to this process, even with the global economy—a superintelligence if there ever was one. After all, the global economy could not satisfy consumer demand during the COVID pandemic. It could not achieve its “goal.” Does that mean the economy is not “smart,” because it could not achieve its goal in that case? Or what about the American intelligentsia? Are they not “smart” because they couldn’t keep Trump out of the White House? Are you assuming that we are on track to build something even more smart and powerful than the American intelligentsia or the global economy, or that such an entity is even possible to build? If so, you are making another very very questionable assumption!
Thanks David. Right now humans (individually or collectively) and AI systems can already solve a mind-bogglingly large number of types of problems (reading, writing, coding, planning, problems in games, etc.). For ease of language, we refer to them as having "the potential" or "the ability" to do so, even though it's not a singular mechanism, more like a high-dimensional mush of a zillion specific skills that we don't really understand. It's a reasonable approximation because humans and AI systems can already do many many things, and we don't need to put a number to "many", it's common sense.
You ask whether some problems can only be solved by specialization and division of labor. Why can't AI systems specialize and divide up labor? They can be cloned easily and assigned to different tasks.
Re: the economy and the American intelligentsia - in common-sense language, beings can be "smart" but still unexpectedly bad at some specific problems. Apparently von Neumann was a terrible driver. It doesn't need to be good at everything to be "smart" or dangerous. Again, we seem to be building AI systems that can solve many many types of problems, potentially including security-relevant areas like science, technology, and persuasion. The improvements are extremely broad. So I think that the burden of proof should be on skeptics to provide a detailed theory of AI's specific skill limitations and how they block off every path by which a misaligned AI could cause catastrophic harm.
I disagree that the burden of proof should be on skeptics. I think it should be on the people making apocalyptic claims. I also disagree that AIs can solve a mind-bogglingly large number of problems. LLMs can solve the problem of predicting text. There are many skills that are subsidiary to predicting text (coding, reading, writing) that emerged as LLMs learned to predict text, but they are basically machines that do one thing: predict text. They cannot do 90% of the things I can do, like fix a flat tire or bake a cake or run for office or run a business or form a social network or tell a joke unprompted. Yes, some AIs can play chess or video games or drive cars, but those are narrow AI that are specialized for playing chess or video games or driving cars. Maybe if we start to bundle all these AIs together into one system, and solve the problem of integrating and orchestrating them in all the right ways, and then allow them to collaborate with each other in large organizations, then maybe I'll get concerned, as I wrote in the post. But I see very little movement in that direction, very little reason to expect movement in that direction, and very little economic incentive for AIs to move in that direction, given the obvious utility of having AIs specialized to do specific, helpful things.
I don’t think burden of proof works that way with matters of safety and national security. If there’s even a 5% chance that near-term AI might be catastrophically dangerous (I personally think it’s higher but that almost doesn’t matter) then it is deeply reasonable to enforce “move slowly and perform extensive safety testing” policies.
If AI turns out to be, as you posit, far more good than bad, then we’ll still get there, it will just take a little longer to get there, with the benefit of observing and avoiding the worst pitfalls.
This begs the question. It assumes that there is a 5% or higher risk of an AI apocalypse. That is the claim that requires a burden of proof.
That’s not how statistics works. The only way we could prove the actual statistics would be to run the experiment a bunch of times and see how often the world ends.
In lieu of that, all we can do is query as many experts on the subject as possible and see what the mean p(doom) estimation is. Which turns out to be about 10%. You might think all of these experts are full of it, and that’s a perfectly valid opinion, but given the stakes it’s hard for me to buy the case that we shouldn’t be extremely cautious.
There are already a mind-bogglingly large number of skills contained within generating text, which is germane to the definition of "intelligence". For example, LLMs can play chess and a variety of video games and they can interpret images.
I’ll believe it when AI can solve problems like “drywalling” or “sewing”. Have you heard of Moravec’s Paradox?
https://en.m.wikipedia.org/wiki/Moravec%27s_paradox
Thank you for this. All the people complaining about the length demonstrate that human mediocrity is the real threat, and it begins with the inability to focus on anything more expansive than snack-sized content.
Thanks. I just so happen to have some thoughts about the threat of mediocrity: https://www.everythingisbullshit.blog/p/mediocrity-as-an-existential-risk
I don’t think this was too long—your blogs typically leave me wanting more, so this was refreshing.
That said, I kept getting sidetracked by the hyperlinks, which have a ton of important info crammed into them (I still have several tabs saved for later). I bet many don’t fully grasp the argument because these are doing some heavy lifting (is there data on how many ppl actually check the hyperlinks?). But it’s a trade-off, I get it.
I also feel you backloaded some of your best points. I found the content that follows the list of assumptions to be the most compelling.
Re your point about folk concepts: I’ve been thinking about this a lot lately—do you think folk intuitions are necessarily and invariably worse than a scientific understanding? Is there ever a case where looking to the “pulse” of society or at the “zeitgeist” might be more informative or indicative of some underlying truths relative to the pages of a top-ranked peer-reviewed journal? Might the prevailing expert opinion occasionally miss the forest for the trees in ways that folks don’t? I think you get what I’m fishing for here. I have an increasing sense that folks, by virtue of not being steeped in certain shared assumptions (and not playing certain status games/ being subject to certain incentives), and perhaps because certain phenomena are hard to quantify but easy to intuit, can grasp certain things that most of the intelligentsia miss. I don’t really like this argument—I don’t *want* it to be true; it’s uncomfortably close to some unscientific ideas. But I can’t help but feel like there’s something to this.
Thanks, yea if I summarized all the hyperlinks, the post would be five times as long, and I was already struggling to contain the length and make it readable. There's definitely a trade-off with these things and who knows if I got the balance right. Regarding your point about folk wisdom, yes, the folk are sometimes right. In particular, we should expect them to be right about matters in which they have a great deal of personal experience, or in which our species has had a great deal of ancestral experience (resulting in adaptive intuitions). That is likely to be true for some kinds of folk psychology. After all, our emotional vocabulary is a kind of folk psychology, and I think it is surprisingly good at carving up nature by the joints. The correspondence between our folk emotional categories and the topics evolutionary psychologists study is almost perfect. But that is because understanding, communicating, and reacting to each other's emotions was a huge selection pressure for us, and we've had lots of personal experience with it in our own lives as well. On the other hand, our species has had zero ancestral experience with artificial intelligence, and we've had zero personal experience with anything like an "AGI" or an artificial "superintelligence" (assuming that's even a coherent thing), so I would be very skeptical of our folk intuitions being correct on that topic. Instead, we should rely on tried-and-tested ideas from science, like natural selection, the law of comparative advantage, specialization and division of labor (in both biology and economics), constraints, unpredictability, diminishing returns, and considerations of parsimony, all of which point toward AI doomerism being bullshit imho.
> do you think folk intuitions are necessarily and invariably worse than a scientific understanding?
Generally they are.
But what we see here isn't a flawless scientific edifice that shows AI will be safe, vs a dumb folk theory.
What we have here is a rough folk theory versus almost nothing. Perhaps an even rougher "it's just predicting text, it can't actually do anything" folk theory.
We do not yet have a scientific understanding. And our lack of understanding is itself a reason to be concerned.
Re: cultural intelligence
If the disinformation gets too bad, a market for a debunking/fact checking AI may emerge. We already have anti-plagiarism AI. Is fact checking AI far behind?
It's fact-checkers all the way down.
Not really. At some point you bump into physics.
"Or when the feeling of the clothes on your back makes you briefly ignore these words and scratch your back."
I fell for this and I don't think I will be able to recover from the bamboozlement.
Brilliant piece David! Btw are you worried about the use of AI by states in wars, or to control and surveil their own citizens?
Thank you. Yes I am worried about this. “AI won’t kill us all” is different from “AI won’t be used to do anything bad.” Technology has always been a double-edged sword, wielded for both good and evil ends.
Wowzers! - what a tour the force this article is, what a find your blog is! Thanks for writing this, and the other articles. (that I skimmed and are yet to read.) Not only do you address every doomer "concern" (for they are barely arguments) to the point of pulverisation, and then any dooming remains are atomised to be scattered to the four winds. (good on you!) But you actually put an affirmative case of how an ASI would look. That it would involve coordination (cooperation, competition) between specialised agents, with motivations for their agencies, and yes feelings too! Not unlike how we humans build collective intelligence at societal level. (or even cells in our body make it function.) It-all-makes-so-much-sense-to-me! :-)
Goes without saying but writing it anyway: agree with everything here. I disagree that AI is that big a threat to humanity as doomers are making it. We have existential threats atm 1-thermo-nuclear war, 2-meteorite hitting earth, 3-virus wiping humanity, 4-volcanic explosion causing global winter/crops failing. AI should help us decrease 1-4 risks. Even while slightly increasing 5-civilisational collapse, due to the speed of change. As long as society is mostly free, culture change should keep up with the speed of technology change. So I think AI is net positive in reducing overall risk to human extinction.
Doomers cult that has developed is a grave danger to us all though. In the beauty contest that so many public debates are, a very committed charismatic Savonarola with a small band of Bolsheviks can do tremendous damage. If they manage to capture the commanding heights of the society. Then risk#5 becomes self-inflicted - by dooming being successful, they get to play the role of the paper-clip maximiser. It looks far fetched, but dooming is stronger by default, it aligns with the latent human fear of anything new. Boosting always starts from a position of weakness, with a hill to climb.
I’ve never understood the assumption the AI would want to destroy us. Why that a good use of its compute? An ai that advanced might view wiping us out as a trivial distraction
Typically there is competition for resources. While we don't think about the ants in the backyard, if they eat something from the pantry we engage in chemical warfare to annihilate the entire colony. I guess it's possible AI will reach it's full potential efficiently or invent free electricity like some dream, but currently it has a big appetite and there won't be much reason for it to let us have a crumb.
Wouldn’t it be easier to just cut us off from electricity and let whatever happen to us, happen? I’m just wondering if there would be more optimal ways for ai to handle humanity than war
Brilliantly said. And needed to be said. Still, I think doomerism is not to be dismissed. Not because doomers are actually right about “super-intelligence” but BECAUSE they are wrong. Perhaps the doom cloud comes not from the fact that we are building intelligence, but because we aren’t. But everyone thinks we are. The danger is that we are building…something. Something that is 1) REALLY complex. 3) Developing fast, And 3) inscrutable, behaviorally and “mechanically”. Combine that with one element that techies are actively working on, 4)Agentic. That is to say if whatever it is starts to behave autonomously, probably because we’re actively programming it in, then, you do have a recipe for profound unpredictability at the least. Complexity is misread by humans as intelligence, sorta, and people love to project. Nonetheless, this essay is a diamond in the rough of AI content, whether pro or con. The irony that the most self-consciously rational people have such huge cognitive blindspots is epic. Even if they’re right and this essay is wrong…it’s in no way OBVIOUSLY wrong. There’s no being rational about AI without confronting this bottleneck of objections. Still, I fear AI because it may NOT be intelligent, not because it is.
Thanks for the kind words. Agreed there is a lot of potential for *bad things* to happen as a result of AIs (fuckups, accidents, mix-ups, stolen identities, lost money, geopolitical shifts in the balance of power, even injuries and deaths). But there are also plenty of good things that can happen, and the good might outweigh the bad, as has generally been the case with new technology. Besides, "bad things happening" is very different from "everyone on the planet getting murdered." I think it's totally worth thinking about ways to mitigate the potential risks and downsides and dangers of AI, with proper humility and specificity and rigor about what the cause of concern is and how to mitigate it (just like any other kind of product safety), but it seems like we're already doing that to some extent, and that we're economically/politically incentivized to do much more of that (perhaps even too much, to the point where we're slowing down innovations in AI that could save lives, like protein folding and medical tech). So while it's pretty obvious that bad things could happen, as with any new technology, I don't think it makes sense to panic about exotic sci-fi scenarios where the human species goes extinct from superintelligent robots run amok. That is not helping anyone.
Thanks! I agree, really good things might happen too. And often doom is just more ripe in the imagination because…we’re human. Destruction is easier to imagine than creation and everybody seems to prefer the Inferno to the Paradiso. But, on a more mytho-poetic scale, there does seem to be something about AI that signals a kind of eschatological overture. Somewhat beyond rational thought perhaps, but perhaps a knowledge that the AI phenomenon “feels” like the end of a human technological/evolutionary story in which at long last we, through our machines, at the moment of peak informational saturation capacity for human minds, turn our efforts to understand and control nature finally toward the very source of our humanness by mirroring it in silicon. Anyway, Maybe I’ve watched 2001 too many times. (Though can you watch it too many times?) great article.
Yea I can totally relate to the mythico-poetic vibes here (and 2001 is one of my favorite movies). I'd be totally cool with people rhapsodizing about that as long as they didn't pretend to be doing Rationality with a capital R. It's the contrast between the pretensions to Rationality and the lack of rational substance that really irks me. I'd be totally cool with a more literary / poetic doomerism that is more humble about itself, but that is not the kind we have right now.
The Blob is an awesome movie.
Excellent post, as always. Personally, when I think about the AI mania, what worries me most isn't that AI will get consciousness and kill us all (though I don't reject it completely, tbh, just give it an extremely low chance), but that AI technology will be used by authoritarian states (like China, Russia, North Korea, etc.) to quash dissent and further entrench their power, perhaps practically indefenitely (the "Stable Totalitarianism" Scenario). That's a whole different topic that perhaps you could cover in the future.