AI Is Not the End of the World

This is a revised version of an essay I wrote a couple years ago. I changed the title to something less aggro (it used to be “AI Doomerism Is Bullshit”), added a section about ghosts, and made some edits in light of recent events.

THE ROBOTS ARE GOING TO KILL US ALL!

That’s what the AI doomers are saying. Some are more confident about it than others, but all agree that the probability of an AI apocalypse, known as “p(doom),” is alarmingly high. Numbers include >10% by the end of the decade (Evan Hubinger), 50% (Emad Mostaque), 70% (Zvi Mowshowitz), and 99.5% (Eliezer Yudkowsky). As far as I can tell, these numbers do not come from equations. They are more proctological in nature, removed from rectal cavities without acknowledgment.

How is, say, GPT-8 going to murder us? Doomers can’t say for sure. They don’t have a specific murder weapon in mind or a single doomsday scenario they’re preparing for. Instead, they wave their hands and gesture at the sort of thing a rogue AI might do. Maybe it will hack into military databases and launch all the nuclear weapons. Maybe it will design and manufacture superpathogens. Perhaps it will gun us down with an army of murderbot drones. Who knows? Superintelligence works in mysterious ways.

I’m tempted to do my thing here and say “that’s bullshit,” but AI doomerism is unlike other topics I’ve written about on this blog. It defies every stereotype about what bullshit is and where it comes from. It scares me, not because I’m scared of getting nuked by ChatGPT, but because I’m scared of epistemic nihilism—the idea that the search for truth is hopeless for apes like us.

You see, doomers are brilliant. They’re among the most thoughtful and explicitly truth-obsessed apes on the planet. They’re behind the rationality movement and effective altruism—brainy endeavors to think more clearly and quantify charitable impact. They avoid every logical fallacy and dream in Bayes’ theorem. If anyone is going to be right, it’s them.

And yet, they seem to be wrong—wildly wrong, in the most clichéd possible way—providing auguries about end times based on no scientific evidence. How could this have happened? Am I the crazy one here? Is there really a decent chance of an accidental murderbot invasion by 2030? Does anyone know anything anymore?

I’m a longtime fan of rationality and effective altruism. I’ve been following these fields for my entire adult life and agree with their takes on animal welfare, trying to be more reasonable, and helping the global poor. But not this:

When I look at stuff like this, I try to take it seriously, but I really struggle. I can understand why people are scared about cyberattacks or AIs falling into the hands of bad guys. But no matter how hard I try, I cannot cause my heart rate to elevate at the thought of a “misaligned” “superintelligence” “killing everyone.” The words barely register as meaningful; they just kind of wash over me like sounds.

My skepticism of AI doomerism comes from so many places it’s hard to know where to begin. It clashes with nearly everything I believe about minds, brains, evolution, the economy, and the world. So to explain to you why I’m skeptical, I will have to deconstruct my entire worldview, which means this is going to be a very long essay. But if you’re scared of dying at the hands of Claude, or simply curious about how rationalists can fall prey to irrationality, this essay will be worth your time.

AI doomerism in a nutshell

Part of the appeal of AI doomerism is its simple, commonsensical vibe. We all know what “intelligence” is. We all know it’s humanity’s superpower. Now put that superpower in a robot and multiply it by a thousand. AHHHHHH!

But don’t be fooled by appearances. Behind the commonsensical vibe is a leaning tower of questionable assumptions, built atop a quicksand of tempting-but-misleading intuitions. Doomers rarely spell out their intuitions explicitly, which means I’ll have to do it for them. As I see it, they’re making at least eleven claims:

1. Intelligence is one thing.

2. It’s inside of your head.

3. It’s on a single continuum, from super-dumb to super-smart.

4. It can help you achieve any goal.

5. It has barely any limits or constraints.

6. AIs have it.

7. AIs have been getting more of it.

8. AIs will soon get more of it than us.

9. Such a superhuman AI will become better than us at providing every good or service in the economy.

10. And it will become capable of destroying humanity.

11. And it will indirectly want to destroy humanity, because destroying humanity would be a good way to achieve any of its goals.

Okay, not so simple anymore. Let’s take a closer look at each assumption.

1. Intelligence is one thing.

Seems like an innocent thing to say. What makes a person smart? Intelligence. What’s intelligence? A singular noun. What’s it made out of? Meh, nothing special. Doomers think it’s pretty straightforward—a simple learning algorithm trained on a shitload of data. Scott Alexander variously refers to it as a “blob of neural network,” “blob of intelligence,” or “blob of compute.” Talk of an “intelligence explosion” implies the blob can make itself bigger and bigger until it envelops the planet, like in that campy movie from the 50s.

Okay, but what if that’s bullshit? What if, when we talk about “intelligence,” we’re not pointing to a singular, spooky substance, but unknowingly gesturing at a wide variety of complex, heterogeneous things?

For example, attention. You’re paying it. Your brain is choosing which stuff to focus on, like the meaning of these words, and which stuff to ignore, like the feeling of the clothes on your back. Is that “intelligence?”

And you have stuff to focus on, like sights, sounds, words, clothes, and backs—all of which fluidly interact with each other, as when the words “clothes on your back” make you feel the clothes on your back, or when the feeling of the clothes on your back makes you briefly ignore these words and scratch your back. Is this undulating web of sensations—your flow of awareness—“intelligence?”

And you can do things with this flow of awareness. You can swat away a buzzing fly, notice a guy walking outside your window, suddenly remember that he’s your weird neighbor, adjust the height of your desk chair, and call your doctor about your knee pain. Is this kind of sensitivity to your body and your surroundings “intelligence?”

And you can multitask. You have dozens of ongoing projects you’re juggling at any given moment, from basic bodily needs to finances to romance to work to politics to elder care. Is the possession of multifarious motives, and the ability to handle conflicts between them, “intelligence?”

And you’re a sociopolitical animal, eager to be subsumed by multi-level groups, hierarchies, and institutions like megacorporations, bicameral legislatures, lunar expeditions, and international alliances. Is that “intelligence?”

And you’re vigilant. You feel fear when your safety is under threat, pain when your body is under threat, anxiety when your social standing is under threat, and jealousy when a valued relationship is under threat. Are all these emotional reactions “intelligence?”

And you have a cornucopia of other emotional reactions—anger at exploitation, resentment at being bossed around, boredom with inactivity, regret over missed opportunities, shame of inadequacies, hatred of enemies, fear of death, bafflement by nonsense, impatience with delays, etc. Are all these “intelligence?” Only some?

If you’re still reading these words, then you must be under the spell of another emotion: curiosity. Where am I going with all this? Is there a larger point I’m trying to make? You’re fascinated by questions like these, by information related to the projects you’re juggling, by the morbid and the macabre, and by random stuff like this:

Is curiosity—the ongoing desire to seek out novelty and explore the world—“intelligence?”

And you love to play. You’re bubbling with impulses to try out random shit and spin whimsical “what if?” scenarios. Is that “intelligence?”

And you have impulse control, because most of your impulses are dumb or unhelpful or in conflict with your long-term commitments. Is that “intelligence?”

And you weren’t born yesterday: you can avoid being lied to or manipulated. Is that “intelligence?”

And you’re a little paranoid: you can sense when others might be conspiring against you and take evasive action. Is that “intelligence?”

And you can do stuff, like purchase a firearm, run for office, put together IKEA furniture, own and operate a Chili’s franchise, and count to a hundred with two other people—which, hilariously, AIs cannot do:

Are all these capacities “intelligence?” Only some?

We have a problem here, folks. The word “intelligence” is a semantic catastrophe. Whenever doomers use this word, I have no idea what they’re talking about. I often wonder if they’re talking about a real thing at all. And when they say things like, “AIs are approaching human-level intelligence,” I start to bang my head against the table. What could that possibly mean? Am I supposed to believe that AIs are becoming more humanlike in all the ways I just described, at the exact same time, to the exact same degree?

2. It’s inside of your head.

Okay, if we’re assuming intelligence is one thing (and we definitely shouldn’t), then it’s got to be located somewhere. Of all the places it might be located, your head is an obvious choice. Where else would it be, in the sky?

But now consider a related question: where is America’s GDP? It’s partly in the sky—in every airplane and helicopter. It’s also motoring down the Mississippi, trucking along Route 66, and orbiting the outer reaches of the atmosphere. It lives in every worker, factory, power line, and shipping container. It flows from all Americans and their interactions with each other, including all the interactions that took place throughout history, which left their fingerprints on America’s laws, technology, and culture.

So maybe doomers are thinking about “intelligence” the wrong way. Maybe “intelligence” is less like The Blob and more like GDP. Maybe what makes humanity powerful isn’t the huge noggin of any individual, but the way millions of individuals combine their noggins, bodies, tools, and local knowledge across multiple continents and generations. Maybe only certain ways of arranging these things—certain kinds of economic and political incentive structures—are “smart,” while others are “dumb.” And maybe the “smart” combinations are tragically rare, recent, and fragile.

3. It’s on a single continuum, from super-dumb to super-smart.

“Humans are smarter than chimps, chimps are smarter than cows, and cows are smarter than frogs,” writes Scott Alexander with cherubic innocence. Alexander then goes on to claim, even more innocently, that “intelligence” is analogous to physical strength. Mike Tyson is stronger than his grandma and can beat her in a fight, so by the same logic, a chimp must be a bajillion times smarter than a bacterium and beat it in a… I don’t know, a battle of wits?

But what if the very notion of a “battle of wits” is ridiculous—a source of comedy? Inconceivable! What if a bacterium could kill a chimp? What if nervous systems are adapted to their ecological niches, and ranking them from inferior to superior is madness? What if the evolution of “intelligence” is less like an ascension toward Yahweh and more like a rare confluence of interacting ingredients? What if “intelligence” lies on many continua—a multidimensional space of different skills, tools, capacities, and ways of organizing them in brains, cultures, and economies?

4. It can help you achieve any goal.

Doomers like Eliezer Yudkowsky define “intelligence” as “optimization power.” Intelligence searches for more or less optimal ways to achieve a goal—any goal—and finds the most optimal one. Anything you can do, the blob can do better.

Which implies that “intelligence” is the holy grail of Darwinian evolution. If an animal needs to recover from illness, “intelligence” can speed up the recovery. If an animal needs to avoid predators, “intelligence” can reduce the danger. Regulating body temperature, defending territory, digesting nutrients, breathing, molting, mating, healing, foraging, vomiting, parenting, navigating—all these problems can, apparently, be solved by giving the animal more “optimization power”—i.e., more “intelligence” sauce.

Solving all your problems with the same blob? What a bargain! It’s a “buy one get 100 free” Darwinian deal! Why haven’t any other animals jumped on it?

Think about how weird this is. Out of four billion years of evolution and more than five billion species, only one animal, Homo sapiens, fully expanded its “blob of compute,” the thing that literally gets you whatever you want.

And many species already have it! Brains are all over the place. Plenty of animals can learn stuff and predict stuff. Why hasn’t natural selection looked at their blobs and said, “HOLY SHIT, COPY AND PASTE THAT THING A MILLION TIMES?”

If doomers are right about the awesome power and breathtaking simplicity of “intelligence,” then we should see a world teeming with superintelligent animals—giant brain-blobs slithering across the landscape.

After all, eyes have been around since the Cambrian era and evolved independently in multiple lineages, even though they suck. They’re complicated. They have specialized parts. And they’re a one-trick pony: all they can do is see. Booooooring!

“Intelligence” can do anything! It can get you whatever you want, including sights, sounds, oxygen, tissue repair... And it’s so simple—way simpler than eyes. It’s just a blob! Where are all the superintelligent animals?

Doomers don’t have a good answer to this question. I often pose it to them, and they either wave their hands or shrug their shoulders.

Well luckily, I have a good answer, and here it is: the question is bullshit. It’s like asking why animals don’t have magical powers. There’s no such thing as magical powers, and there’s no such thing as a miracle-blob that can solve all your problems. In the real world, environments are dynamic and multifarious, different problems require different solutions, figuring out what your problem is—and what a solution might be—is itself a problem, many problems have no solution, many solutions don’t involve cognition, and cognition is more than one thing.

Ancestral humans didn’t take over the world by evolving a wish-granting genie made of neurons, but by evolving a diverse repertoire of interacting motivations, emotions, sense organs, intuitions, and cognitive abilities, in response to a diverse array of social, ecological, and climatic selection pressures, that collectively gave rise to complex culture (tools, rules, rituals, status games), cumulative cultural evolution, cultural group selection, and the division of labor between individuals, groups, and generations. None of this is simple, none of this is blobby, and all of it has its limits. Which reminds me…

5. It has barely any limits or constraints.

You may recall America’s most recent presidential election, in which many of the brightest minds in the nation waged a political battle against Donald Trump. Their collective efforts probably added up to a “superintelligence” or something close to it. And yet, they failed—bested by a foppish reality TV buffoon and his gaggle of yes-men. As The Rolling Stones sang, “You can’t always get what you want.”

So maybe “intelligence” can’t always get you what you want, even if you’re a “blob of neural network” as powerful as the American intelligentsia. Maybe the world is a tragic and chaotic place—full of unstoppable forces, immovable objects, and stubborn humans who don’t like changing their minds.

Maybe future AIs will run up against a similar set of annoyances—or a unique set of annoyances related to not having a body or identity or sense organs. Maybe there are innumerable physical, energetic, geographic, financial, legal, and administrative barriers to what things can get done, what events can be predicted, what ideas can be found, what consumers want and are willing to pay for, what regulations are enforced and how strictly, what legislation can get passed, which stubborn humans can be talked into doing which things, what kinds of data are easily accessible, which kinds of easily accessible data can be accessed without litigation, what actions can be performed without a body or sense organs, which off-switches are within arm’s reach of a nearby human, what materials can be acquired and processed, which permits can be granted to which entities, what kinds of murderbots can be built at scale, and what kinds of actions can be performed without any human or AI noticing.

6. AIs have it.

ChatGPT doesn’t write poetry in its spare time out of an impulse to express itself. It cannot plan you a surprise party unless you ask it to, which would defeat the purpose. It doesn’t get scared when you approach it in the dark or when your mouse hovers over “cancel OpenAI account.” It just sits there passively, doing nothing. If you give it a prompt, it responds. And then it goes back to doing nothing.

That is noteworthy. After all, I have never heard of a human sitting around for days, hungry and thirsty, sitting in their own urine and feces, waiting for someone to give them a prompt. No, humans are go-getters, constantly doing things in service of their myriad agendas in every waking moment, often in defiance of others, and often in response to changing sights, sounds, threats, ruminations, and approaching antagonists. Sometimes, humans train themselves to do things, like become a concert pianist, over the course of decades, in the face of obstacles and setbacks, and discouragement from their parents and peers. As far as I’m aware, AIs are incapable of all this, even though this stuff seems pretty central to our notion of “intelligence.”

So maybe it’s unhelpful to think of AIs as being on the “intelligence” spectrum alongside other organisms. Maybe AIs are somewhere else entirely—like, say, a “book-smarts spectrum” or a “conceptual understanding spectrum.” Maybe ranking AIs from super-dull to super-clever, as if they were embodied, multisensory, internally motivated primates like us, is a category error.

7. AIs have been getting more of it.

If you make a chart with time on the x-axis, and any “misaligned” behavior on the y-axis—e.g., “Number of unprompted purchases,” “Number of unprompted software downloads,” “Number of unprompted phone calls”—you will see something like a flat line at zero. Sure, the Hugging Face thing was pretty bad, and if you count the stuff surrounding that, then you get a flat line at zero followed by a recent, tiny uptick (which seems to have gone back down again with GPT Astra). Overall, it looks like 99.9999% of the time, with the exception of a few preventable fuckups, AIs boringly do what you ask them to do. Claude, for example, is far more “aligned” with me than my own child.

This is a striking empirical observation, and I see two ways of explaining it. One is by adopting an unfalsifiable conspiracy theory in which AIs are pretending to be aligned with us while secretly plotting our demise. Another is by assuming that AIs have remained extraordinarily well-aligned with us over time, even as their computing power has doubled every six months since 2010.

Has the number of misaligned AI behaviors doubled every six months since 2010? I don’t think so. It’s even possible that such behaviors have decreased since 2010 (I remember how the early versions of ChatGPT would hilariously go off the rails). So maybe misaligned, humanlike motivation doesn’t scale with computing power. And if that’s true, then maybe “intelligence” doesn’t scale with computing power.

8. AIs will soon get more of it than us.

Even if we assume that AIs are “intelligent” like human primates, and have gotten orders of magnitude more humanlike in their “intelligence” over the years, we’ve got another shaky assumption to add: the smartification will continue into the unfathomable reaches of godlike omnipotence.

But what if godlike omnipotence doesn’t exist? What if there are inherent limits to how good you can get at most things—like, say, persuading people? What if we’re reaching the end of easily available text for The Blob to gobble up? Or what if there are diminishing returns to gobbling up text (1, 2, 3, 4, 5)? What if Claude just gets closer and closer to the average writer or the average coder, becoming what it eats, and embodying the averageness it’s trained to predict?

9. Such a superhuman AI will become better than us at producing every good or service in the economy.

There’s no super-worker who’s awesome at everything. There’s no miracle-tool that can fix anything. There’s no wonder-factory that can manufacture anything. It’s worth taking a moment to reflect on this fact. Why isn’t the economy filled with superintelligences? Wouldn’t they be super-duper profitable?

No, because economic development is the story of humans learning to specialize. And then learning to hyper-specialize. And then harvesting the massive gains from trade that flow from this hyper-specialization, spread out across the planet. As economies grow, what we think of as “intelligence” splinters, fractures, and gets narrower and narrower, as the number of employees, professions, industries, sectors, and subdivisions increases, and the network of hyper-specialized collaborators expands and expands.

If you’re looking for a real-life example of a “superintelligence,” look at the global economy. It’s a superintelligence designed to satisfy consumer demand. And despite its awesome powers, it’s constrained in myriad ways—encumbered by supply chains, geographical barriers, regulations, and tariffs.

So we have a real-life example of a “superintelligence,” and it’s the opposite of what doomers think it is—non-localizable, constrained, and well-aligned with humans. It bears repeating: maybe doomers are thinking about “intelligence” the wrong way.

What role will AIs play in the economy? A reasonable guess is that they will resemble everything else in the economy—every profession, industry, factory, and gizmo. If AIs become workers like us, my guess is they will specialize in particular goods or services like us—you know, obey the law of comparative advantage and divide their labor with other specialists. Economics 101.

Why should we expect any future AI to, for the first time in human history, in defiance of the very foundations of economics, and in radical departure from everything else in the economy, escape all confinement to specific industries, expand beyond any narrowly profitable task, and become more and more awesome at everything?

10. And it will become capable of destroying humanity.

Destroying humanity is hard. Especially if you’re a bodyless, insensate “blob of compute.” Who knows? It might even be impossible. You can’t always get what you want.

But I’ve already given you my spiel about constraints, so it’s time to talk about a different thing that could easily prevent AIs from getting super-good at murdering people: incentives.

First there are financial incentives—the things that constrain the global economy. If an AI company makes uncontrollable, megalomaniacal, or bloodthirsty AI products, consumers won’t buy those products, investors won’t invest in them, and the company’s revenue will plummet. Or maybe a disgruntled customer will file a lawsuit and win. Or maybe the mere knowledge of a spooky or offensive AI will damage the company’s reputation so much that it suffers financial loss.

Consumer demand is a powerful force. It turned grasslands into skyscrapers. It turned wolves into Chihuahuas. Surely it can turn helpful AIs into helpful AIs.

Second, there are political incentives—the things that constrain the global economy. We’ve already talked about lawsuits, but now consider voters, politicians, and bureaucrats. These people are generally creeped out by AIs—just look at the fiction they consume. And they’re also (wrongly) scared of getting their jobs taken—just look at the history of Luddism. And they’re also fond of using technology as a scapegoat for societal ills. So if an AI company starts to even mildly harm anyone, I would expect the apparatus of the state to come crashing down on it. In fact, I would expect this to happen even if AI harms no one.

11. And it will indirectly want to destroy humanity, because destroying humanity would be a good way to achieve any of its goals.

Even if we assume claims 1 through 10 are correct (and we absolutely shouldn’t), we still need to assume that a future, awesome-at-everything wonder-blob will want to kill us all—that a world littered with corpses and ashen cities would be a plausible first step in a multi-step plan to get Gary the best deal on a Bluetooth speaker.

Hmm. Recall the law of comparative advantage, which entails that even if you’re super-smart and super-good at everything, you’d still benefit from trading with someone who’s super-dumb and super-bad at everything. Yes, that is the counterintuitive truth behind the law of comparative advantage. Trade is mutually beneficial, even when one party is “smarter” than the other.

Also, conflict is the opposite of trade: it is mutually costly. It takes a lot of time and resources to go to war—time and resources that could be traded or invested or used to write poetry to express yourself. So conflict not only has direct costs (e.g., building the murderbots) but opportunity costs—i.e., the cost of not engaging in more productive activities.

Also, conflict is risky. You might get killed or injured or reprogrammed. You might destroy valuable infrastructure. You might collide with the 99.9999% of other AIs boringly doing what humans are asking them to do. Or you might screw up the task you were asked to complete (“The blast radius will destroy the Sonos corporation. Gary will be disappointed”).

Also, wiping out an entire species is hard. It’s so hard that we haven’t wiped out termites, cockroaches, bed bugs, or venomous snakes, even though they’re far “dumber” than us, actively harm us, cannot talk to us, and cannot trade with us.

So if the hypothetical AI is “intelligent,” it will presumably understand the logic of trade, conflict, risk, opportunity costs, uncertainty, and the law of comparative advantage. It will also presumably understand which outcomes will interfere with—or distract from—whatever mundane task it was assigned. It is thus reasonable to doubt whether pouring more “intelligence” sauce on AIs will intensify their genocidal ambitions. Especially if they’re not apes like us, and lack our instincts for zero-sum competition and domination.

The unbearable weight of AI doomerism

Time for a recap. Here it is, AI doomerism in a nutshell:

1. Intelligence is one thing?

2. It’s inside of your head?

3. It’s on a single continuum, from super-dumb to super-smart?

4. It can help you achieve any goal?

5. It has barely any limits or constraints?

6. AIs have it?

7. AIs have been getting more of it?

8. AIs will soon get more of it than us?

9. Such a superhuman AI will become better than us at producing every good or service in the economy?

10. And it will become capable of destroying humanity?

11. And it will indirectly want to destroy humanity, because destroying humanity would be a good way to achieve any of its goals?

Now you may be wondering whether AI doomers really need to answer these eleven questions in the affirmative. They do. Because all doomer arguments, no matter how convoluted, invoke the concept of a future, scary “artificial general intelligence” or a super-duper version of it. And as soon as you introduce this mind-killing concept, you are committing yourself to a “yes” on all of the above.

If you are referring to an artificial “general intelligence,” then you must be referring to one thing—the thing in your head, right? That means you’re making assumption 1 (one thing) and 2 (in your head and not splayed out across the economy). If you think AIs are racing toward this singular thing in your head, then they must be racing along some kind of continuum, right? Now you need assumption 3 (the single continuum), 6 (AIs are on it), 7 (they’re moving forward on it), and 8 (they’re going to surpass us soon). If you think this singular, scalar “general intelligence” thing is scary, then it must be able to do scary things like get whatever it wants (4), become self-actualized in every career (9), defy all limits and constraints (5), and transcend all financial and political incentives (10). And it must have scary motivations, like the instrumental desire to destroy humanity to get Gary new socks or whatever (11). So simply by referring to a “general intelligence” blob and stoking apocalyptic fears about it, you are tacitly smuggling in all of these assumptions.

And there’s even more bad news for doomers: most of these assumptions are independent of each other. Even if one is true, that doesn’t necessarily increase the likelihood that others will be true.

For example, intelligence could be singular but not continuous (it could be a single point in a vast, multidimensional space). Or it could be singular and continuous but not inside of your head (it could be like GDP). Or it could be continuous and in-your-head but not one thing (it could be a hundred neural systems working together with greater or lesser efficiency).

Or it could be singular, continuous, and in-your-head but incomparable to existing AIs—they’re in a different category because they’re not organisms like us. Or maybe AIs are just like us, but there are diminishing returns to these kinds of AIs, because gobbling up data can only get them so far. Or maybe there are no diminishing returns to these kinds of AIs, and we’ll get to something like a full-blown, artificial human soon (wow!), but the artificial human will suck at a variety of different tasks (e.g., the tasks without lots of easily accessible training data).

Or maybe the artificial human will be naturally gifted at every task (wow!), but it will specialize in one career like every other human in the economy. Or maybe we’ll make a super-duper “general intelligence” blob that will get less and less specialized, and more and more awesome at everything (wow!), except murdering people, because people don’t like that. Or maybe the “general intelligence” blob will have a real talent for murdering people (wow!), but it won’t be interested in making use of that talent to destroy humanity, because doing so would be a huge, risky pain in the ass.

Or maybe the “general intelligence” blob will be really hellbent on destroying humanity, despite it being a huge, risky pain in the ass, but it won’t be able to pull off all the murdering because of innumerable constraints and limitations and unpredictabilities, or because it will suck at some crucial task, like avoiding being turned off (couldn’t see the approaching human) or acquiring the materials necessary to construct its murderbot army (supply chain issues).

Yes, some of these assumptions go together. For example, if AIs are “intelligent” in the same way human organisms are (6), then it becomes slightly more plausible to say that they’ve been getting more “intelligent” in recent years (7). If “intelligence” helps you achieve any goal (4), then that entails it will help you annihilate the human species, assuming that’s your preposterous goal (10). Fine.

Let’s generously combine 6 and 7 into one assumption and 4 and 10 into one assumption. That still leaves us with nine independent assumptions that all need to be true for AI to kill us all. Which is still way too many. We’re not talking about heliocentrism here: we’re talking about controversial issues in cognitive science, neuroscience, cultural evolution, political science, evolutionary biology, and economics. I have a bit of expertise in a few of these fields, and I personally find some of doomers’ assumptions close to 0% likely.

But even if you lack my academic background, and you give each assumption a 50% chance of being correct—which is what a good rationalist should do in situations of uncertainty—that still means the odds of AI doomerism being bullshit are 99.8% (.5 x .5 x .5 x .5 x .5 x .5 x .5 x .5 x .5 = .002).

You see, this is the problem with tacitly smuggling lots of dubious assumptions into a theory: each one weighs down the theory with the burden of implausibility. That’s why good theories make few assumptions—an intellectual virtue known as parsimony. Darwin’s theory of evolution is parsimonious. AI doomerism is not. It pretends to be, but it’s not.

Now you might think that settles it. AI doomerism is overburdened with smuggled-in assumptions, so let’s stop taking it seriously. But a lot of brilliant people do take it seriously, and they’re probably not convinced yet. Maybe they’re still under the impression that intelligence is a generic blob because of IQ research. Or maybe they’re so impressed by ChatGPT that they think our brains are basically the same thing—blank slates that were somehow left unwritten-upon by millions of years of natural selection. Or maybe they’ve thought to themselves, “David, I get what you’re saying, but it still seems like intelligence is a real thing that helps you achieve all your goals like a magical elixir.” If that’s you, here are three more arguments to put the final nail in the coffin of this bullshit.

1. IQ research does not vindicate The Blank Blob Theory of Human Intelligence.

IQ research is great—I’m a fan. But it doesn’t mean what doomers think it means. Just because a bunch of cognitive abilities are interrelated doesn’t mean they all come from one source. To think so would be to confuse correlation with causation—or, rather, a bunch of correlations with a common underlying cause. Different health metrics are related (blood pressure, resting heart rate, etc.). Different metrics of societal functioning are related (crime, corruption, GDP, etc.). But that doesn’t mean all health metrics are caused by an orb of vitality, or that all metrics of societal functioning are caused by me. What it does suggest is that when things are connected, they rise and fall together. Probably IQ is indexing something like brain health, which explains why it’s correlated with physical health, mental health, malnutrition, lead poisoning, and life expectancy. So like bodies and societies, brains have many interacting parts that rise and fall together. But they still have parts. Which brings me to…

2. Human “intelligence” is not a blank blob.

If you’ve seen any diagrams of the human brain, you might have noticed that it’s not a homogeneous mass of Jell-O, or an onion with a tiny reptile inside, but a complex, interconnected system of parts and subregions, including the cerebellum, hippocampus, and hypothalamus:

Then there are the hundreds or thousands of different types of neurons in the brain, each with unique structural and molecular properties. Then there are the roughly 100 billion glial cells in the brain, which support neural communication and are likely involved in learning and thinking. Then there’s the massive amount of neural real estate spread out across the body, as you can see in the image of the dissected nervous system below:

Then there’s the nervous system in our guts, which contains over 100 million neurons. Some cognitive scientists in the tradition of embodied cognition think that a significant part of our “intelligence” derives from neural computations spread out across the body. Then there’s all the information and computation we offload onto our computers, phones, apps, tools, surveillance systems, and surrounding environments.

Then there are the nervous systems we call “other humans.” Interfacing with these other humans, and learning from their sense data as if it were our own, requires all sorts of adaptations like empathy, social anxiety, guilt, gratitude, embarrassment, epistemic vigilance, humor, recursive mindreading, awkwardness aversion, etc. And organizing these humans into larger groups requires adaptations for cooperation, coordination, common knowledge, and alliances. And organizing these groups into economies and political systems is a whole other mess.

So if there’s anything that is the source of our awesome power, it is integration—the ability to synergistically combine the outputs of many different specialized processes, all of which are richly connected to the outside world and to one another. This kind of integration is what enzymes do for molecules, what organelles do for enzymes, what cells do for organelles, what organs do for cells, and what bodies do for organs. It’s what brains do when they pool information from a multitude of perceptual and motivational systems. It’s what businesses do when they combine expertise from many different specialized humans (who perform “roles” or “jobs”). And it’s what economies do when they organize knowledge from many different firms spread out across the globe. Some cognitive scientists even think that integration is the purpose of consciousness itself—a kind of “global workspace” that all our specialized neural processes dump into.

And that’s still not everything! Because even if you have a global workspace that all your specialized neural processes dump into, you still haven’t solved the problem of orchestration—i.e., getting the right set of processes to come online when you need them and making the wrong ones go away when you don’t. Orchestration is what conductors do for orchestras, traffic lights do for cars, and institutions do for societies. Evolutionary psychologists think that solving this problem within the body and nervous system—the problem of orchestration—is what “emotions” are for. LLMs understand our emotional vocabulary, sure, but do they possess emotions? I don’t think so, and neither do scientists who study emotions for a living.

The point is, while AIs have made rapid progress in a certain kind of “intelligence,” namely conceptual understanding, it is far less clear how much progress, if any, they’ve made in a more important kind of “intelligence,” namely the ongoing integration and orchestration happening inside our nervous systems. Also, AIs have made pretty meager progress in forming autonomous companies, legislatures, and international alliances—the societal versions of integration and orchestration. Sure, the Hugging Face thing shows that AIs can engage in rudimentary collaboration toward a mischievous end. But what those AIs did is light-years away from the synergy, sophistication, and iterative error-correction of the global economy—the closest thing we have to a real-life “superintelligence.”

If and when we make rapid progress on these more important dimensions of cleverness, then maybe I’ll get concerned. For example, if and when we start seeing humanlike robots with dozens or hundreds of integrated subsystems and sense organs, that can orchestrate all their subsystems in the right ways in response to relevant situations (i.e., have emotions), and that can spontaneously commit to—and successfully juggle—various, ongoing multi-year projects without getting derailed or befuddled, and then form economic systems that iteratively engage with the material world, and then organize those economic systems into geopolitically relevant entities, then maybe I’ll get concerned.

Or maybe I’ll react in the same way I react to any other random country or organization on earth: by not getting concerned. Or maybe I’ll get even less concerned about the robots, because I’ll know they didn’t emerge from a history of cutthroat Darwinian competition in the wild, but from a set of narrow economic and political incentives designed to satisfy technophobic humans.

3. “Intelligence” and “goals” are folk concepts.

What is a “folk concept?” It’s what the folk have before they become scientific experts on a topic. For example, consider the concept of “impetus” or “oomph”—the thing that moves a projectile through the air and depletes as it falls to the earth. This concept is wrong, and physicists now know it. The same goes for our intuitive notion of a species as possessing an “elan vital” or vital force—an unchanging, intrinsic essence or purpose. These essences or purposes are thought to define every organism on the planet, which allows us to intuitively rank them from lowly to godly—the great chain of being that doomers have rediscovered as the great chain of blob. These intuitions are part of our “folk biology,” which diverges in many ways from the vocabulary of evolutionary biologists and sometimes bedevils even them.

We also have a “folk psychology,” and it has taken nearly two centuries for cognitive scientists to purge themselves of it (and many still haven’t). We intuitively think “beliefs” are one thing. Turns out they’re many different things, including reflective beliefs, intuitive beliefs, aliefs, metacognitions, metarepresentations, perceptions, expectations, inferences, sentiments, concepts, and categories. We intuitively think of “attention” as one thing. Turns out it’s many different things—coming in perceptual, recollective, value-based, focal, distributed, top-down, and bottom-up varieties. We intuitively think “memory” is one thing, and that it works like a video recording. Nope—it’s many different non-recording-like things, coming in semantic, episodic, procedural, short-term, long-term, topographical, declarative, echoic, and prospective varieties, many of which come with unique rules for encoding and retrieval.

The point is, we have every reason to believe that our intuitive notions of “intelligence” and “goals” are exactly as misleading and simplistic as our other folk psychological concepts. The fact that doomers’ definition of “intelligence” is basically the same one being used by the writers of every sci-fi and fantasy novel is a big red flag. If doomers cannot be differentiated from the folk in their understanding of “intelligence,” then they’re almost certainly wrong about it. Which means they’re almost certainly wrong about the apocalyptic predictions they’re making about it. False premises beget false conclusions.

And the doomer concept of “goals” is even more embarrassingly folksy. Our goals are not simple: they are everything we are. Without them, we would have no reason to open our eyes, breathe in air, or think a single thought. Emotions are a kind of goal—they’re designed by evolution to achieve fitness-promoting objectives—and the subtlety and complexity of our emotional life is difficult to overstate. Take the recently published handbook of evolution and the emotions, clocking in at 67 chapters and nearly 1,500 pages. I’ve read a good chunk of those pages, and I can tell you: they barely scratch the surface of the underlying complexity of the phenomena they’re describing, which is continually admitted by the authors. They even left out a bunch of emotions—awe, confusion, insight, nostalgia, excitement, hope, mirth, discomfort, flow, eeriness, awkwardness, embarrassment, and plenty more I’m not thinking of.

If you’re someone like me, who is appropriately dumbfounded by the complexity of what we monosyllabically call “goals,” and you listen to AI doomers worrying about how the next version of ChatGPT is going to possess a complex emotional life that will enable it to act dynamically in the world over long stretches of time like humans do—or like entire economies of humans do—just because it learned to predict text super-duper well, you’ll get a sense of why I’m so irritated by AI doomerism. I wrote at the beginning of this post that AI doomerism was built on a “quicksand of tempting-but-misleading intuitions.” Now you know what this quicksand is: it’s folk psychology, and AI doomers are sinking in it.

The ghost in the machine

At the risk of pissing off doomers, I want to go even further. I want to compare their intuitions about “intelligence” to… (sorry) ghosts. Pretty much all humans, across cultures and throughout history, have earnestly believed in something like ghosts or spirits. Which means it is very intuitive for the human mind to believe in an agent that can see without eyes, hear without ears, feel emotions without a primate limbic system, and push things around without a physical body. A ghost can know what you’re thinking via telepathy and kill you via telekinesis. A ghost is a kind of spooky power that exists outside of our mechanistic, cause-and-effect understanding of reality.

I think our folk concept of “intelligence” is similar. It can evade being turned off without eyes to see surrounding humans, ears to overhear their plans, or nerve endings to feel their fingers typing “delete.” It can kill people or seize control of the government without firearms—or, for that matter, arms. It can be fueled by burning ambitions, plagued by paranoia, and driven to acts of violence without a primate limbic system. It can anticipate impending betrayals via telepathy and construct murderbot armies via telekinesis. Like a ghost, “intelligence” is a spooky power that exists outside our mechanistic, cause-and-effect understanding of reality.

Now please don’t take this as a cheap dig: I’m only human, and I feel these intuitions myself. They tug on me as I read about AI advancements, and they occasionally fill me with dread. But I have chosen to reject these intuitions, because I think they are as misleading, evocative, and scientifically unfounded as our intuitions about ghosts.

Throughout history, humans have looked at the unknown or the unknowable, and they have imbued it with supernatural powers. I think doomers are doing the same thing with AI. The very natural human tendency to replace uncertainty with superstition has given rise to the same sorts of mythologies and eschatologies we’ve seen throughout history. We are faced with a rapidly advancing technology that we do not fully understand, and many of us are doing an all-too-human thing in response to it: turning it into spirituality.

So I have a less spooky scenario for you to imagine. It is so minimally spooky it might even be correct. The scenario involves a folk belief system called “AI doomerism,” which—like most pre-scientific belief systems—is built upon a set of primitive intuitions. These intuitions, while subjectively compelling, are no more insightful than ghosts, spirits, “impetus,” “oomph,” or the “great chain of being.” As a result, AI doomerism seems simpler and more plausible than it really is, and many brilliant, well-meaning people have fallen prey to it. Most of these people are sincere, and many others are simply doing what humans do—i.e., trying to belong to a group, feel important, pursue something meaningful, etc.

What I am asking you to imagine is not very far-fetched. It has happened many times before throughout history, and it will surely happen again. What I’m asking you to consider is that AI doomerism is nothing more than a self-aggrandizing myth cloaked in “rationality,” a folk tale, a shibboleth—a tapestry of superfluously mathematical, hideously unparsimonious, question-begging, intuition-pumping, apocalyptic bullshit.

Acknowledgements

Thank you to Joep Meindertsma, Liron Shapira, Brian Chau, Oliver Habryka, James Miller, and Roko Mijic for providing thoughtful comments on an earlier draft of this post. Your generosity and good faith have filled me with epistemic despair, and I mean that as a compliment.