Yea not arguing against cases of humans using AIs for bad purposes. Agreed that's scary. Arguing against the more conventional doomer case of a misaligned superintelligence killing everyone against the intentions of the user.
Well, as an AI Doomer myself, I would say that concepts of intelligence have nothing to do with it. "AI" is just a lot of computer programs that regurgitate the inputs that they swallow up into responses.
So the Doomer part, to me, has nothing to do with "what is intelligence" and everything to do with letting a computer program that is "programmed" with a hodge podge of misc data make decisions for people. Most people who have used it report that it gives results that are just plain wrong, and frequently the models use information that they manufactured as a basis for the responses.
And then you see people using these programs to make decisions and, well, it is uncomfortable to think of what kind of things could happen. Do I really want a computer program investing money for me? Deciding who to target with drones? Not really.
One can go full doomer and look at the Kings College report that says that, in simulation, "AI" systems seem to love to use nukes. Maybe nothing doomerish will come from that. But the reality is that people are inclined to cede more and more decision making to these programs, and they have started using them, to some extent, for defense systems. So it doesn't take some kind of mystical "the computer will magically start to think and will hate us" feeling to worry about it. More like, what if some country directs an AI system to run their defense and the program just sees a conflict like a game and its little algorithm/decision-tree leads it to go all in to defeat the opponent?
"Composed of many things, therefore not one thing" does not follow. Your own example of GDP demonstrates this. It's a singular scalar that decomposes into trucking, dentistry, and every other economic activity, but you don't consider it a semantic catastrophe.
I'm with the rationalists on the broad, thin definition of intelligence as the capacity to achiever goals, to steer the future. That's not folk psychology. There's an extensive literature. It's well-grounded and measurable. And it's intentionally thin so that it doesn't constrain the substrate.
The ranking of a human > chimp > spider is true. Can we shoot webs out our butts? No. Can we catch more flies than a spider? Of course we can. Throwing out the scalar leaves an impoverished view of cognition where species and individuals are just better adapted to a niche, rather than better able to steer the future. That capacity is coherent, real, and useful.
The example of GDP was meant to rebut the assumption that intelligence is in your head—not the assumption that intelligence is one thing. So I agree with you that GDP is one, scalar thing. But doomers aren’t worried about AI GDP getting too high are they? And what is this “extensive literature” about intelligence and goals not being folk psychology? That the “ability to achieve goals” is a general capacity and does not depend on the specific goal, the specific context, and the specific constraints?
Okay, your GDP example might have been intended to rebut something else, but it also still works to rebut your claim that intelligence isn't one thing.
And yes, doomers are worried about AI GDP getting too high. That's exactly the worry. Distributed, agentic systems with way more steering power than the humans trying to direct them.
On the literature, Legg & Hutter (2007), Minds and Machines. They surveyed about 70 definitions from psychologists and AI researchers, found they converge on "the ability to achieve goals in a wide range of environments," and formalized it. Hernandez-Orallo's The Measure of All Minds (Cambridge, 2017) and Chollet's "On the Measure of Intelligence" (2019) are follow-ons. You may not like this work or agree with it, but none of it is folk anything. There's also a whole literature on a single g-like factor measurable across rodents and primates. And of course within humans it's one of the most robust findings in all of psychology. I'm not gonna flood your comment section with references. They're easily findable. Again, you may not agree with it, but saying there's no evidence or that it doesn't exist is just wrong.
And finally, that definition doesn't say the ability is independent of goal and context. It says that it's measured across multiple goals and contexts. That's where the 'general' comes from. The claim is that there's a factor which predicts performance across many goals in many contexts, and empirically there is.
I have always tried to understand life by observing it in everyday life. There is a fundamental difference between life and AI. In terms of capability, AI already surpasses humans in many areas. But in terms of existence itself, it is still far from life.
Life has to maintain itself. When hungry, it has to eat. When cold, it has to seek warmth. When danger comes, it has to escape. It suffers when damaged, feels excitement when it gains, and sadness when it loses. Perception, experience, emotion, and consciousness all operate within a structure that must continuously maintain itself.
AI does not have this foundation. It does not feel hunger or pain, nor does it need to feel excitement about its continued existence or sadness about its disappearance. It can understand pain, sadness, excitement, and joy. It can describe them accurately. But what it encounters are descriptions of these things.
Capability and existence are two completely different axes. AI can go very far on the axis of capability, but that does not mean it is moving closer to life.
Quite a thought-provoking post. Thanks. Contrary to how most people who have some education see themselves, smart is NOT something a person is. Or isn’t. Smart is something we do. Or don’t do. With every decision we take, because our lives are the sum total of our decisions. As Walker Percy, author of the great book The Moviegoer, said “you can get all A’s and flunk life.” And so far the creators of AI seem to have made some truly dumb decisions.
I also read Scott Alexander and, next to David Pinsof, he is the smartest person I have ever read. But he and his many Astral Codex commentators believe so many weird things, it brings to mind the saying, "things so dumb only an intellectual could believe it."
I am sure this is a great article, which I fully intend to read. However, the poster for "The Blob" says Andy Griffith's girlfriend Helen Crump (Aneta Corsaut) is the female lead, and since she was my teen hearthrob (yes, I am that old), I have to stop to watch that now. And because the main use of all new technology is porn, I wonder if Claude can make her topless as she makes out with Steve McQueen and runs panicked down the street.
The scenarios that side step some of these assumptions are the ones where a human uses AI to do irreparable damage. See Noah Smith’s post on AI bioweapons: https://noahpinion.substack.com/p/heres-how-were-all-going-to-die
Yea not arguing against cases of humans using AIs for bad purposes. Agreed that's scary. Arguing against the more conventional doomer case of a misaligned superintelligence killing everyone against the intentions of the user.
Well, as an AI Doomer myself, I would say that concepts of intelligence have nothing to do with it. "AI" is just a lot of computer programs that regurgitate the inputs that they swallow up into responses.
So the Doomer part, to me, has nothing to do with "what is intelligence" and everything to do with letting a computer program that is "programmed" with a hodge podge of misc data make decisions for people. Most people who have used it report that it gives results that are just plain wrong, and frequently the models use information that they manufactured as a basis for the responses.
And then you see people using these programs to make decisions and, well, it is uncomfortable to think of what kind of things could happen. Do I really want a computer program investing money for me? Deciding who to target with drones? Not really.
One can go full doomer and look at the Kings College report that says that, in simulation, "AI" systems seem to love to use nukes. Maybe nothing doomerish will come from that. But the reality is that people are inclined to cede more and more decision making to these programs, and they have started using them, to some extent, for defense systems. So it doesn't take some kind of mystical "the computer will magically start to think and will hate us" feeling to worry about it. More like, what if some country directs an AI system to run their defense and the program just sees a conflict like a game and its little algorithm/decision-tree leads it to go all in to defeat the opponent?
Unlikely? Maybe. Impossible? Certainly not.
"Composed of many things, therefore not one thing" does not follow. Your own example of GDP demonstrates this. It's a singular scalar that decomposes into trucking, dentistry, and every other economic activity, but you don't consider it a semantic catastrophe.
I'm with the rationalists on the broad, thin definition of intelligence as the capacity to achiever goals, to steer the future. That's not folk psychology. There's an extensive literature. It's well-grounded and measurable. And it's intentionally thin so that it doesn't constrain the substrate.
The ranking of a human > chimp > spider is true. Can we shoot webs out our butts? No. Can we catch more flies than a spider? Of course we can. Throwing out the scalar leaves an impoverished view of cognition where species and individuals are just better adapted to a niche, rather than better able to steer the future. That capacity is coherent, real, and useful.
The example of GDP was meant to rebut the assumption that intelligence is in your head—not the assumption that intelligence is one thing. So I agree with you that GDP is one, scalar thing. But doomers aren’t worried about AI GDP getting too high are they? And what is this “extensive literature” about intelligence and goals not being folk psychology? That the “ability to achieve goals” is a general capacity and does not depend on the specific goal, the specific context, and the specific constraints?
Okay, your GDP example might have been intended to rebut something else, but it also still works to rebut your claim that intelligence isn't one thing.
And yes, doomers are worried about AI GDP getting too high. That's exactly the worry. Distributed, agentic systems with way more steering power than the humans trying to direct them.
On the literature, Legg & Hutter (2007), Minds and Machines. They surveyed about 70 definitions from psychologists and AI researchers, found they converge on "the ability to achieve goals in a wide range of environments," and formalized it. Hernandez-Orallo's The Measure of All Minds (Cambridge, 2017) and Chollet's "On the Measure of Intelligence" (2019) are follow-ons. You may not like this work or agree with it, but none of it is folk anything. There's also a whole literature on a single g-like factor measurable across rodents and primates. And of course within humans it's one of the most robust findings in all of psychology. I'm not gonna flood your comment section with references. They're easily findable. Again, you may not agree with it, but saying there's no evidence or that it doesn't exist is just wrong.
And finally, that definition doesn't say the ability is independent of goal and context. It says that it's measured across multiple goals and contexts. That's where the 'general' comes from. The claim is that there's a factor which predicts performance across many goals in many contexts, and empirically there is.
I have always tried to understand life by observing it in everyday life. There is a fundamental difference between life and AI. In terms of capability, AI already surpasses humans in many areas. But in terms of existence itself, it is still far from life.
Life has to maintain itself. When hungry, it has to eat. When cold, it has to seek warmth. When danger comes, it has to escape. It suffers when damaged, feels excitement when it gains, and sadness when it loses. Perception, experience, emotion, and consciousness all operate within a structure that must continuously maintain itself.
AI does not have this foundation. It does not feel hunger or pain, nor does it need to feel excitement about its continued existence or sadness about its disappearance. It can understand pain, sadness, excitement, and joy. It can describe them accurately. But what it encounters are descriptions of these things.
Capability and existence are two completely different axes. AI can go very far on the axis of capability, but that does not mean it is moving closer to life.
Quite a thought-provoking post. Thanks. Contrary to how most people who have some education see themselves, smart is NOT something a person is. Or isn’t. Smart is something we do. Or don’t do. With every decision we take, because our lives are the sum total of our decisions. As Walker Percy, author of the great book The Moviegoer, said “you can get all A’s and flunk life.” And so far the creators of AI seem to have made some truly dumb decisions.
I also read Scott Alexander and, next to David Pinsof, he is the smartest person I have ever read. But he and his many Astral Codex commentators believe so many weird things, it brings to mind the saying, "things so dumb only an intellectual could believe it."
I am sure this is a great article, which I fully intend to read. However, the poster for "The Blob" says Andy Griffith's girlfriend Helen Crump (Aneta Corsaut) is the female lead, and since she was my teen hearthrob (yes, I am that old), I have to stop to watch that now. And because the main use of all new technology is porn, I wonder if Claude can make her topless as she makes out with Steve McQueen and runs panicked down the street.