There’s this idea of the superhuman AI, the one that gets to a certain threshold and just accelerates and improves itself forever, the one that either kills us all or makes us immortal, one AI to rule us all. We can thank science fiction writers and the doomers for that vision.
I don’t buy it.
That whole idea rests on the false assumption that intelligence equals power or success. If that were true we’d live in a world run by physicists and mathematicians. Instead, we get [insert your favorite mediocre politician here], in the real world you need a combination of skills to make things happen.
IQ itself is overrated when it comes to usefulness. It helps and it’s probably the one trait that people can’t learn. But just analyzing stuff doesn’t get it done, Kevin Kelly calls this thinkism: the belief that thinking about things will solve them. And when people worry about the superhuman AI taking over they’re making the same mistake, assuming that just because it’s AI, IQ will suddenly matter more.
What we call AI today is narrow AI: tools that are incredibly powerful at specific tasks but incapable of “taking over” anything. The superhuman AI scenario only makes sense if we’re talking about AGI: a fully autonomous system with real creativity and agency. But when we eventually get to AGI, I think we’ll realize it’s just another person with better hardware.
The part that makes an intelligence general is creativity: the ability to create new knowledge, as David Deutsch says, and that’s software, not hardware. You can’t brute-force your way to creativity with more compute.
Humans are already general intelligences with that software, the difference with AGI will be hardware: more speed, more memory. But there’s no reason to believe humans can’t also take advantage of that same hardware.
So yes, the first AGI will be a genius, probably the smartest mind on the planet. But one genius, no matter how capable, is still just one person, clone it a million times and the marginal value of each copy drops toward zero, the law of comparative advantage makes sure of that, people are valuable because they are different.
We do need the high IQ physicists, mathematicians, engineers, sure, but we also need philosophers, artists, teachers, salespeople for a society to actually move forward. Different problems require different types of intelligence.
The “one AI to rule us all” scenario doesn’t make sense. Intelligence isn’t winner-take-all. Reality is way more interesting than that, what we’ll probably get instead is what we’ve always gotten with new technology: specialization and diversity. Many different AGIs and AIs working alongside humans solving problems faster than ever.
"The part that makes an intelligence general is creativity: the ability to create new knowledge, as David Deutsch says, and that’s software, not hardware. You can’t brute-force your way to creativity with more compute." This.
Also it seems to me IQ is only helpful insofar as it's able to be demonstrated in terms other humans understand. That's when it becomes social capital. It doesn't have to be all humans; it can be specialists in certain institutions, etc. But I think it's the reason we have these "creative geniuses" whose works were only fully admired after their death because there wasn't someone (or someone with enough influence) capable of recognizing the intelligence during their lifetime.
Incredible insights. Loved it!