Would you trust a bridge built by it with zero human interference? A train? A plane? A skyscraper? What about fight in a war? Unless you can, I wouldn't consider it AGI, because you're actually trusting human intelligence to verify the bridge or train or plane or skyscraper is safe or that the robot is following orders. And even then, you're trusting human-influenced guardrails etc. I would consider it AGI when an AI-created LLM can do all of these things and you trust them with your children's lives. Would you trust an AGI cop to protect your children from a violent criminal? Unless you can, believing what we have as AGI is just an empty opinion with no meaning behind it.
Would you trust a bridge built by a single human? In reality, we have lots of guardrails to ensure that we don't screw up and kill a lot of people by deploying defective bridges (or cars to drive on them). Those guardrails often are written in blood, and still occasionally fail.
Why not have AGI check other AGI? Peer review by fellow humans is what gives humans assurance, to the degree in which review was done by peers of equal or greater intelligence.
AGI checking other AGI should give you that same trust, no? Deepseek says my ChatGPT bridge is stable, you should trust it. Claude says it's stable. The humans say it isn't, but they aren't AGI. You can trust this bridge because it's been vetted by AGI. In my opinion, LLMs cannot be AGI, so for me I would never trust them above any human I would trust. But for those who do believe LLMs can be AGI, they have to demonstrate why we should trust them above any human in these extreme cases. Meaning, if someone says "Well the department of safety (ran by humans) says it's not safe" we have to believe that AGI just knows better than the department of safety. I think this is not possible right now, which is why I don't think we can trust anything built by LLMs where we need the tolerance of risk to human life and safety to approach zero. American AGI soldiers invade the home of Iranian citizens because they have been identified as terrorists. Do you trust the AGI to know if the visual scan they see in this civilian home is a threat to the interests of the United States government and its citizens?
This is really the point: We build systems and processes which help ensure that we get reliable outputs (bridges) from unreliable actors (human workers). The point is not that we will one day replace all of these processes with AGI. The point is that we will still want processes and systems in place to ensure reliability.
The importance of safety process is largely a function of the risk of failure (ie, how unreliable the production process is) and the cost of failure. If you've got a very reliable production process, perhaps you need less safety process. But if the cost of failure is measured in lives, you're still going to want to have checks in place, regardless of who/what is responsible for designing the bridge.
One step further, our current generative AI is not discrete. “AGI checking other AGI” doesn’t really make technical sense. Currently, every interaction between two generative AI units is treated as agglomerative. We talk about “Claude”, not the 70 subagents that Claude spun up to achieve a task.
In other words, as soon as two generative AIs interact, they become one. Our current definition of AI (generative AI operating in feedback loops) is dependent on that.
Edit: this is somewhat an epiphany to me. Our current generation of “AI” isn’t an “entity”, it’s a “process”. I guess it’s hard to define formally, but I would compare it to how law and the pursuit of justice is a process, not an entity.
And I think that’s a fundamental limitation to achieving artificial general intelligence.
Yes, and that is why you can't say "its as smart as a single human" and think its as smart as humans, because humans can get smarter by cooperating, if the AI can't do that to create a smarter AI it is not as smart as humans, meaning an AI that is not as smart as a team of humans is essentially not as smart as a single human either.
I’m not so sure this is trivial and strongly depends on the biases of the agents that the group is composed of. It seems like some species of animals (humans, ants, wasps, etc) seem to select for intelligence in group settings more than individual intelligence. One of my favorite facts about human evolution is that in the process of becoming homo sapiens our brains actually shrunk relative to our ancestors possibly due to our reliance on social ties and specialization allowing for each individual unit to be “dumber” while the collective becomes more intelligent than any individual ever was.
It’s not apparent to me that current LLM’s have the right biases for a trivial unit of “more LLM” = smarter. It seems apparent that LLM’s are under many metrics more “intelligent” than the average person but they still don’t rival our corporations or collectives in terms of intelligence and I don’t see them being plug in replacements for humans and maintaining the benefits of the collective structures yet. Not impossible but definitely not trivial in my mind. We and our cultures evolved to work together and the modern world is the result of the emergent structure that resulted.
As a plug my favorite field relating to this is called stigmergy and basically describes how individual units like ants build intelligent collectives that are far greater than the sum of their parts.
