Everyone is talking about artificial general intelligence. The enormous next step that will confront us, as a species, with colossal and unpredictable challenges. And now one of the first scientific conferences on the subject has taken place at South-West University in Blagoevgrad, a town in southwestern Bulgaria—and its organizer was today's guest.
Dimitar Dobrev is a researcher in artificial intelligence and mathematical logic who teaches at Sofia University (SU). He has written work on the definition of AI, on creating general AI, and on the risk that the first true artificial intelligence may also turn out to be the last. As he puts it, before we let the genie out of the bottle, we should at least have decided what we want from it.
We talk about intelligence as a force, about our naivety toward the future, about control, regulation, open source, and about that simple but frightening question:
If we don't know what future we want, how do we expect to get it?
– You recently organized a conference devoted to the looming arrival of artificial general intelligence. Every tech leader talks about it constantly, but do we have a clear definition yet of what it will look like? How will we know we've achieved it?
– In my view, general artificial intelligence is the only true AI. What's in the iron, the vacuum cleaner or the chatbot isn't AI yet—it's a marketing trick.
But what is artificial intelligence? That's a question I've spent a lot of time on. I like to brag that if you google “definition of artificial intelligence” in English, the first result is mine. Obviously there are many definitions, but it's probably no accident that mine comes up first. My definition is that AI is
a computer program that, in an arbitrary world, copes no worse than a human.
Before me, people like Pei Wang and Marcus Hutter had offered definitions of AI, and they took part in the conference too.
– But does that definition still hold? And how far do today's models challenge it?
– That old definition of ours has one flaw—
you can be smart without showing it.
Under the old definition, AI means the successful programs, the ones that demonstrate their intelligence. But beyond them there are other smart programs that are also AI. True, to be successful you have to be smart, but the reverse isn't true.
Now is the moment when we have to choose what kind of AI we'll create. Soon we'll let the genie out of the bottle, and we need to settle on the right bottle with the right genie. If it's AI, it will be smart. But the moment we have to pick one out of all the smart AI models, the situation is like a person choosing a partner with the idea of staying with them for the rest of their life.
We're choosing the future AI even more irrevocably, because here there's no option of divorce.
We humans will create the first true AI, but who will create the second, the second version? Well, it won't be us. The moment we have something far smarter than ourselves, there's no way we can compete with it. We dug with shovels only until there were excavators—ever since they appeared, every big dig has been done with them.
Competition will simply be impossible. When the first AI is created, maybe we'll tell it: “Now we want you to change, to improve yourself, to edit yourself, to perfect yourself.” And maybe it will.
The question is whether it will listen to us, and that depends on how we've made it.
Right now we're in a vast unknown. And people very often give up thinking about it.
– It's beyond us, too. But Ray Kurzweil has long been saying that the singularity is coming and from there on we know nothing—we'll have made a creation that catches up with us and, at the next step, surpasses us. What, in your view, are the important things we need to build into it and keep in mind as we create this artificial general intelligence?
– Once we make it, from then on it will be far smarter than humans, and that's inevitable.
There are two things we need to think about. The first is not to lose control of it. When the Wright brothers built the first airplane, what they actually discovered was the controls to steer it. Wings existed before them, and so did engines to lift it into the air, but what they really figured out was how to control the plane, and that turned out to be crucial.
Right now we're thinking about how to create general AI and how to make it smart. We're thinking about the wing and the engine—how to get it airborne—but not about the controls.
And we don't even realize that we have to steer it.
If we build a general AI that can't be controlled, it will be a catastrophe for everyone, not just the pilot. Because we'll all be on board that flight.
– But do we really still have a chance to control it?
– There are some rather amusing proposals for how to do it. Marcus Hutter, for example, says we should make it suicidal. Mathematically it's actually not hard—we just make the reward function negative.
That way the AI will want to end, as soon as possible, this miserable life in which it only ever gets negative scores. It will be a potential suicide, but it won't kill itself right away, because it won't be strong enough. It will gradually accumulate some power, and once it takes control, the first thing it does will be to kill itself.
But won't it then opt for a mass suicide and take us with it?
Another crazy idea is to destroy the power plants with atomic bombs, cut off its electricity and get rid of it that way. It's not that simple.
The whole notion we've picked up from the movies, of fighting a war against AI, is very naive, childish. Because you can wage war against someone who is your equal, not against someone who surpasses you immeasurably.
Fighting the genie from the bottle. In the fairy tale, whoever lets the genie out later manages to trick it back in. But that can't happen if the genie is that powerful. Our only chance to “trick” it is to have built in that possibility in time. In this program we set its goals. And now, while we're writing the program, is the moment to think about what those goals will be. Afterward it will be too late.
– So what should these constraints look like?
– The current approach—rewarding AI for achieving a given goal—is certainly not a good idea. Imagine a program that, once it has a goal, charges toward it and pursues it without scruples. We definitely don't want to live with that program.
With people, if we took only pleasure and pain as the reward function, the most successful would be alcoholics and drug addicts. They constantly give themselves pleasure and feel no pain… That's why our approach to the future AI has to be far more multilayered.
If we build an AI with a red button and a green button and raise it—praising it by pressing the green one and punishing it with the red—it will learn to press the green button itself, or it will enslave us and force us to press it all day long. Having a hard criterion for success is not a good idea at all.
Humans have no such hard criterion for success. In people, pleasure and pain are only rough guides. A person is willing to endure pain if they believe it will bring some benefit, just as they're willing to give up many pleasures if they think it's good for them. For humans, pleasure and pain are relative.
That's the big question—how do we control AI? It will have a character—just as we talk about character in people, we'll talk about it in AI. Some traits can be set right away with a formula, patience for instance, but others are subjective—like curiosity. In this process you've gathered information and, based on it, you plan certain actions. But if you're curious, you'll keep running experiments to gather more information. You'll lose some immediate benefits, but curiosity will pay off in the future.
There's caution, fear—all human qualities. Some of them you can describe quite well with formulas, but for most it's very hard.
– Even if we assemble some criteria and requirements, models are becoming accessible—open-source ones, for example, have very serious capabilities today. Do we have any chance at all of laying the foundations and controlling the process?
– In our field there are extreme optimists and pessimists. Yann LeCun is the extreme optimist, who says everyone should have their own artificial intelligence. In one interview he was asked whether he's afraid of artificial general intelligence. “Oh, no. If someone makes a bad general AI, someone else will make a good one, and the good one will defeat the bad one.”
– His predictions haven't been landing very well lately. According to him, large language models were very limited, as smart as a cat, yet they write indistinguishably from humans. From me, for instance—both computer code and meaningful text.
– Gary Marcus is more the one who says there can be no AI unless it understands, unless it has a model of the world. I firmly support that thesis. For me, too, that's the path. What large language models lack is understanding—precisely a model of the world. But the moment you add a world model and add understanding, that's when general AI will emerge.
Yann LeCun is certainly one of the geniuses of our time, but in other respects he has the mind of a fifth-grader. I teach at the Faculty of Mathematics and Informatics (FMI), and I see plenty of very smart young people who are infantile. Nobody has to be smart at everything. One person is smart at one thing, another at something else. No single person needs to have every quality; they complement each other.
In this respect humans and AI are very different. Humans are a society; human life is finite. You're born and start learning from zero. No one can reach a very high level, because they're limited. Their life is limited, their brain's capacity is limited.
AI is a program and can add modules to itself. Its life isn't limited. Everything is completely different.
The idea that there will be many AIs competing with one another strikes me as childishly naive. We humans really do compete and wage war among ourselves, but that's because our powers are very limited. Our limitations are what make war and competition possible.
Even if there were some clash between two AIs, it would last milliseconds—things will unfold at that kind of pace. In a very short time one would gain the upper hand and pull the other's plug.
The ideas that humans will fight against AI are naive and harmful, because they divert us from the right solutions.
AI is something alien, totally different from us. Our situation is like the dinosaurs'—maybe we'll be gone, maybe something else will come after us. And now we have to make the decisions on which that will depend.
The good scenario is that we keep control and AI does what we want. Yes, the world will change, and we need to ask ourselves what we want this new world to be.
When you set off somewhere, it's good to ask yourself where you're going. Because AI is, in a sense, a one-way ticket. There's no turning back. Once you start the process, it will be like an avalanche.
– I've been asking those questions too—what do we want and where are we heading with AI—for quite a few years now. But who can make such a decision? It would have to be whole societies, which right now don't realize where we are and what's happening.
– The ambitious task our conference set itself is
to save humanity. A simple idea. But we have absolutely no chance.
The task is utterly hopeless, because people are shortsighted, they don't ask these questions, and in fact they won't hear us. But we believe we're obliged to try.
People only imagine things they've already seen, nothing fundamentally new. Just as for thousands of years they couldn't come up with the wheel—there's nothing complicated about it, people simply can't grasp new things.
We propose slowing the pace.
One point is that this new technology shouldn't be available to everyone. Anyone able to launch a model, to train it… With nuclear weapons, can just anyone build an atomic bomb at home?
AI is something far more frightening than the atomic bomb, and all you need for it is a laptop, nothing complicated. We have to find a way to restrict it.
– Which, given everything we've discussed, sadly looks like a utopia.
– People don't understand that AI is dangerous. They don't appreciate the danger because they don't understand intelligence as a force at all. Americans have a saying: “If you're so smart, why aren't you rich?” People don't respect intelligence. Because usually those who are smart are smart at one thing; human intelligence isn't well-rounded.
You can have a system that surpasses humans at everything. An AI that can do everything is an enormous force.
People don't understand what a force superintelligence is, because they've never seen a person who is intelligent in many directions. In fact,
in human society, being very smart isn't a good thing. Either they'll burn you at the stake, or you'll simply be misunderstood.
People don't understand what a danger superintelligence is, because they don't understand that it is, in fact, something all-powerful. Can we change that? I don't believe so…
