People have all sorts of hobbies—some collect stamps, others collect euros, and some even have the patience to watch Bulgarian football. Well, besides my habit of testing my meager knowledge on TV quiz shows, I have one more: to question things, to experiment, to look for viewpoints on modern technology that differ from the stock ones.
What is the intelligence quotient of advanced artificial intelligence?
Does it reach, or even surpass, the average 100 points of today's humanity? Are the tests that single out the most intelligent people on the planet an adequate measure of progress? (Where, incidentally, a Bulgarian woman tops the all-time ranking.)
Let's find out together!
Brains, So to Speak
You've probably been let down by a friend at some point. Well, it happened to me a few weeks ago. I decided to pit ChatGPT (may it count me as a friend once it gets even smarter) in a quiz contest against some of the undisputed winners of The Last One Wins, a quiz show on Bulgarian National Television (BNT). A year ago, Atanas Atanasov and I had run a similar experiment—we entered the electronic know-it-all in the competition, and it took an honorable fourth place out of 11 teams! This time we wanted to see how far the models had come, so we put OpenAI's flagship up against the champions. I was already savoring how it would crush everyone by a huge margin… and the result turned out to be tragicomic (see here).
Why did that happen? Are the models doing worse, even though, according to the news from the AI world, each new one is more astonishing in its abilities than the last? What happened?
I still have no explanation, even for myself. So I decided to put this “friend” to the test once again. Would it “betray” me again this time?
Too Clever by Half
I'm joking, I'm far from ready to accept an algorithm as a friend (although soon I'll tell you the story of an AI that was let “out of prison” to have sex, so… you never know).
I had been putting off this experiment for a long time; I prefer to run tests I come up with myself, or ones you give me the idea for (here). I came across an article that checked my “friend's” IQ exactly two years ago, and not just anywhere, but in Scientific American. It truly amazed me! The author, a Finnish psychologist, had given the algorithm part of the classic Wechsler test for measuring intelligence, and the result was a whopping 134 points! The algorithm, and in those early versions of it that were far from perfect, had turned out to be more intelligent than the vast majority of people! Or, to put it in numbers, the AI had landed in the top 1 percent of the most intelligent “people” on the planet!
I should point out that in that case only the questions that could be phrased as text were submitted—so Eka Roivainen had administered only five of the 11 subtests in the battery.
The Wiser One Gives Way
And still: “ChatGPT's verbal IQ is 155, superior to 99.9 percent of the test takers who make up the American WAIS III standardization sample of 2,450 people. As the chatbot lacks the requisite eyes, ears and hands, it is not able to take WAIS's nonverbal subtests. But the verbal IQ and full scale IQ scales are highly correlated in the standardization sample, so ChatGPT appears to be very intelligent by any human standards,” the author wrote.
Now, however, the new models take in and process images beautifully, and they handle Bulgarian so well that even leading literary figures can't tell them apart… So I decide to show no mercy whatsoever, especially to a traitor (just kidding, ChatGPT!)! This time it's high time to run the real test, the way people take it. What will happen?
Making Sense
The question of how exactly we test how “good” the models are is vast and touchy even in scientific circles (as we've discussed before). We've all heard of the Turing test, but… it isn't nearly that simple. There is a huge number of criteria by which the various models are evaluated. In recent months, the Bulgarian institute INSAIT has been publishing remarkable comparisons along these lines. Even so, the criteria for machine intelligence are becoming more countless than those for measuring human intelligence. They do have one advantage, though—the models are deliberately trained to meet them, so every new set of specific tasks quickly turns out to be within reach of the next generation of algorithms. That can't happen with us humans.
That's why I settle on the traditional test which, as civilization has developed, we have resigned ourselves to using to measure human intelligence.
Unfortunately, there's no way I can give ChatGPT the actual MENSA test, established both in Bulgaria and around the world as the undisputed standard. It is held under strict conditions, in the presence of a psychologist, and nobody but the psychologists has access to it, so that the results are reliable. A person can take it only twice in a lifetime!
Aren't You the Clever One
The closest thing to it I have is the test offered as a sample by the brainy organization itself in Bulgaria, prepared by its Norwegian branch (you'll find it here). Thirty-five questions following exactly the logic of the real test, 25 minutes.
Which of ChatGPT's models would perform best? Who would know, if not they themselves! I ask the “contestant,” and it picks two representatives—ChatGPT-4o and o1—the first as the best for a wide range of tasks, the second for its skill at finding logic. No problem, I'll test both. More participants, more accurate results! (Once again it's important to mention that my experiments are journalistic and in no way claim even remote scientific value.)
I copy the image of each task, give it to the algorithm and ask it to give its answer without any further explanation. Then I enter it. Yes, my “guy” is pretty fast under these conditions: only seven of the 25 allotted minutes have passed, and the form asks me whether I'm absolutely, completely sure and certain that I don't want to use the remaining time. No, I can't wait! I submit the answers. Will ChatGPT turn out to be a genius or an intellectual pariah?
Where the Road to Hope Begins
Well, we'll wait a little for the answer. The form informs me that the result is below 100 points and can't be any more specific in its reasoning.
Fine, maybe the second one will give an exact result.
Alas, the result is the same.
Could it really be, as the form also says, that this isn't a software problem? Some time ago ChatGPT and I competed in “the Bulgarian Turing test”—the matura in literature, Bulgaria's school-leaving exam, and I managed to beat it. What would happen here if I joined in too?
I admit I don't try too hard, I'm tired (everybody says that, I know), and still I get a result saying my IQ is 130.
(And it's higher. Programmer's honor! Besides having a document certifying 148 points, they even wrote about it in the newspaper when I was young:
)
But that's beside the point. Has ChatGPT really failed? Does below 100 mean 20 points or 99? Exactly how intelligent is artificial intelligence today?
Who's to Say?
No, I won't give up that easily. I dig around online and find the most convincing online test.
Yes, for 30 leva (about $16.50) (I'm not one of those collectors, I'll part with them) I'll be able to pit the contestant against a version of the aforementioned Wechsler test. Here, I'm sure, there will be a result, even if it's a negative one—after all, it costs money.
This time our hero gets 20 questions, again following the familiar logic of a visual element that completes the sequence. Once more I don't look closely or think it over, I just feed in the images, enter the answers and finally…
I get a certificate!
Excuse me—not me, of course, but ChatGPT.
100 points!
That's quite a solid result. Yes, I know the results aren't official in any way, but it's still remarkable.
It Won't Go Hungry
The algorithm gets exactly the score considered the average for the entire human population.
You do realize we're talking about a technology that, until a few years ago, even its own creators didn't think it reasonable to call “artificial intelligence,” hedging their bets with the neutral term “machine learning”? The same algorithms that performed tragically in rare languages like Bulgarian. And that promise, by the end of this year, to be able to solve every single task better than the best-prepared members of the human species…
Interesting times are coming!
Wool-Gathering
Since I did pay 30 leva, I also get more detailed data on where “my guy” stands, in which areas it performed excellently and in which merely satisfactorily. Of course, it's no surprise that it leads in the speed at which it solved the tasks. That's one of the big differences—we humans can complete the test for an average of 100 points in 25 minutes (if the questionnaire is designed as it should be), but it is much faster.
It's important to note something else: I'm using a general model that hasn't been trained specifically for tasks like these. Just as 100 IQ points is the standard for all kinds of people. So, if we compare the two values in a brutally crude way, today ChatGPT is as “intelligent” as we are. And if the quotation marks around that word remain fully in force, you'll soon learn why from the next digital story.
There are five main areas this test checks. My friend (it didn't betray me this time!) performed best at deductive reasoning and weakest at finding numerical logic. That's not surprising at all—in recent years, simple arithmetic has often turned out to be AI's Achilles' heel.
I hope you'll also be amused by how far ChatGPT lags in this test behind Charles Darwin, and even behind such erudites as Drake and Justin Bieber. Then again, we all know who laughs best.
All right, we won't make it an astronaut—at least not before the next generations of AI, which will do even better on this test and every one after it.
I should point out once again: here ChatGPT, in this case the o1 model, simply received an image, which is definitely not its strong suit. And although it wasn't easy, it clearly coped.
Club 100
IQ 100. It sounds so unrealistically round that it almost seems rigged. But then, as Milan Kundera wrote, “Chance and chance alone has a message for us.”
Is that a lot or a little? What does it mean for all of us as a species? Why must intelligence never stop being a goal and a direction?
For these and many other important questions, look out for the next Digital Stories. And we'll continue down the road of intelligence in an interview with MENSA's national psychologist.
Stay smart and keep searching! Of one thing I'm certain: it's the only way we can survive…
