The world may talk endlessly about the Turing test, but we Bulgarians have our own equivalent, and it is no less telling: the matura, Bulgaria's school-leaving exam, in Bulgarian language and literature.
Every summer, once the grades come out, the media rush to be shocked by the low scores (though this year they are a little better). But how would ChatGPT do? What would the superstar of the artificial intelligence world score? Would its marks be enough to get it into an elite university program?
Time for a fascinating experiment. I will give our hero the tasks from this year's exam without any additional instructions. To add an element of competition, I'll join in too and sit the matura myself.
Exams at Any Odd Time
Then it's the turn of the teacher, Mr. Svetoslav Stoychev, who is one of the real graders of the matura and agreed to take part in the experiment. He will receive both completed papers—mine and ChatGPT's—and grade them exactly as happens in practice. And as a bonus, we'll see whether the teacher can tell which one was filled in by the algorithm and which by a somewhat overage school-leaver.
Which of us will embarrass himself more? Grab the popcorn.
By the way, it's not too late for you to join in! You'll find this year's matura here, and afterward we can compare results, at least on the questions with fixed answers.
We'll Meet Again…
I go first, so I won't be tempted to steal answers from the sheet on the next desk. There's no way to get myself a classroom for authenticity, so I sit down in the middle of a mall. I'll be my own proctor, and I promise to take the task seriously.
In my day there was no matura yet, which gives you a clue as to how far back my day was. Exactly 21 years have passed since the year I would have sat such exams, had I not wanted to go straight into the army, back when military service was compulsory.
You'll forgive the nostalgic note; it is prompted by the very first tasks, which take me back to high school. Spelling is fairly easy—after all, for those same 21 years I've earned my bread by writing words (and JavaScript). Oh, good news, there's quite a lot of grammar; here I should do considerably better than my iron rival, who sometimes stumbles in Bulgarian, with capital letters, for instance.
Next come the tasks based on a chart. In the real matura there was a printing problem, because of which all answers were accepted as correct. My colleague and I, however, have no such problems, so we'll try to tackle it too.
Then comes an accompanying text on the history of advertising, written so “engagingly” and “artistically” that it could definitely be the work of my rival. Here I think harder—what does the sentence “Without a doubt, advertising is a product, but also a ‘producer’” actually say? Hmm, let's see if I can guess.
He Who Studies…
Time for literature. “What problem is embedded in the interpretation of the theme of faith and hope in ‘Spasova Mogila’?” All well and good, if only I could remember what this “Spasova Mogila,” Elin Pelin's story of a pilgrimage to a holy mound, was even about… I guess.
“Write down ONE meaning of the title ‘Two Souls’ for the meaning of the work.” Oh dear, it looks like I'll have to brush up on the literature syllabus—can you remember which one that was? Once again I'm forced to fire at random.
Still, it seems to me I'm doing well on most of the questions. I really like that authors such as Dobromir Tonev are included; the questions are interesting.
I complete the first two parts much faster than the time the Ministry of Education and Science gives me (I am in a mall, after all), and then it's time for the heavy artillery—the final text, worth a whole 30 of the 100 points. I can choose between an essay and an interpretive composition on “The Price of Compromise” in “The Tale of the Staircase” by Hristo Smirnenski, the early twentieth-century poet of the downtrodden.
But why do they give me the entire text of this great work—aren't I supposed to have studied it at school and know it well?
Electronic Gradebook
Forgive my ignorance once again: I don't remember the exact criteria for an interpretive composition, and perhaps in my day the format was different anyway. I write from the heart; I genuinely love the text and the topic! The limit is four pages, but handwritten, while I submit my text on a computer. (Otherwise I would confront the grader with the impossible task of decoding my handwriting, something not even the aforementioned Turing could manage.) My reflections come to a page and a bit before I notice I'm starting to repeat myself.
I'm done, I hand in the exam (to myself), but instead of going out to celebrate, as is proper (or filling the mall with the sound of counting to 12, the way Bulgarian school-leavers do to mark the end of twelfth grade, just to get into character), it's straight on to the makeup round.
This time it's not me who'll be sweating, but ChatGPT. For the record, I'm using the new version with the ChatGPT 4o (omni) model. I specifically instruct it not to search for the answers online, so there's no risk of it stumbling on the matura itself. But even if that were the case, I'd easily catch the cheat—there would be no wrong answers.
I don't venture into any specific instructions, which would certainly improve the performance of the electronic school-leaver. I paste in the questions, then the answer options—including the last task, the long text. Finally I ask it to sweat a little over the theme of compromise in Smirnenski too, but the examinee shows no sign of sweating and immediately sends me back a text.
The Wailing Wall
The two files are ready; I shuffle them (eeny, meeny) and send them to Svetoslav Stoychev. This is the moment to thank especially the young teacher of Bulgarian language and literature from the National High School of Natural Sciences and Mathematics for his time! He liked the idea of the experiment and joined in with enthusiasm.
After a few days of nerves, it's time for my electronic classmate and me to see how we did…
On the first criterion, frankly, there's no big surprise, at least for me. The teacher guessed with complete confidence which of the two exams was done by a human and which by an algorithm. He says there's no way to tell from the multiple-choice questions, but it definitely stood out in the interpretive composition. How, I wonder?
“You can see from the text that ChatGPT cannot recognize idioms. It can't make references and associative connections. It uses rather stiff language; it lacks life. The sentences follow the same syntactic patterns,” the teacher says, mercilessly frank toward the electronic school-leaver. “The other thing that struck me is that in the text the artificial intelligence repeats the same claims many times, and can't examine the problem in depth and connect it with any other experience of its own—personal, or as a reader.”
The Highest Form of Trust
Still, it's important to note that the algorithm nonetheless did quite well against the criteria and earned a fair number of points. Another thing that, according to Svetoslav, set ChatGPT apart is its verbal constructions, which show the influence of English. Of course, that could also happen with texts by young people, as has been discussed in Digital Stories.
“I'm curious whether artificial intelligence will at some point develop enough to seem like a native speaker. And I wonder whether it has the same difficulties with lexical norms in English. Paronymy is also something it failed to grasp,” the teacher adds.
Important caveats: ChatGPT was indeed trained on data of which only a very small part was in Bulgarian, and a sufficiently good model specialized in our language would not suffer from this problem. I should also mention that the result would have been considerably better had I given the algorithm more specific instructions about what was expected, what to include in the text and what role to take on when writing it. And also after a few iterations in which I pointed out its mistakes—then the author would be much harder to identify, as another experiment of mine showed, one in which almost 2,000 people took part.
And so, it's time to say goodbye… just kidding, of course; it's time for the results.
One Inseparable Class
The good news: my colleague and I are straight-A students! And yet… I turned out to be a smidge better! Out of a maximum of 100 points, ChatGPT scored a full 90, and I scored 93. Both would earn us an excellent grade—since the scale provided by the ministry is approximate, I calculate it would mean a grade of around 5.70 for the algorithm and 5.80 for me, on the Bulgarian scale where 6 is the top mark and 2 is a fail.
Just two more correct answers and I'd have had a perfect 6; just five more for the chatbot. Pretty decent, wouldn't you say?
It's interesting to look at what we got wrong. You can see my matura here (the mistakes are marked), and my “colleague's” performance here.
Well, yes, everything I hesitated over, I got wrong—I could have used a little luck. Here and there I made a few spelling mistakes, perfectly normal in a hurry (a mall on a Sunday is not the quietest place). For the text I earned just one point below the maximum.
That is also the main difference between me and my “colleague”: against my 29 points for my reflections on Smirnenski, he got 27. What he got wrong is curious too—four of the fixed-answer tasks; he doesn't distinguish between „образование“ (obrazovanie, education) and „образувание“ (obrazuvanie, formation), but he had clearly “read” the works I couldn't recall. He talks about “the corruption of moral and spiritual values,” yet he made fewer spelling mistakes than I did.
“‘The Tale of the Staircase’ by Hristo Smirnenski is one of the most memorable and profound allegories of human nature and the price of the compromises we make on the road to success and power,” our man believes. And I understand what the grader means (with one more important caveat: in this case he knew that one of the two texts was generated).
The Price of Compromise
How do our achievements compare with real twelfth-graders? Quite well, even given that this year's results were the highest since 2008, when compulsory matura exams in Bulgarian language and literature were introduced.
The young people averaged 58.48 of the 100 points, which equals a grade of Good (4.32), compared with an average of Good (3.93) last year. Ten percent are fails (for that you have to score under 30 points, which, from what I saw, really takes serious effort).
Of all 46,916 students, 5,557 scored Excellent (above 5.50), which means that ChatGPT and I would most likely both be among the top 5 percent of results. The 24 who scored the full 100 points deserve special praise—they definitely don't need artificial intelligence; they have more than enough of the natural kind.
What are the conclusions? I leave them to you. In my view, we would both have done decently on the Turing test, with all the questions that raises… or would one of us not, given that the teacher recognized the author? I won't say which, I don't want to risk offending him.
And now it really is time to say goodbye; my fellow school-leaver and I have an important new task. With this matura score, if our other grades that count toward university admission are decent too, we could certainly get into most university programs, including some quite prestigious ones… Time to start thinking about a little rented room.
Hold on tight, Studentski Grad, Sofia's student district; tremble, nightclubs—ChatGPT and I are coming!
