What would happen if we made three of the most powerful artificial intelligence models sit the matura, Bulgaria's school-leaving exam, in Bulgarian language and literature? Their efforts will be graded by two experienced examiners from the real exams. To make the contest even fiercer and more revealing, we're adding two real graduating students and me, the author of these lines.
Will ChatGPT, Claude and Gemini get perfect 6s—the top grade on Bulgaria's 2-to-6 scale—or will they fail with a bang? Will the teachers recognize which papers were written by artificial intelligence and which by a human? Who will get the highest grade and who the lowest? What will the results tell us about the most telling exam in Bulgarian education, about the progress of algorithms, and about the big questions that come with new technologies?
The invigilators walk in, the topic has been drawn. Time for the exam!
Maturing
The matura in Bulgarian language and literature—the compulsory exam with which we measure what young people have learned over 12 years of school—is a real… Turing test.
If you don't have time to read the whole piece and only care about the conclusions,
here they are in short:
- Three different AI models—Gemini, ChatGPT and Claude—passed the literature matura with an Excellent grade
- They did better than 90 percent of the real graduates
- Gemini (Excellent 5.68) beat Claude (Excellent 5.52), ChatGPT (Excellent 5.50) and even one of the humans who took part in the experiment
- None of them got a perfect 6
- I (Excellent 5.68) “got” a higher grade than my three AI rivals—Excellent 5.77
- The two teachers, who are also examiners for the real school-leaving exams, told the generated interpretive compositions apart from the three written by humans with great accuracy
- ChatGPT turned out to be a very good grader—its criteria were very similar to those of the two teachers
- The experiment holds many more details, nuances and things to think about, which are worth reading more about…
A year ago (you may remember) I challenged myself to step into this arena against ChatGPT. We both got Excellent grades and promised each other we'd carry on to university, or at least to a proper party in Studentski Grad, Sofia's student district, but… for one reason or another it didn't happen, and here we are on the exam bench again. To see what has changed in the matura… and in the world of artificial intelligence over the past year. To run the next, even more important experiment, adding more human and non-human rivals to the race.
A year ago our papers were graded by the young teacher Mr. Svetoslav Stoychev from the National High School of Mathematics and Natural Sciences in Sofia. This time he is in the referee's seat again, joined by a second grader—and what a grader! Mrs. Donka Mateeva is a senior teacher at the Romain Rolland Foreign Language High School in Stara Zagora and has been grading school-leaving exams for as long as they have existed in the Bulgarian education system. We'll also have a third grader—you may guess who—but… we'll get to that.
The Starting Grid
The rules are as follows: I (I'd be somewhere around 33rd grade by now, if anyone counted that way) try to solve the official paper from this year's compulsory school-leaving exam in Bulgarian language and literature (you can, too—here). Then I hold an internal contest among the AI models and pick the best three. Meanwhile I've asked two girls who have just graduated from high school to fill in the matura electronically, as close as possible to what they submitted at the actual exam. I make the 6 completed papers look identical, shuffle them and hand them to the graders.
I go first, so that I'm not biased and don't “copy.” Once again I choose to do the matura on a weekend, surrounded by playing children (you guessed it—I'm looking for an excuse in case I end up at the bottom of the rankings).
I have the advantage of having done last year's exam (well, and of having earned my living by writing for more than 15 years), but then I lack the special preparation, for example on the works included in the curriculum.
Even so, most of the tasks seem extremely easy to me. I still can't understand how so many of the students fail to scrape together the insultingly low 30 points needed for a 3, the lowest passing grade. A full 70 of the maximum 100 points come from multiple-choice or short-answer questions, without even bothering to write a text. I know not everyone has a potential man of letters dozing inside them (or at least not everyone knows it), but the insultingly low criteria (I'm joking) don't help anyone, I think. Anyway, that's my personal opinion.
Literati and Illiterati
Some questions give me trouble. “What does the title ‘Two Souls’ refer to in the context of the work”—I simply can't remember what this work is or whose. Then come questions about poems by Damyan Damyanov and “Honest Cross” by Boris Hristov—great!
I also, alas, have no idea about the notion of time that appears in “Prayer,” or whose prayer it is. There's also a poem by V. Popov… I'll turn out to be rather illiterate for someone who has been reading about 50 books every year since graduating: I don't know who V. is.
I reach the cherry on the cake, the dreaded “argumentative text.” I can choose between an essay on “Life: A Wheel or a Ladder” and “Reason and Feelings—an interpretive composition on The Peach Thief by Emilian Stanev.”
I choose to make it hard for myself and practice on one of the classics of Bulgarian literature. I think I know it well enough, and even if I didn't, the students receive generous excerpts that seem perfectly sufficient to me.
All that remained was to know how to write an interpretive composition…, but I interpret as much as I can, compose quite a bit and… done.
In total the students have a full 4 hours; I'm done after one (yes, you guessed it, I'm still looking for excuses. Maybe I should have written the essay after all… Or the “essay”?). Either way, no harm done. It's rather strange afterwards to read that over 1,200 students decided… to copy out the text instead of interpreting it. Clearly it earns no points, and writing, as I recall, is taught in first grade.
What's done…,
…is done. Now it's the electronic colleagues' turn to work. But fair's fair, as Orlin Goranov says—we don't want an easy win for the three humans in the contest. So, unlike us, the artificial intelligences have primaries. I give the first 15 questions, worth 24 points in total, to the leading AI models. Nobody is flawless, but they all do quite well (even the weakest performers would get a grade waaaaay above a 3). See the rankings here.
The favorites are clear and not particularly surprising. Google's Gemini 2.5 missed only 1 point (so did I, woohoo!)! ChatGPT and its daddy OpenAI will compete with the o3 model, and for better representation we give Anthropic's Claude a chance.
So now I have the papers of four “graduates”; what's left are the ones without the quotation marks. Quite understandably, in the season of proms and university applications, it takes a little longer than expected to complete the experiment and announce the results together with those from the real exam.
The Robot Swot
We've all been young (even ChatGPT), so I fully understand and thank from the bottom of my heart the two recent high schoolers who agreed to spare some of their time to fill in the matura once more (after they'd already got rid of the real obligation). The two young ladies chose to remain anonymous, and I wish them that the Excellent grades from their real matura will be the beginning of a meaningful education and future!
The six files are here; I shuffle them and off they fly to Mrs. Mateeva and Mr. Stoychev. I thank them even more—just when they could rest from the responsible task of grading countless real matura papers…, 6 more turn up, along with a challenge that is also extremely responsible.
We're even luckier because we're graded by a teacher from the capital, Sofia, and one from another big city, a young teacher and a colleague of his with enormous experience as an examiner.
I can't wait for the result—can you? Luckily, I don't have to wait long at all.
Ta-daaa…
A Matura Is a Matura
What happened?
All six contestants have Excellent grades—we beat over 90 percent of those who sat the real matura! The two real graduates get grades very close to those from their actual matura, which is one more sign of the grading skills of the two teachers. One of the graduates and I got Excellent grades, the rest are “Very Good.” Once again two of the “human” papers beat those of the algorithms, and yet… they're breathing down our necks, and Gemini is better than the third human contestant!
If we start looking in detail…, there are so many conclusions to discover.
You can see the completed papers: for Graduate 1, Karamanev, Gemini, Graduate 2, Claude and ChatGPT (poor thing, it didn't do quite so well this year; it will get its revenge next time, rest assured. Besides, it “scored” 84 points, which is just enough for an Excellent 5.50).
Sit Down, That's a 3
I have an Excellent for the second year in a row—drinks are on me! But before that there are some waaaay more interesting conclusions.
Let's start with the similarities. All six papers have few mistakes on the multiple-choice and short-answer questions. (I was in a hurry, you know…)
Gemini found a comma that I missed and still can't grasp—my apologies to the readers and to the Bulgarian Academy of Sciences.
Photo Finish
All three virtual and all three real participants chose to write the interpretive composition, which applies to only 1/4 of everyone who sat the matura. I put it down to our well-known self-confidence—The Peach Thief is hardly familiar to many graduates, and every year more and more of them choose to write the essay.
As for the average points… Yes, there isn't a single straight-A student among us, but the grades all six of us would get are extremely high. Far above the average of 57.53 points for all students, which is a little over a 4.
The positive takeaway from the real matura—that over 80 percent of students coped with extracting and processing information from a text and a chart…—I won't comment on how positive it is; in this part practically none of our magnificent six (not all of them people, you know) made mistakes.
Did the teachers tell the generated papers from those filled in by humans? One of the graders was flawless; the other was slightly misled and judged one of the human texts to be AI. I think, however, it is very telling that both rated the algorithms' interpretive compositions quite highly too. And in this case they knew they were dealing with generated texts as well—allow me to assure you that things change drastically if the jury doesn't know whom it's judging.
6,6,6
48,082 sat the real matura; there are fewer of us. If we'd been from the same school, we'd have been near the top by average grade.
And yet: six of us and not a single 6? This is the moment to boast that I would have had one. One of the graders gave me a full 97 points, which would automatically have turned into a round, fat, shiny 6. I'd have blown everyone away and been ready for Studentski Grad, but…
The other teacher judged that I'd taken too essayistic an approach, which I shouldn't have done after choosing the interpretive text.
You know what happens when there's a big gap between grades? We call in an arbiter. I don't have the nerve to take up another teacher's time, so let's see how ChatGPT itself, after topping the bottom of the rankings as a student, performs as a grader.
I give it the detailed instructions the teachers receive. Of course, I secretly hope my 6 will be rehabilitated.
But no. The algorithm gives grades quite close to those of the teachers! Slightly higher; it practically doesn't diverge on the closed questions, and it turned out to be a bit more generous about the quality of the texts.
Fun fact: the AI rates the AI graduates higher, and highest of all… itself. That's how it is—“modesty adorns a person” is a saying from bygone times.
And it doesn't raise the grade of only one of the candidates. Mine. Thank you from the bottom of my heart; I'll seriously think about whether to renew my $20 subscription.
“While the other students stand out with precision and completeness of argumentation, student 6 sometimes strays from the topic or gives overly general answers. His composition contains the most personal reflections and the least concrete detail from the text, which makes it less well argued than the others,” says the “grader.” Is that really what we're looking for? No reflection, even when we interpret a text?
But let's not be petty—the high grades are there.
Time to celebrate—come on, ChatGPT, Claude, Gemini, top of the class! A special night awaits us!
An Inseparable Class
Here are some of the teachers' impressions that impressed me (if you spotted the repetition, you have a shot at a 6 too).
“The students know the works and answer briefly but to the point. Their compositions are schoolish,” says Mr. Stoychev. “The style isn't polished enough, some mistakes have been made, but the texts are well structured. In two of the papers the AI is recognizable, because the answers to the literature questions are downright absurd. The compositions are too long, not well structured, and often repeat the same thing. One of the papers, though, gives itself away mostly through the language, not so much through the literature.”
Mrs. Mateeva even identifies the “authors” of two of the papers by name, and for one of them she is helped by an extremely curious detail. During the exam she ran an experiment of her own, and Gemini was the only algorithm to guess that V. Popov is actually… Vladimir Popov! And as she adds, “ChatGPT still can't create a text that meets the requirements for writing an interpretive composition! Still…”
Still… Because although the grades from this year's and last year's experiments are similar, you can also see the enormous progress in the way AI builds a text—in Bulgarian, a language exotic to it, no less.
The Last Battle
Here are the most important conclusions I drew from this experiment:
- The matura is designed so that the low grades are extremely easy to reach and the differences lie in the high ones. By lowering the criteria so much, I believe, we're not helping anyone.
- AI is getting better and better at solving the wide range of tasks this exam includes. The compositions it creates would undoubtedly be excellent with a more precise brief (prompt). That wouldn't have been fair for the purposes of this experiment, though.
- The teachers who took part in the experiment rated AI highly, significantly above the average grades from the exam as a whole.
- Isn't all this a reason to think about and discuss whether this really is the exam with which we can assess what a person has learned in 12 years of school? Whether they're ready for life? Whether they deserve to be admitted to a university and go further?
- Are these the skills and knowledge we're looking for, when algorithms are able to reproduce them to such a degree?
- AI turned out to be… a skilled grader. Isn't it worth learning how to use it to help us learn? Students and everyone else…
- In advanced AI we now have an extremely powerful tool that can be useful to us in an extraordinary number of ways. It's time for conversation, reflection and more unexpected experiments on how to use it in the best way!
- Over 90 percent of parents believe that education isn't adequate for today and that we need to change it seriously. We have one more serious reason to talk about how to prepare for tomorrow and even for today.
What the two examiners think, I hope we'll find out soon—I've invited them to tell us themselves in a follow-up on the topic.
What are your thoughts, impressions, conclusions? I'll be glad to hear and share them. As well as your ideas for the next Digital Stories. I look forward to you writing them to me here.
For making the experiment possible, I want to give special thanks to Mrs. Donka Mateeva, Mr. Svetoslav Stoychev, and the two graduates who gave their time. The help of Georgi Petrov and Evgeni Cherepov was invaluable. Thank you!
