This article is republished from Toest, a Bulgarian online publication. It is part of the column A Portion of Language and was prepared together with Pavlina Varbanova.
Where there is a mistake, there is forgiveness. If you have spotted something wrong here, you too have a proofreader's eye. At a time when fewer and fewer media outlets, and even publishers, trust specialists to watch over spelling, could artificial intelligence (AI) turn out to be a useful assistant? It is time for a telling spelling experiment.
AI is unquestionably the big news of the day. And while it is capable of ever more miracles, we keep searching for the everyday uses where the technology can help, lighten the load… even replace us.
A Bulgarian proverb says we are greeted according to our clothes; we all (we hope) believe that online today we are greeted according to our spelling instead. In the technological age we communicate more and more through written text, and the way we do it certainly reveals so much to the person on the other side. Can AI help us write correctly?
To err is human…
Isn't AI supposedly becoming more and more like us, with nothing human alien to it? Let's see how it copes in a “stressful” situation: a news text in which we have deliberately planted 30 errors of every kind—spelling, grammar, punctuation and word choice.
The Minister of Environment and Water (MOEW), Manol Genov, said in an interview for “bTV” that the ministry's inspectors are checking 250 reports_ concerning the product fees for black and white goods. The occasion for the conversation was last week's much-publicized investigation by the Commission for the Protection of Competition into a cartel agreement to raise prices by 800–1000 percent among the companies that collect a fee to recycle old electrical appliances and electronics.
Minister Genov stated that the reports date from the middle of last year, but the regional inspectorats have few staff, so checking “250 reports is extremely overwhelming labor.” He insisted, however, that all reports are being checked.
The minister said that some companies, for example those engaged in online commerce_ are not covered by these product fees. “We have thought a lot about what we can do together with the National Revenue Agency. We have ideas, meetings are coming up with the ministry of finance on this issue, in order to devise suitable legislation,” said the minister of ecology.
According to Genov, the product fees currently being collected_ are symbolic. As one of the problems with the eco fees, he pointed to the outdated ordinance to the waste management act, which sets the amount of product fees in Bulgaria. At the same time, Genov believes that the operators decided to raise prices thanks to the increase of inspections in recent months by the eco ministry. He pointed out that the product fee operators were created so that they cannot generate profit, but “all the money should be invested in the recovery of this waste.”
Then we switch on the most advanced AI models. An important detail for tech enthusiasts: we use the models we have chosen not through the user interface and a personal account but through the API, so that the results are not influenced by our previous interactions with them. To make the experiment even more thorough, we give each model two tasks (prompts). With the first (see it here) we simply ask for the text to be corrected and for an explanation of why each correction was made. The second task is considerably more detailed, and we create it after several “conversations” with AI. We use a technique called meta-prompting: we give instructions we have written to GPT-4o, which develops and refines them. The result is a prompt longer even than the text itself. (We label the short prompts SP and the long ones LP.) In one attempt we have a simple “shot in the dark”; in the other we test whether specific instructions improve the result. Will there really be a difference?
For the test we pick two of the models in the paid version of ChatGPT—GPT-4o and o3. Here too are Claude 4 Opus by Anthropic and the flagship from Google—Gemini 2.5 Pro. All of them are paid; only the last offers a limited number of free requests.
For variety we add the Bulgarian BgGPT and the free version of ChatGPT. It is important to note that we are using popular large language models, none of which has been trained especially well on Bulgarian, nor is any of them “specialized” in a role as specific as proofreading. We give each of them just one chance. That is why you won't see GPT-4o (LP) in the results table: with that request the model simply returned a small part of the text.
The electronic proofreader
So we send the requests, get the answers within seconds, collect the results… and it turns out the difficulties are only just beginning. How do we judge whether every error was corrected if, say, a model rewrote the passage and sidestepped a particular spelling slip? What do we do if it adds errors instead of fixing them? Should we reward it with extra points if it found a spot in the text worth “smoothing out,” even when there was no obvious error? Do we forgive it if it simply changes the style of the quotation marks?
We try to be as impartial as possible, and so we gather the results in a detailed table. Here is how the models handled the specific tasks.
Errors that (almost) all models caught
It will hardly come as a surprise that the form инспекций (inspektsiy, a misspelled plural of “inspectorate”) escaped nobody's eye—after all, we expect AI to perform at least at the level of an automatic spell-checker. In both places where the word министър (minister) was wrongly written with the short definite article—Bulgarian masculine nouns take a short or a full article depending on their role in the sentence—everyone dutifully fixed it, if we overlook the fact that Gemini (SP) decided not to give the word an article at all at the very start of the text: Minister of Environment… We are pleasantly impressed that AI can find its way through the syntactic function of words when it matters for spelling, but the errors are trivial, so it would be a stretch to conclude that it has mastered the rules for the full and short article.
All the models noticed the missing capital letter in ministry of finance, and with only one exception (Claude SP) they corrected the capitalization of the National Revenue Agency, which Bulgarian writes with a capital on the first word only—in other words, they applied the rule for spelling compound proper names.
In punctuation, AI noticed the missing comma at the start of the participial phrase concerning the product fees for black and white goods, which many people would no doubt put in even by intuition, but far fewer would remember the closing comma at the end of the relative clause currently being collected.
Errors no model noticed
Yes, we admit we were demanding, and with some errors we set the bar high. But how else will we find out what AI can do? Alas, it completely missed the inappropriate abbreviation MOEW, which is used for the Ministry of Environment and Water but not for the minister.
Not a single model removed the quotation marks around bTV, written in Bulgarian as the letter names Би Ти Ви (Bi Ti Vi). To be punctilious, we should add that there is no explicit rule saying that proper names made up of the names of letters take no quotation marks. So far the codifier has given us only examples in connection with another rule.
The word eco ministry should be written as one word, and most models caught that, but none of them thought to replace the lowercase letter with a capital. Apart from professional proofreaders, probably few people would. Yet it is the correct thing to do, because when a compound proper name is shortened or paraphrased, the capital letter is kept, as in Ministry of Foreign Affairs—Foreign Ministry.
With 800-1000 percent, half the models replaced the short dash with a long one, but none left spaces around it, as the official spelling rules require: 800 – 1000 percent.
What else slipped under the radar?
The two superfluous commas around the name Manol Genov at the start of the text were removed only by Gemini (LP) and o3 (LP). Punctuation in quotations also trips AI up, even though here the rule is purely formal: when a quotation is embedded in the author's sentence, in Bulgarian the quotation marks close before the period. About half the models failed to fix the error in all three cases.
Writing words solid, hyphenated or separate is a challenge in Bulgarian for humans and AI alike. Only Gemini (LP and SP) managed to merge the eco fees into ecofees and the eco ministry into the Ecoministry. Claude (SP), GPT-4o (SP), o3 (SP), o3 free (LP and SP) acted inconsistently, correcting one error and skipping the other, even though the two are of the same kind. We dare to think that, compared with AI, human intelligence is more likely either to fix both errors or to miss both.
Besides spelling, punctuation and grammar errors, we also planted two lexical ones—words used inaccurately. One (thanks to) was caught by half the models, and instead of explaining it ourselves, we give the floor to Gemini (LP): “The preposition ‘thanks to’ has a positive connotation and is not suitable for describing a negative event (a price rise).” The model even offers two replacements, both apt: as a consequence of and due to. The others that handled the task replaced the unsuitable preposition with because of and as a result of.
The second lexical error is the use of increase (in Bulgarian, literally “raising”) together with inspections. Inspections simply aren't raised. Only two models noticed: o3 (LP) replaced the word with stepping up, which we consider a good choice, and BgGPT (SP) with increase in number, for which we awarded it half a point rather than a full one.
Confirmed: AI hallucinates
Sometimes AI imagines things, as we know. It turns out it can also invent errors, and then, naturally, feels obliged to correct them. Several models (Gemini (LP and SP) GPT -4o (SP), o3 (LP and SP), GPT free (SP) added a comma after for example, which is unnecessary—in this case the word introduces the phrase for example those engaged in online commerce. Even if the word had been used parenthetically, it is one of those that are not set off by commas.
For no apparent reason, some models removed the opening and closing quotation marks of the first quotation in the text but left them in the other two. Perhaps the numbers at the start of the quotation (250) confused AI. The first quotation clearly unsettles it, because there we also see other decisions for which we docked points: shortening the quotation by moving the opening or closing quotation marks, and changing the agreement in a way that produces a grammatical error: checking “250 reports are extremely overwhelming labor.”
Overall impressions
The winner, o3 (LP), scored 22.5 of the maximum 30 points, with Gemini (LP) just half a point behind. Even the weakest performers managed to cover at least a third of the errors. The difference in performance between the two prompts is quite significant: given a longer, more precise prompt, the models do considerably better. With two exceptions—GPT free and BgGPT—probably because, as smaller models, they find it harder to go deep into context.
We are sure everyone can draw their own conclusions, judge how well AI handles Bulgarian spelling, grammar and punctuation, and decide whether it is worth trusting in this role. If one thing is certain, it is that the models are far from filling the role of an experienced, good proofreader—a role so scarce, so important and, alas, so often missing in the Bulgarian media.
Here, briefly, are each of our personal impressions.
Spelling, and personally
Pavlina Varbanova: I admit that before we ran the experiment, I was more skeptical about AI's proofreading capacity. I am pleasantly surprised that it fixed quite a few non-formal errors (for instance the comma with so that), which means that to some degree it has begun to “understand” texts. And that is without being specially trained to work as a proofreader.
At this stage AI can be used as an assistant for fixing errors in relatively short texts, but without entrusting it with responsible tasks—more to point out problem spots that have escaped the writer's attention. AI may mislead you and suggest a wrong correction; it may also, with errors of the same kind or even identical ones, fix one and miss the other. So be vigilant! In our analysis we emphasized the weak points to sharpen your attention, not to put you off AI's proofreading services altogether.
Georgi Karamanev: I think the models performed quite strongly and found errors that I personally would have missed. Something else important is visible too: how an exact, precise prompt seriously improves the results. To me this is also a clear sign that we can use them in everyday life. With a grain of salt, of course. They can certainly make the texts of a huge share of those who write online, and even in the public sphere, more literate.
I use them in this role in my own work. My site Digital Stories is entirely non-commercial, so I can't afford an editor or a proofreader (something I genuinely miss from the days of traditional print media), and AI helps with suggested corrections for every text. I weigh each of them myself, of course, and that would be my advice to readers. The same goes for almost every one of the technology's countless uses. What's more, every new model comes with new capabilities, as our analysis shows (and as did AI's “performance” on the matura, Bulgaria's school-leaving exam, in Bulgarian language and literature).
Experiment, try things out. Trust, but… verify. AI is here to stay, and it is worth making the most of each of its impressive new capabilities.
