1910. That is the year Mother Teresa and Jacques-Yves Cousteau were born, while the world was still enjoying its prewar calm. But we have gathered here to talk about Bulgarian history. Exactly 1,910 people took part in the Digital Stories experiment, which set out to test how well we know Bulgarian history—whether we can tell undisputed historical facts from the textbooks from ones invented by an artificial intelligence whose goal is to mislead us.
Thirty statements—half historically true, half fabrications of the algorithms.
The results, at least to me, are impressive. That is why the time has come to sum them up and take a closer look. Did artificial intelligence manage to trip up the participants? Do we really know the Bulgarian past, such an important subject today? (The quiz is open again, this time without collecting results. You can try to solve it here before reading on, if you don't want to find out the correct answers.)
So: a full 1,910 people! My heartfelt thanks to every single one of them!
A Cautionary Tale
Let me hurry to tell you who was the first to fail the test… I confess, it was me. One more proof of how hard, and yet how important, it is these days to check historical facts. Having tried the quiz out on more than 10 relatives and friends, two of them historians, I was confident the statements had been verified. But on the very first day after it went live, emails and messages started pouring in, telling me that some of the statements I had marked as true were nothing of the sort. There was a wrong year, some inaccurate facts, even a historical cliché that I later debunked with the help of the historian Alexander Moshev in the article on the subject. Two of the statements had to be replaced entirely, and others had to be seriously refined. I take this opportunity to apologize to anyone I unintentionally misled!
While looking into whether ChatGPT could mislead us… I misled myself with no small share of the statements I had supposedly confirmed and checked! I think there is quite a lesson in that. Either way, thanks to the very possibilities technology gives us, the statements were corrected, and only a small number of participants took the test while it still contained inaccuracies. The answers to the misleading questions have also been left out of all the statistics I present to you here.
A Strong History
Time to turn to it. And here is the remarkable result! A full 36 people were not fooled by a single statement! Special congratulations to them and to everyone else with a high score! Of course, I am heartily glad that people with serious historical knowledge clearly took part (for the sake of the statistics, we shouldn't rule out repeat attempts, since the quiz was anonymous).
The vast majority of respondents took the quiz seriously. For the sake of precise statistics, in the patterns that follow I will exclude those who answered fewer than 20 of the 30 questions.
The average score is 20.55! Close to 70 percent of the answers were correct on average. In my view, that is quite a high figure, showing that people definitely knew a large share of the statements (which was not the case when I set them the task of telling generated texts and images from human-made ones).
The most common scores fall between 20 and 26 points. And yet fewer than 2 percent of those who took part identified all 30 statements without a single mistake!
Were the participants more inclined to take real historical facts for fiction, or ChatGPT's imagination for truth?
The results in the two groups are quite similar. In other words, the chance of failing to spot an invented historical statement is roughly the same as the chance of thinking a genuine one contains an inaccuracy. Of course, the quiz itself invites this, since it is a challenge precisely to doubt.
A Familiar History
As this list shows, ChatGPT chose to generate facts mostly about the Middle Ages—the era of the medieval Bulgarian khans and tsars—and far fewer about the Bulgarian National Revival, the eighteenth- and nineteenth-century cultural awakening under Ottoman rule, or about modern history. As in all the other Digital Stories experiments, my idea was to keep my own intervention to a minimum—to let the algorithm decide for itself where and how to take the task. Why exactly these historical facts were recognized by nearly 80 percent or more of the respondents, I leave for you to judge. But where did history sound most unbelievable? Which historical facts did the largest number of participants take for fiction?
Clearly, Khan Presian, the ninth-century ruler of the First Bulgarian Empire, Tsar Ivan Alexander and Enravota, the Bulgarian prince martyred for his Christian faith, are historical figures we don't know all that well. Perhaps that explains the difference from the previous chart—these are not the most popular figures in it.
But let's also see how things stand with the invented statements. There, too, ChatGPT had complete freedom to choose periods, figures and what exactly the misleading part would be. It could certainly have done better at its task if I had put more effort into the specific prompt. And yet every one of the fabrications, and of the facts as well, confused at least 1 in 10 of the participants who chose to answer.
An Invented History
And perhaps the common thread is once again fairly popular figures. Or else “facts” that sound far too unlikely, such as Tsar Smilets's collection of exotic animals and Tsar Mihail Shishman's order of scholarly alchemists.
As for the 5 fabrications that misled the most people, once again they come down to specific details. Personally, I was surprised that Tsar Peter I's university and the Knights Templar as allies of Tsar Kaloyan managed to fool quite a few of the participants.
A Short History
So, if we have to sum up everything so far:
-
A full 1,910 Digital Stories visitors took part, which makes this the most popular experiment on the site to date. Clearly the subject is extremely important and exciting for many people.
-
The average score shows that a large share of visitors managed to tell genuine historical facts from those invented by artificial intelligence.
-
Still, there were definitely quite a few statements that succeeded in misleading participants. One thing is certain: in the age of the internet and advanced AI, it is more important than ever to check, to question, to discuss and to give the floor to the experts.
An Important History
History turned out to be such an interesting and important subject that more digital stories about the past are definitely on the way. In case you missed them, I recommend all the pieces that have become part of the series:
And now for the promised prizes! Three of the participants who chose to enter the draw receive an autographed copy of the book Digital Stories as a gift. They are dimit******@abv.bg, l.hristo***@gmail.com and gerani***@gmail.com.
And once again, heartfelt thanks to everyone who took part! If you have ideas for future Digital Stories, I'd be grateful if you shared them here, and a modest prize awaits you. Thank you!
