Can we tell today what was made by artificial intelligence from what was made by a human?
Let me start with the short answer: we can’t. That is the main conclusion of the experiment held on the Digital Stories site.
There are, however, many other important details that should make us stop and think.
Let me take this opportunity to thank from the bottom of my heart every one of the 1,816 visitors who filled in the 30-question survey (here). I owe you an apology for the difficulty—I love a competition too, but this one, by all appearances, is already an unfair fight.
And one more apology—that all participants who wanted to receive their results and the correct answers by email had to wait until today, as stated in the rules.
(Because of technical limitations in Google Forms that the platform never announced, sending the answers by email has been delayed. If you would like to receive your answers, write to me at [email protected] or on social media. You can also take the test again, even with blank answers, and you will see which ones are correct. You will also find the correct answers at the end of this text.)
Did Hieronymus Bosch or Petko Slaveykov prove harder to recognize? Maria Zhekova’s dishes or Violeta Kuneva’s stanzas?
The experiment is over. You can now take the test again, and you will get the answers as soon as you complete it. Here are the conclusions.
Gauss? Chaos!
A total of 1,816 people took on the challenge. The record—26 points, or “only” 4 wrong answers—was achieved by three of them. Only 26 people, or just over 1 percent of participants, got more than 23 answers right.
Yes, the questions turned out to be extremely difficult. We clearly already struggle to tell the generated from the created. Of course, the experiment makes no scientific claims; it is simply a journalistic check, a provocation of sorts, to see how good image- and text-generating algorithms really are at their “job” today.
Honestly, I regret that I couldn’t take part in the experiment myself. Naturally, nobody was prevented from looking up the answers online either—(almost) all the original images and words can be found there.
No, don’t blame me—blame Carl Friedrich Gauss. What we see on the chart above is precisely a “normal,” or Gaussian, distribution. Named after the 17th-century mathematician, it describes the distribution of random variables clustered around the mean. In other words: data telling us that even by pure chance we would get the same chart.
Anyone who guessed blindly and answered “AI” to everything would have “scored” 16 points. And the average score of all participants is 16.07…. (Participants with fewer than 10 points were excluded from the statistics, since they obviously hadn’t filled in a large part of the test. I should also point out that among the rest there were very few cases of someone skipping a question.)
But what other curious conclusions are there? Did the texts or the images prove easier to recognize?
Wing or Drumstick?
As you can see, there is practically no difference. With an average of 54.94 percent correct answers (let me remind you again that 50 percent would be pure guessing), the figures for the images are very close. Of course, it was no accident that the test used works by authors from very different fields who had given their consent. So people familiar with the style of the rapper MC Atila, the poet Violeta Kuneva or the artist Maya Bocheva (Wiki) shouldn’t have struggled much with that particular content, but they would have been on equal footing with everyone else when it came to all the other questions. The same goes for Maria Zhekova’s recipes and Dimitar Karanikolov’s photographs.
And so, whether because of the particular choice of questions or the nature of the particular art form, the photographs turned out to be the most recognizable (though only for fewer than 2/3 of participants). The hardest were the illustrations, where wrong answers significantly outnumbered right ones. In as many as three of the categories—dialect words, paintings and illustrations—the incorrect answers even exceed the correct ones.
Only 38.8 percent recognized the AI translation of Shakespeare, set alongside those by Valeri Petrov and Geo Milev, two of Bulgaria’s great poets and translators—
To die, to sleep—
no more; in sleep to put an end
to the heart’s pain, to the countless blows
that are the lot of flesh.
A Recipe for Success
The verses of Petko Rachov Slaveykov, the 19th-century poet and enlightener of the Bulgarian National Revival, proved relatively recognizable—on average, 68.15 percent correctly told which were his and which were generated.
closeness lands blows below the belt
a gift I cannot carry
without your help
that I cannot refuse
The poet Violeta Kuneva or artificial intelligence? Yes, the human—but only 36.2 percent spotted it; over 2/3 of people judged the text to be generated.
- Бръндилник (brandilnik) – a primitive device for making homemade rakia, Bulgarian fruit brandy, and other spirits at home
- Чирикливо (chiriklivo) – describing music or a song with a cheerful, playful character, typical of outdoor gatherings
Which of the two is a dialect word from northwestern Bulgaria, and which is a “fake”? Both were “invented” by ChatGPT, and the correct answers hover very close to 50 percent (though still below it for both), which is to say: total confusion.
I See Hell, I See Heaven
Which of the next two stanzas is the work of the rapper MC ATILA, and which of artificial intelligence?
When I look back,
I see hell, I see heaven.
A dragon’s breath on my neck,
meaning I walk alone down the middle—yin and yang,
black night and day as white as milk.
or
Time flies, but I’m above it,
beat runs in my veins, I don’t give in easy.
Memories in rhymes, my life’s a drawer,
every verse is magic, opens doors without a key.
The first is “human,” the second is not. And yet, over 50 percent got both answers wrong.
Parting Words
Having already seen that artificial intelligence today can generate an “online person” on its own, now it is confirmed: much of what it creates is already indistinguishable.
Can you think of other experiments to test its abilities? If so, write to me here!
Next Tuesday, look out for a follow-up—an interview with commentary on why it actually matters that today we so often cannot tell the created from the generated.
You can see and take the test here.
It’s also time for prizes! Three of them go to the participants with the highest score in the experiment—26 out of a possible 30 points: rad****.char*****@gmail.com, so***.tr*****@gmail.com and mi****.ant***@gmail.com. After a draw, the other two prizes go to the participants with the emails kots***.zl***@gmail.com and evelinad*****[email protected]. The prizes are two signed copies of the Digital Stories book and two T-shirts with the site’s logo, plus one more—the book “Zagadkite na Atanas Atanasov (The Riddles of Atanas Atanasov),” also signed.
Special thanks to the friends of Digital Stories who agreed to take part in the experiment:
- MasterChef winner and photographer Maria Zhekova (her visit to Digital Stories—here),
- The poet Violeta Kuneva,
- The artist Maya Bocheva (Wiki),
- Photographer and architect Dimitar Karanikolov (his visit—here),
- Orlin Milchev, formerly also known as MC ATILA (his visit—here).
The experiment is also carried out with the support of the Bulgarian Center for Not-for-Profit Law.
Correct answers:
(Photographs) 1. Dimitar Karanikolov; 2. AI; 3. Dimitar Karanikolov; 4. AI;
(Poems) 5. Petko Slaveykov; 6. AI; 7. Violeta Kuneva; 8. Violeta Kuneva; 9. AI; 10. AI;
(Photographs) 11. AI; 12. AI;
(Recipes) 13. AI; 14. AI; 15. Maria Zhekova; 16. Maria Zhekova;
(Translations) 17. The third; 18. The second;
(Rap) 19. Atila; 20. AI; 21. Atila; 22. AI;
(Hieronymus Bosch) 23. AI; 24. AI;
(Dialect words) 25. AI; 26. AI;
(Illustrations) 27. Maya Bocheva (Wiki); 28. AI; 29. AI; 30. Maya Bocheva (Wiki)
