Can you tell a human voice from one generated by artificial intelligence?
2,036 people took part in the Digital Stories experiment looking for the answer to this question, along with many other important nuances. The result, in my view, is unambiguous…
TL;DR
This article is long and detailed. If you're only interested in the main conclusions, you'll find them
here.
- Of the more than 2,000 people who took part in the experiment, the average score was 2.5 out of 6 possible correct answers.
- Had the participants simply guessed, they would have scored higher.
- It doesn't merely turn out to be impossible to recognize a generated voice. We are more likely to be fooled by it than to pick out the human one.
Is it a human or AI you hear in this recording? 82 percent of the participants in the experiment got it wrong. This is artificial intelligence, created by the venture of Assoc. Prof. Mariana Todorova and Ivan Vankov (Gatakka).
A Voice in the Wilderness
On this site you may have seen other challenges meant not only to test your skills and intuition, but also to show whether, and how far, artificial intelligence in its various forms has already advanced. Generated poems made it into the top 10 of two consecutive prestigious poetry competitions; an artificially created influencer appeared in the feeds of 85,000 people without anyone noticing that something was off. Perhaps you've taken the tests on telling generated from human-made poems, paintings, children's drawings, historical facts…
And in many of those experiments the algorithms turned out to be frighteningly good. Until we arrive at the results of this experiment, where for the first time the wrong answers… drastically outnumber the right ones.
Six short audio clips. The correct answers are highlighted in orange. Numbers 2 and 3 are human voices; the rest are generated. They were recognized by 51 percent and 34 percent respectively. The first of the generated voices proved relatively easy to guess (though more than 1/4 of people mistakenly thought it was human), and the others… see and hear for yourself.
72 percent of those who took the test judged this voice to be human. It isn't.
I Hear Your Voices
Clearly the time has come to admit that, after texts and images, AI-generated voices too can fool any of us into believing there's a human on the other end. How many important conclusions follow from this? If you follow the Digital Stories site, you can guess at waaay too many potential problems. (And opportunities!)
Two-thirds of the participants think this voice is generated, but it's human.
Is there a universal tell that could help us distinguish the generated from the human? Even if we find one, it's very likely to be a detail that will mislead us one step later, once the next generation of algorithms arrives.
If you were to look through the comments under the social media posts in which I invited people to join the experiment, you'd be convinced that anyone can easily recognize a generated voice. It makes mistakes, it gets the intonation slightly wrong, it doesn't sound good in Bulgarian…
Yes, perhaps a longer text would be a different challenge. There are always variations that could change the result. And yet, in this case, the result is pretty obvious.
Vox Populi
Before we part, it's worth pausing on some of the answers to the question in which participants were invited to write down their impressions of the voices.
- I'm scared that I can't tell them apart and I'm sort of answering at random.
- In my opinion, the male voices are harder to guess than the female ones, and I expect my mistakes to be exactly there.
- Very interesting!!! Thank you!
- It turns out to be a hard task!
- A wonderful way to raise awareness of the subject. Especially with phone scams on the rise.
- It's all artificial intelligence
- Frightening.
- AI has gotten pretty good.
- They're very close, but AI still pronounces things a bit clipped.
- They're very easy to recognize!
- I can't believe how much AI has improved. Most likely it's all generated.
- I expected it to be easier.
- I had the feeling the humans were slightly imitating the generated voices
- 5 is a human.
No, 5 is not a human, I promise!
(One of the comments clearly showed that in this case the test was filled out not by an artificial intelligence but by a natural one, which, however, is not in abundance: “КурЗаЛефски” (KurZaLefski, a misspelled obscene chant against Levski, a Sofia football club).)
How the Echo Fades
So here, once again, are the conclusions in brief:
- Of the more than 2,000 people who took part in the experiment, the average score was 2.5 out of 6 possible correct answers.
- Had the participants simply guessed, they would have scored higher.
- It doesn't merely turn out to be impossible to recognize a generated voice. We are more likely to be fooled by it than to recognize the human one.
- If you haven't yet tried to recognize the voices, you can challenge yourself here.
It's also time for the promised prizes! The three participants who got all 6 answers right and were chosen by a draw are: ivana.****@ka***.bg, di***@gmail.com and pt***@gmail.com. They will receive a selection of books and congratulations on their impressive result!
If you have ideas for the next journalistic experiments on Digital Stories, I'll be wholeheartedly grateful if you share them with me. A modest gift awaits you too!
