Facial recognition turned out to be an extraordinarily successful application of artificial intelligence. It is one of the few branches of the field that has practically reached perfection. A few years ago Facebook became frighteningly good at working out who was in a given photo, and then… officially and quietly abandoned the service.
Today we can pass through passport control at the airport once the camera has “convinced itself” that the face on the passport is ours. In Moscow, for instance, you no longer need a ticket to get into the metro: the system knows at once who is standing in front of it.
By Their Faces Ye Shall Know Them
We are so spoiled by the conveniences of technology that this enormous breakthrough made no impression on us whatsoever.
The subject, however, is a vast one. First, because it is more than interesting to understand how the algorithms work that allow a sufficiently good photo to “say,” practically without error and instantly, which of the people in a database it shows.
And second, so that we can discuss when this might turn into a problem.
Face to Face
Every direction from which we expect a revolution in the world of artificial intelligence can be summed up like this. We have a large enough database, and specialists in the field apply their refined algorithms to it in order to find the patterns. Their goals can be all sorts of things, but in the end they boil down to just two. Either to predict what comes next in a given sequence on the basis of that pattern. Or to classify the next unit of information. In our case, that is precisely the goal: to determine whose face this is.
There are quite a few algorithms that do this. Some rely on ordinary machine learning, but the real breakthrough came when deep neural networks entered the race.
At Face Value
Here is how the algorithms work, regardless of which of their two generations we are talking about.
Their first task, without a doubt, is to “find” exactly where the outline of the face runs within the image they have been given.
After that… you might think the program goes looking, the way they did in old detective series, for a police sketch? For the ears, eyes, eyebrows, nose and cheekbones closest in shape and size?
Nothing of the kind. This is precisely where the power of modern technology, and of deep neural networks in particular, lies. To a large extent they abstract away from the details. Their main need is to have big enough data (and that data is the gold of our time). Then, gradually, step by step, layers of algorithms search for the patterns and pass them on to the next layers.
Strange Birds
Along the way, some images emerge that are strange, to put it mildly. A typical example is the psychedelic pictures produced by “deep dreaming” algorithms (here).
No less interesting are the results from the intermediate layers of facial recognition systems. As you can see, they are rather blurry images, again somehow as if pulled from the depths of the human unconscious. It is no accident at all that the work of neural networks is increasingly associated with the way our brain works.
The analogy becomes even more precise when it comes to facial recognition. Because our brain, too, doesn’t “remember” faces as a sequence of separate elements, but rather as patterns layered on top of one another at a subconscious level. Neither we nor the algorithm stop to wonder whether we are looking at the person with blue eyes, sideburns and… who knows what else; we simply say at once that it is them, because of the combination of proportions that make up the face. The distance between the eyes, the shape of the jaw, the space between the nose and the mouth… And so the classification is complete.
Face to face
Of course, for all this to happen with neural networks, the photos have to be turned into numbers, the only thing this type of system works with. There have long been sufficiently standardized approaches for this, and they work flawlessly.
After passing through the many stages of analysis, the algorithms reach their final result. Then they can tell us who is in the photo with incredible accuracy, which today approaches 100 percent if the photo is clear, of good enough resolution and shows enough of the face.
Just as when we see someone we know. The difference is that our “database” holds hundreds or thousands of people, while that of our “colleague” the algorithm sometimes holds billions, as was the case with the facial recognition Facebook offered.
Then came far harder tasks. Simulating and “creating” faces, for example. Here you can see how frighteningly good algorithms already are in this area too.
Meanwhile facial recognition, once a terrific luxury that helped us sort everything we had shot on our phones into albums… has clearly turned into an awkward subject.
Face-book
The way the service quietly vanished from the biggest social network is telling. After all, its very name comes from “face” and “book.” How did faces become awkward? What could the reason be? The recognition system worked practically without error.
And it still works, just not for the users. The fact is that today it isn’t particularly hard, with a large enough base of photos, to build a perfect recognition solution. It simply turned out to be an awkward undertaking that opens up a lot of questions.
Among the officially announced reasons was concern for personal data (ha-ha). Also the fact that people of color and Asians are confused by the algorithm more than 100 times as often, and that the systems identify people by the banal division of the sexes into men and women, with no way of placing someone who fits into the nuances.
American law has permitted and applied facial recognition algorithms since 2001, at that year’s Super Bowl. That year is considered the official first step of this type of technology. Even so, another decade would pass before the systems began working with sufficient precision.
If, in the middle of a crowd, say at a football game, a person committing a crime can be recognized, that’s a good thing, isn’t it? But the danger of total control travels down the very same road.
The Face and the Flip Side
Yes, the moment we bring up this subject, the first thing that probably springs to mind is China’s social credit system and the fact that in that country the enormous number of cameras, estimated today at 500 million, is an important link in the chain. Perhaps, but is the problem only there?
Facial recognition was a dream for a long time; today it is an inconvenient truth. The first projects on the subject in the US date back to the 1960s. Back then, though, the efforts went no further than scientists searching for patterns in faces, measuring with rulers and analyzing… with little chance of success.
Whereas in the second decade of the 21st century the algorithms turned out to be good enough to cope. In 2011 the systems helped identify and kill Osama bin Laden. In 2014 came Facebook’s service with the then rather cute-sounding name DeepFace. That same year facial recognition was officially used to crack several crimes in the US in spectacular fashion.
Also in 2014, Edward Snowden published documents revealing that the American government was collecting huge volumes of images, including from unauthorized sources, in order to build a perfect facial recognition system.
Then the subject more or less disappeared from public view.
Little Face
Today almost every new phone can be unlocked by seeing its owner’s face. There aren’t even neural networks involved here; it’s a matter of the much simpler algorithms of machine learning, which aren’t as precise and don’t need to be.
The market for software related to facial recognition is estimated at about $7.7 billion today, against $4 billion five years ago. The demand has more and more applications, and, as we have seen, the good performance of these systems has long depended less on the algorithms themselves than on the volume of data they have at their disposal.
How exactly the data is got hold of… is the interesting question. You need faces and names, and quite a few social networks already make it possible to get them. And that is exactly the problem: today, thanks to social networks, our photos are everywhere. This makes things easier for the algorithms and puts the trump cards in the hands of those same social networks, which in turn have access to even more photographs.
An amusing example of this is the story of the billionaire John Catsimatidis, who one October evening in 2018 was invited to a super-luxurious Italian restaurant in Manhattan to meet his daughter’s boyfriend. He asked the waiter to snap a photo of them as a keepsake and quietly stepped aside to find out who this young man was. He used the facial recognition app Clearview AI on his phone, saw that the would-be son-in-law was the founder of a dubious company in San Francisco, concluded he was a charlatan and told his daughter. And… he turned out to be right.
But is this really the future of human relationships? Or are we already living in the world of the good old dystopias?
On the Face of It
In China there are other applications too. At the end of 2018, in the city of Jinan, systems were unveiled that helped save toilet paper. Thanks to facial recognition, every visitor to a public toilet could get another 70 centimeters of toilet paper only 9 minutes after their first visit. Many meters of the luxury commodity were supposedly saved this way.
We won’t draw comparisons with the mobile toilets of Sofia, where 70 centimeters of paper would be an unearthly joy.
Classic facial recognition has reached perfection and is now digging into the details. For example, solutions that recognize a person from only part of a photo keep getting better, which is rather useful in years when masks became a compulsory fashion accessory.
Or take the recognition of faces that succeeds even across significant age differences. The algorithm can recognize you in your childhood photos, even though you are now a grown adult.
It turns out that technologies in this area cope surprisingly well with the help of certain additional developments. Apparently the basic features that carry our essence stay the same once we leave infancy. There is logic in this, because, as mentioned, the algorithms look for patterns and proportions, exactly as our brain does.
Persona Non Grata
When DeepFace was created in 2014, Facebook’s program recognized the same person in a photo with an accuracy of 97.25 percent, while humans were still leading the race, if only barely, with 97.53 percent.
In June 2015 Google took the lead with its FaceNet system, which achieved an accuracy of 99.63 percent.
At the risk of oversimplifying… perhaps it makes sense that these two particular companies took the serious steps, since they have access to larger bases of images?
Yes, there are huge volumes of freely available datasets online that can be used to train neural networks, but when it comes to the actual classification… it is logical that the leaders are those who have more photos as a potential result.
Face the face
A study by the US National Institute of Standards and Technology shows that in 2014 the error rate of such algorithms was 4.1 percent. By April 2020 it was only… 0.08 percent. With the important caveat, of course, that this applies to good-quality photos. If they are clear, clean frontal shots, like passport photos, the error drops to 0.03 percent. Well, that explains why it is now far more precise for a computer to “photograph” us and decide whether we are who we say we are than for the customs officer at the airport to do it.
And yet… not everything is that simple. In 2018 Newsweek ran a curious experiment in which Amazon’s facial recognition system… wrongly identified 28 senators as people who had been arrested for criminal offenses.
Alongside the refinement of the algorithms, solutions began to appear that make their job harder. The Russian inventor Grigory Bakunov, for example, offers special makeup that makes people unrecognizable to the software. His German counterpart Adam Harvey, for his part, sells special fashion accessories.
Another strategy holds that a mask and sunglasses are enough… I have never read about a sack over the head; maybe it’s worth patenting.
Masks Off
There are attempts “from the other side” as well. Fawkes is the name of software that invisibly alters tiny pixels in the images you upload to your social networks, so that the algorithms can’t recognize you afterward. Since images are turned into numbers, a small deviation in the right place might really make you invisible.
But whether that is really so… only a neural network knows.
In the US, the debate over the government’s use of facial recognition runs along the usual fault line, which has grown ever deeper in recent years. While he was president, Donald Trump regulated the technology’s use by border services, while cities that traditionally lean Democratic, such as San Francisco, Berkeley and Boston, banned the technology outright.
So far we have no such problems here in Bulgaria; the mayors of Kaspichan and Opaka, two sleepy little towns, are keeping silent.
Although the question points to eternal themes from the social sciences and ethics, what matters more today, in my view, are its aspects tied to how big companies apply it. And that debate always seems to be pushed aside by others that are far less important. Such as the notorious dispute over whether Facebook may store European users’ data across the ocean.
Whereas the more important question is how this data is used. And when it is turned against us, to control and manipulate us in one way or another. I hope we get to that conversation too…
