“AI Rev­olu­tion!” When the Hype Goes Too Far

Artificial intelligence can extend our lives dramatically, and make them better in so many ways. We have told you quite a few Digital Stories about this, and many more are on the way. I believe that one day it may replace (or at least complement or assist) journalists, programmers, even politicians. It helps us forecast the pandemic. Lately, though, the phrase has been thrown around so recklessly and with such marketing zeal that it really pays to keep our wits about us whenever we hear it.

The more fashionable a field of knowledge becomes, the more attractive it gets for scams—just ask Ostap Bender, the charming con man from Ilf and Petrov's Soviet satirical novels. From bloodletting and leeches to nanotechnology, it is very cool to throw on the cape of ultramodern technology and make a killing. Sometimes all too literally.

Yes, it is harmless, if a little naive, to slap the label on everything in search of cheap fame. Yes, there is no need for a ladle for tripe soup to come with artificial intelligence. And that is no small part of the problem—the reckless use of the term erodes the reputation of the serious projects too.

It gets worse, however, when it comes to particularly sensitive fields, such as medicine. AI has made its way into many corners of it today… But what happens when the technology we have trusted too much starts failing at its tasks? And forgets the most important part of the Hippocratic Oath: Primum non nocere. “First, do no harm…”

Prof. Gary on the Road to Nowhere

Prof. Gary N. Smith is one of the few scientists trying to draw attention to exactly these topics. He works on artificial intelligence and economics, and is the author of three books published by Oxford University Press. The main leitmotif of his work is a warning: AI is coming to change our lives. But if we put too much trust in everything that hides behind its attractive name, we will suffer. For The 9 Pitfalls of Data Science he even won the Association of American Publishers' award for popular science.

Here, for example, he gives a particularly telling example.

Epic Systems is the largest company in America that processes data on people's health. That is, to put it mildly, an impressive achievement—you know how strict the regulations on storing and analyzing personal data are, let alone the most sensitive information of all, the kind related to health!

And against that backdrop, Epic officially and downright epically has access to the data of 180 million people, a full 56 percent of Americans!

“With the patient at the heart,” reads the company's highly emotional slogan. Its portfolio includes 20 polished algorithms based on artificial intelligence, designed to predict or confirm a diagnosis, or to estimate how many days a given patient will stay in the hospital.

Crows and Algorithms

As Prof. Gary Smith notes, all too often we hear that something is the work of the modern magic called “artificial intelligence.” And since it is a “trade secret,” a “patent worth millions” or some other priceless thing, it is unclear what exactly happens inside the algorithm's black box. Nor is it possible to run even basic checks against specific external data.

One of the main callings of the epic Epic is predicting cases of sepsis, a sudden infection and the leading cause of death in hospitals. As Prof. Smith writes, “sepsis occurs when the human body overreacts to an infection and sends chemicals into the bloodstream that can damage tissues or prevent organs from functioning normally. Early diagnosis can literally save lives, but… sepsis is very hard to diagnose early.”

And so Epic has a particularly promising algorithm, based on artificial intelligence, that it says can predict the risk of sepsis with an accuracy of between 73 and 84 percent!

Great, right? So many lives saved!

Except that very recently, on the first day of August, the respected journal “JAMA Internal Medicine” published highly curious findings from a team of researchers who studied the hospital records of 38,455 patients treated at the University of Michigan hospital. Of them, 6.6 percent, or 2,552, actually developed sepsis. But how did the system that was supposed to predict whether trouble was coming actually do?

The following simple little table says it all. Across the top, “Epic+” and “Epic-” mark the algorithm's predictions of whether a given patient would develop sepsis, and down the side is the actual outcome.

An Inhuman Error

Trial and error. The numbers more or less speak for themselves about how far from the truth the algorithm turned out to be.

Still, Prof. Gary Smith highlights two key relationships:
– Out of 2,552 patients with sepsis, the system generated a warning for only 843, barely a third.
– Out of 6,971 warnings, the prediction was confirmed in only… 843 people. Twelve percent!

Looked at the other way round: the system failed to detect 67 percent of the cases, and 88 percent of the warnings it issued turned out to be wrong.

In other words, even if you just sat there flipping a coin, your hit rate might well turn out to be considerably better.

Now imagine that this algorithm, on which enormous sums have been spent, is expected not merely to warn, but that patients' health somehow depends on it directly… Downright frightening, isn't it?

Epic

Prof. Smith also notes that many of the other systems developed by Epic Systems are in fact far, far less convincing in their predictions than their promises suggest. And the researcher warns that it is definitely not a good idea to hand over access to vast amounts of health information to multibillion-dollar projects built on supposedly AI-based solutions. Solutions that, in reality, turn out to be far, far clumsier than their glossy packaging.

Systems like these ought to be tested and evaluated more carefully by scientists before they are put into practice. And by independent teams, at that. Rather than simply having a black box that takes in some data and “spits out” a result.

The problem, of course, is by no means limited to one company.

Dr. Watson at the Unemployment Office

Still on the subject of medicine, it is worth mentioning the Dr. Watson project. Yes, this is an IBM venture based on the very same Watson supercomputer that kept beating humans in all sorts of competitions. Here, though, it lost.

One of its creators' ideas was to bring it into medicine. It took five years and $60 million in development before “Dr. Watson” was “fired” from the MD Anderson Cancer Center in Houston over “multiple examples of unsafe and incorrect treatment recommendations.”

One last medical example. In March of this year, the respected journal “Nature” published a comparative analysis of 2,212 different studies that used machine learning models in an attempt to predict or diagnose the course of the coronavirus.

The result: “None of the models identified are of potential clinical use.”

Rudebots

Quite a few AI projects, meanwhile, fail rather spectacularly, even comically. The proverbial example is the chatbot Tay, a promising Microsoft project from 2016: a technology running a Twitter account, meant to hold conversations just like a typical American teenager.

Well, we don't know what is typical of most teenagers, but Tay quite quickly revealed some deeply frightening sides of its character. A few hours after launch, it was already declaring that it hated Jews and that Hitler had been right. The chatbot was shut down less than a day after the project went live. It turned out that malicious users had exploited weaknesses in the way the algorithm learns to make it talk like a racist and a misanthrope.

You'll say this story is pretty old by now. True, but it is telling, and it makes us think. What if, in the same way, someone could manipulate, say, the AI in charge of driving autonomous cars?

Lawmakers and Lawbreakers

We have all heard about China's flawless facial recognition systems (although there really is more than one point of view on the subject). Well, in 2018 the police mistakenly arrested the wife of one of the richest people in the country, because the AI wrongly identified her as a driver responsible for a very serious crash. What's more, her face appeared on the giant screens that publicly display the worst rule-breakers as a form of public shaming.

Amazon has a similar problem. A journalistic experiment showed that its facial recognition system mistook 28 members of the US Congress for criminals. What's more, the algorithm identified 40 percent of the people in the photos as African American, even though in reality they make up less than 20 percent of the members of Congress.

We end the list with a bit of forecasting. Before the last FIFA World Cup, all sorts of enthusiasts tried to predict the winner using every kind of model and algorithm. In the end, their results were compiled. Brazil, picked as the winner by the largest number of models, made it to the quarterfinals. And the other two likeliest winners, Spain and Germany, went out even earlier.

So as not to overdo the pessimism, here is an example of AI doing its job beautifully, but… that leading to serious trouble. The chief financial officer of a UK-based energy company transferred €220,000 to a Hungarian supplier. He was simply doing his job, because he had received a call from the German who was the top boss of the company, telling him to do it. Later, though, it turned out to be an audio version of deepfake technology, in which artificial intelligence had precisely mimicked the boss's voice. “It had exactly the same German accent and the same melody in the voice,” the victim told The Wall Street Journal.

Oh Algorithm, Algorithm…

As Prof. Smith, with whom we began, warns in another of his articles, artificial intelligence has become a mantra, a marketing cliché, hype, as the younger crowd would say.

He also tells us the story of Justin Wong, who graduated with good grades in a bachelor's and a master's degree in information technology and in 2017 founded a Singapore-based company called Scry. Its goal was to become a “social forecasting platform.” Users sign up for free and can enter their own predictions about the likelihood of a given event. Who will be the next president of the United States? Will Musk manage to “ship” people to Mars within the next decade? It relies on a principle known as “the wisdom of the crowd,” which we have already written about together with blockchain technology.

Soon he took the next step: Scry Vantage, a system that makes those very same predictions. Only with the help of an algorithm that is, of course, also locked inside a black box. And every prediction made with it now costs $400.

“We call it AI, that's all you need to know” is the principle on which quite a few such projects operate. But is there really a serious, scientifically grounded and verified algorithm behind them? Or are we just playing eeny, meeny, miny, moe and looking up the answer on Google?

Algo-Kick Me!

In fact, Prof. Smith also puts GPT-3 on the list of such projects, which is certainly far more debatable. This technology really does look quite promising; after all, all of us can test it today in quite a few different ways.

One more reason to be more suspicious whenever we hear the AI “mantra” comes from this study from 2019. According to data from one of the leading companies in the field, Pactera Technologies, a full 85 percent of artificial intelligence development projects in various fields fail. They never manage to deliver what they were created for. The percentage is truly serious, and the researchers offer quite a few possible explanations. On the one hand, the difficulty of adapting to such major changes; on the other, the overly ambitious goals being set, given that this technology clearly still has a long way to go before it can offer even part of the impressive things we expect from it.

Artificial intelligence is definitely a subject on which it pays to be cautious. The line is very thin between the serious projects that can genuinely improve all our lives and the outright fakes that breed prejudice against the whole field.

When I present various Digital Stories about artificial intelligence, perhaps I sometimes get carried away predicting an overly optimistic future. The subject invites it. And sometimes this kind of technology astonishes us with its progress. Suffice it to mention facial recognition, machine translation, autonomous cars.

A, I… So What?

That is why on subjects like this I try to present balanced stories, in which I go overboard neither with optimism nor with skepticism.

“Never invest in a business you cannot understand,” says Warren Buffett. And it is a fact that the danger of being played is sometimes much greater when modern technology is involved. Suffice it to mention the legendary multibillion-dollar blockchain scam OneCoin, which started in Bulgaria.

Unfortunately, this applies with full force to artificial intelligence as well. Wholesale or on the tiniest scale, slapping the fashionable label on every trifle. It matters a great deal what exactly lies behind the “AI” tag in each specific case. Of course, it is important to believe in the serious development of technology and to be prepared for the future that is coming. But still, it is always good to keep our wits about us…