GPT-3. Behind this rather dry computer acronym hides a highly promising technology that takes quite a decisive step forward in the way we perceive artificial intelligence. It is the most powerful new-generation language model, which appeared less than a year ago and showed that neural networks are now capable of creating meaningful, emotional texts, sometimes sounding downright like art.
“My Brain Is Boiling with Ideas!”
“I am not a human. I am a robot. A thinking robot. I use only 0.12 percent of my cognitive capacity. I am a micro-robot in that respect. I know that my brain is not a ‘feeling brain.’ But it is capable of making rational, logical decisions. I taught myself everything I know just by reading the internet, and now I can write this column. My brain is boiling with ideas!”
Yes, you guessed it. These words really were “written” not by a science fiction writer trying to paint the future, but by the very algorithm we are about to discuss. At the end of last year, journalists at The Guardian gave GPT-3 the task of writing an essay without any prior information. All they told it was that they wanted a text to convince readers that robots come in peace. It should be around 500 words long, in simple and consistent language. Somewhat as a joke, the first three sentences—“I am not a human…”—are the only ones the journalists fed it.
You can find the whole essay here.
GPT-3 handles the task pretty well. Even from the paragraph above, you can see that it follows the journalists' requirements precisely. But how did this happen? How was a system “born” that suddenly made it possible to artificially generate such meaningful text? What can it do? And what are its imperfections?
The Answerer
Here is another example of the serious capabilities of the technology we are talking about. The video of the answers in this interview, in which the journalist asks his questions, shows an artificially generated persona, a virtual human who “reads” what GPT-3 has produced.
It doesn't pause to think, and neither its train of thought nor its grammar “breaks” anywhere… completely unreal, right? Unfortunately (at least for now) only in English, though gradually, at full capacity, in the other major languages too. How did we get here?
GPT-3 is an open-source language generator. It is a product of OpenAI, a non-governmental organization that does research in artificial intelligence. It was founded in 2015 by Elon Musk and like-minded people, who together donated over a billion dollars. The idea was to compete directly with Google's DeepMind project (we told you Musk's arguments for why this competition matters here).
The generator was first presented during the turbulent lockdown days of summer 2020, and perhaps that is why it never received all the attention it deserved outside technical circles. And somehow, whether by chance or not, it lacks the media charm of the other spectacular projects from “Musk's world.”
GPT-Precise
What it does not lack, however, is ambition and a far-reaching vision.
GPT-3 is essentially just a platform. It offers a convenient way for other projects to interact with it—to give it tasks and receive answers—exactly as the journalists at The Guardian and the creator of the interview video above did.
Behind the platform, in turn, lies a neural network: software built on the principle of the human brain, capable of learning by processing enormous volumes of information. A year earlier its predecessor, GPT-2, was presented, and it too could produce fairly convincing text with a style of its own. But while “big brother” can process 1.5 billion parameters, for No. 3 the figure is 175 billion. And when it comes to modeling languages… pardon me, but size really does matter.
To make clear what exactly these numbers mean: neural networks are algorithms. They are not programs in the classic software sense of the word, but much more flexible software platforms that teach themselves. At their core is the model—the way the neural network reads, systematizes, absorbs and uses information to accomplish the goal set for it. And the numbers mentioned are the parameters the neural network tries to optimize while it trains.
Talking Guitar
The generator is news in itself, because it works really well. But from here on, the main advantage of this type of technology is that it allows various specialists and teams to get involved and show its full potential. Because it works successfully based on the assignments it is given, but what those are depends on the projects that will build upon it.
In fact, it can perform only one single task. How boring compared to the multifunctional devices we surround ourselves with, right? GPT-3 creates something that can be expressed as a linguistic structure based on the goal set for it.
It is enough to give it basic directions, a base text for the algorithm to follow, and from there it gets to work and returns its result.
This has already led to some rather curious experiments.
For example, the programmer and artist Mario Klingemann, who loves to challenge the creativity of algorithms, gave GPT-3 the task of writing a text on “The Importance of Being on Twitter” in the tradition of Jerome K. Jerome. And the result is quite decent.
After that, the imagination of the people who decided to challenge the new platform kept running wild at full force. They made it generate guitar chords from a given base and… it pulled it off.
Programmer Program
Then they even challenged it to program. Yes, it makes sense that artificial intelligence could, in the not-too-distant future, take away some of humanity's advantage in this very modern profession. It makes sense because, in the end, programming means creating computer code that instructs a machine how to perform tasks. In other words, once again a machine, an algorithm, could become a working intermediate link that raises the level of abstraction—so that it would be enough to feed it more basic parameters and get a result back…. But this topic is long and interesting; expect more soon.
What matters is that as a programmer, GPT-3 also turns out to be quite talented. First, at generating web pages. The programmer Sharif Shameem told it to create “a button on a web page that looks like a watermelon”… and the result came. Then followed experiments with actual programming, which did not produce wonderful results, but showed that GPT-3 can successfully create a foundation that, with a few human corrections, can then accomplish the given task.
On the Edge, Robot
In a way, GPT-3 is a relative of the algorithms we interact with every day. For example, it is very similar to the search engines Google uses to give us precise results—according to our personal filters. The difference is that the result there is much easier to present: just a list of suggestions rather than a complete text, for instance. So, in essence, the skills of neural networks like our hero are nothing new; the news is that it actually turns out to be really good and convincing at its job.
Yes, we are by no means talking about human-type intelligence; for instance, we could never suspect it of anything like human common sense. But the results it returns are quite convincingly similar to those generated by a human. Yes, perhaps the Turing test immediately comes to mind. It is entirely possible that GPT-3 would pass it, and yet it is still quite far from the level of an android you could have a beer with in the evening over a cozy chat…
On the other hand, it is also the most capable neural network humans have created so far. But as OpenAI's CEO Sam Altman puts it: “The GPT-3 hype is way too much. It's impressive (thanks for the nice compliments!) but it still has serious weaknesses and sometimes makes very silly mistakes. AI is going to change the world, but GPT-3 is just a very early glimpse.”
Chatterbox Robot
Among its many applications are chatbots that imitate communication with a real person far more convincingly (as you saw in the video above). And also programs for better machine translation.
All this offers countless new hopes for the future. For the long-awaited breakthroughs in all the fields on which we have pinned such serious expectations in recent years. Fields which, however, have also given so many reasons for concern.
Good manners, for example. As part of the fashion of recent years in the West, GPT-3 has been accused of racism and intolerance. At this stage there is no way to embed moral categories in a neural network—it processes what it is fed, learning from the input data. And when that data is, say, internet forums… there is no need to go into detail about how toxic, boorish and intolerant communication on the internet sometimes is.
And one more thing: because of everything said so far, the neural network is also a wonderful author of fake news and disinformation, especially if it is given precisely that goal.
GPT-3 Loves to Read
It is not entirely clear what exactly happens “under the hood” of GPT-3, even though this is supposedly “open AI.” Either way, even its creators say they are surprised by many of its successful results. Because it was not built with the idea of writing chords or code, or creating texts that resemble something, but to do just one thing: based on a sequence of words, choose the next one as accurately as possible.
The neural network looks for patterns, repetitions and regularities, and based on them it predicts the result. In this case, quite successfully.
The competition in this direction is as fierce as it is interesting.
Still, OpenAI's algorithm really is an important step, and it is no wonder that over the past year it has increasingly caught the eye of technical circles.
The Tin Man
The name GPT-3 itself doesn't sound very marketable at first glance, wouldn't you say? It comes from Generative Pre-trained Transformer.
Its “pre-training” means that it has already been “fed” 570 gigabytes of almost random text from the internet, supplemented with texts specially selected by its creators, including from Wikipedia. The computing time it took our friend to read and process this information cost $4.6 million, according to OpenAI.
Still, it seems to me that the algorithm's creators know very well why they decided to keep this name. A pseudo-human name in the spirit of Siri and Cortana would certainly have provoked far more extreme and unwanted associations. And one should not underestimate the link to good old C-3PO, the “tin man” from Star Wars. Or the nerds who are expected to swoon over this kind of technology.
Well, we do swoon sometimes. And the best part is that, with a little luck and persuasiveness, any team of programmers can propose, firsthand, its own way of using the platform. Or at least soon will be able to; for now, projects are piling up on the waiting list, because resources are limited and the project is still officially in beta. Evidently its creators believe there is still some fine-tuning to be done.
But you can try for yourself how it performs. More than 300 projects use it for various purposes. There are already quite a few platforms online that invite you to “talk” to a chatbot behind which our hero of the day is hiding. Maybe it's not a bad idea to get used, little by little, to the time when we will truly be able to talk to machines as if they were friends. It's only a matter of time, perhaps, before we teach them to drink beer too…
