In the much-hyped world of artificial intelligence, there is hardly an algorithm more famous than GPT-3. Here you can read in detail how and why OpenAI's creation turned out to be a serious breakthrough in a whole range of fields. It appeared at the end of 2020 and, little by little, gave researchers from various directions the chance to use its abilities to demonstrate breakthroughs never seen before. In writing and editing texts, in translation, in generating computer code, in super-advanced chatbots, and who knows what else.
And now the next contender is on its way. Although it is running late compared to the original expectations, GPT-4 is expected to be unveiled very soon. Can we get even more out of it? How does it differ from its predecessor? In which fields do we expect it to bring something truly different?
G P Three
So we are talking about a model from the top flight of artificial intelligence—deep neural networks. This type of algorithm becomes exceptionally good when it has access to massive volumes of information, on the basis of which it learns to look for and find patterns. Then it carries out the task it has been given—for example, generating its own text according to specified parameters.
The abbreviation stands for Generative Pretrained Transformer. Each of the versions is actually not a single model but a family of them, each with a different purpose.
When it comes to some of the big companies, they rely on their own algorithms for this kind of work. The great advantage of OpenAI, as its name suggests—the venture was launched by Elon Musk and like-minded partners—is that it gives a wide variety of projects access to a pretrained model that copes well with a broad range of tasks. From there on, it is up to the individual teams to formulate their brief and prepare it in a way that takes advantage of the algorithm's skills.
The most impressive successes came in every field related to text. GPT-3 was an undisputed breakthrough when it came to any kind of manipulation involving words. Of course, these abilities were above all tied to the English language. Applications appeared that processed text with incredible success—editing it, rewriting it in different styles, and who knows what else. It turned out that the platform worked incredibly well in other directions too—for example, generating computer code or sheet music. Anything that can somehow be reduced to written symbols.
G(PT)enius
And also in imitation—for example, in holding a conversation.
Will the next member of the GPT family bring a similar breakthrough, only in the next set of fields? That is one of the questions specialists in artificial intelligence are asking themselves. As futurologist Assoc. Prof. Mariana Todorova said recently, great things are expected of GPT-4. Everyone in the field has their ears pricked, while, unfortunately, concrete news about the future algorithm's skills remains vague.
Take, for example, the number of parameters it will be trained with. The data is terribly contradictory. At first there was talk of tens of trillions, then of a trillion… until in the end it turns out that it will more likely involve even less input than its predecessor was trained on.
GPT-1 arrived in 2018, and a year later came its successor, trained on 1.5 billion parameters. The famous version 3 was unveiled in 2020 with its 175 billion units of information. So the fourth version is lagging behind the release rhythm of the others, and now it is surprising us on the other key indicator as well…
Gee, It's You
At first, amid the euphoria that came with the rise of the third version, there was talk that the next one would be trained with 100 trillion parameters, thereby matching for the first time the number of synapses—the connections between neurons—that each of us houses in our brain. And yet the step will rather be backward—to even fewer than those 175 billion.
The bet will be more on more advanced algorithms. And on more careful selection of exactly what the data for self-learning will contain.
The fact is that, until now, the laws in this direction seemed unavoidable. For a long time, the development of neural networks was held back by computers' inability to process data in the volumes they needed. Then, somewhat naturally, the hardware advanced, and it was the software's turn to catch up. Because it turned out that volume only matters for improving results when it is fed in meaningfully. When neural networks learn not from the phone book, but from more and more texts that are, however, meaningfully structured.
The news that GPT-4 will be trained on fewer parameters came from the words of OpenAI's CEO Sam Altman, who, unfortunately, rarely gives interviews on the subject. He also said that although the base the future algorithm will be “fed” with is more modest, it will be considerably more complex than its predecessor. Training it will require significantly more computing power, but it will be able to handle many more tasks.
Size…
Either way, GPT-3, although undoubtedly the best known so far, is far from the artificial intelligence holding the record for the number of parameters it was trained on. Gopher, for example, has “read” 280 billion, PaLM 540 billion.
Training is the most crucial part of how deep neural networks work. At the start they are simply sequences of algorithms incapable of solving problems. They have to be “fed” the largest possible volume of information, on the basis of which they get used to finding patterns, searching for the important factors, giving more weight to those parts of the network that turn out to be more telling.
For a long time it seemed the relationship was direct—the more data, the more precise the results. According to many scientists, however, we have already reached the limit beyond which it makes sense for data to keep growing in volume, while what matters far more is exactly how the algorithms work and what the architecture of the neural network is.
The G(PT) Spot
An important clarification we must make is that, when it comes to deep neural networks, alas, we know far too little about the specific way they function. They are a closed black box, a series of layers of algorithms for which we know only the input and output data. We can influence them with various additional parameters, but on the whole, exactly how they were trained by taking in their data remains an enigma. What happens “inside,” which parameters “excite” them—these are still important and insufficiently answered questions, for which unconventional solutions are being sought (such as “deep dreaming”).
Somewhere here comes the next big question: if GPT-4 eventually starts giving accurate answers in more and more fields, won't it also be yet another step toward a possible future general artificial intelligence? Or at least the thing that increasingly resembles it.
Another curious thing already known about GPT-4 is that it will once again be trained on text alone. In recent years, ventures pursuing multimodality have become popular—the data fed in is not only text but also images, sounds, video. A popular and impressive example in this direction is DALL·E—the algorithm that creates stunning images based on a description.
And yet… DALL·E also stands on the shoulders of GPT-3; this is the algorithm that “painted” the pictures you see. Because it turns out that a neural network is capable of miracles when it is given additional training, too. Yes, the AI artist was trained on the connection between text and image, but only as a form of building on top of the purely text-based algorithm. It is a kind of specialized variant of GPT-3, as is the platform for generating computer code, Codex.
Many specialists expected the GPT team to head in this direction as well. And yet, for the moment, it turns out that it is more effective to arrive at successful solutions to a wide range of problems using text alone as the foundation. Which is strange, insofar as neural networks are supposed to resemble our brains, which, of course, are “fed” far more varied information. Evidently, OpenAI's idea is to develop the potential of text models to the fullest before turning to the even more complex multimodal future.
In the Beginning Was the Text
In this regard, the opinion of Ilya Sutskever, the researcher who leads OpenAI's scientists, is intriguing. Back in 2020 he predicted that “by the end of 2021, language models will start to understand the visual world. Text on its own can express a tremendous amount of information about the world around us, but it is incomplete, because we also live in a visual world.”
And since we have mentioned several times how important the right information is, it is interesting to look at what it actually is. For GPT-4 we can still only guess, but this much is known about its famous predecessor.
Over 90 percent of the data it was fed comes from English-language sources, which also explains why the algorithm is so good specifically in its “native” language. The entire volume of data comes to 45 terabytes—you can imagine how much text fits inside. The largest share—410 billion units—comes from the Common Crawl collection, a freely accessible database that scans and stores information from the entire internet. In other words, that is precisely where the most varied texts end up. This base also accounts for a full 60 percent of the weight—how much importance the algorithm assigns to them. Another 22 percent comes from a base collecting Reddit posts that have received at least 3 upvotes.
GPT-3 has “read” a total of 67 billion words gathered in various books from two databases, with a combined “weight” of 16 percent. The smallest share—3 billion units—comes from the English-language Wikipedia, which nonetheless carries a weight of 3 percent, considerably higher in proportion.
An interesting reason to stop and think… what if its creators had given it more books to “read” and fewer random texts?
You Are GPT
All this gives us still more reasons to ponder what awaits us with the arrival of version 4. If the fourth GPT really is a considerably more perfect algorithm, it could, for example, be capable of generating, extremely quickly, fake videos indistinguishable from the truth, better known as deepfakes. Or texts, masses of information that cannot be told apart from those created by humans. If chatbots can still easily be caught red-handed, won't they now become even more convincing?
And so come the next important questions, which are constant guests at Digital Stories. About our future as human beings, about technologies that can and do change everything, however skeptical we may sometimes be. The interesting future has long been here, and it keeps getting more interesting. And without a doubt, part of the big picture very soon will be GPT-4—loaded with enormous expectations and hopes… and, of course, the pinch of worry that is obligatory in this subject.
So which should there be more of? Watch for the next Digital Story! Together with Prof. Preslav Nakov, one of the world's leading specialists in AI and natural language processing, we will fix our gaze on the future!
