People will argue for a long time over whether algorithms can create art. And while we argue… here that art is.
The technology called deep dreaming reveals what artificial intelligence algorithms see before, objectively speaking, they “get the job done.” And the picture is magical, surreal, and could easily be attributed to a human artist. Because it peers beyond the visible and finds in photos and paintings its own patterns, accents and nuances, images that stir something in us.
And isn’t that exactly what art is?
The pioneering technology that resembles it… was born from a programmer’s nightmare.
What’s more, it gives us not only extraordinarily interesting images but also helps us better understand one of the most talked-about technologies of recent years. It might even… prove a useful tool for learning more about human creativity itself, about the way our brains work, about our relationship with the Universe.
I Am the Sandman
May 18, 2015. Alexander Mordvintsev is sleeping fitfully, tossing in bed, and suddenly… he senses that an uninvited guest has entered his home! He has only recently started working at Google and is living in a rented apartment in Zurich with his pregnant wife and their three-year-old son. Alexander shakes off sleep and…
Realizes the feeling was a nightmare; fortunately, there are no burglars. But now he is wide awake, he doesn’t want to wake his family, and he sits down at the computer to do some work. He starts to wonder: how do neural networks dream?
Mordvintsev puts into action the idea that came to him in his sleep a little earlier… and is greeted by the first striking image. He tweaks the parameters a little and is amazed again. He sees that the algorithm, too, is “dreaming up” extraordinarily interesting images.
A picture like something from an LSD “trip,” only produced not by hallucinogens. By algorithms.
Sometimes I’ll Come into Your Dream
At 2:32 a.m., Mordvintsev posts what he has seen on the internal network of what was then Google Plus, used by the company’s employees. “I’m not sure if it was a really good idea to try a deep neural network at 2 a.m.,” he writes. And adds: “How am I supposed to sleep now?…”
That image turns out to be the first in the whole scientific field of deep dreaming. It shows us a sweet kitten turning into a sinister creature, as if stepped out of a hallucinatory dream. The first “deep dream” goes down in history under the fitting name “the nightmare beast.”
The sun never sets on Google’s engineering centers, and the picture quickly catches the eye of impressed colleagues on the other side of the planet. A second later comes the first like, garnished with the comment “My eyes! My eyes!” In no time it gets more than 100 comments; people tell him it reminds them of Lovecraft’s books and that Mordvintsev has unleashed the forces of darkness.
The wheel starts turning, and just days later, on June 17, 2015, deep dreaming reaches the world.
You Dream a Dream, Oh Sorrow, Oh Barren Youth
What occurred to Mordvintsev is a very small but brilliant idea that makes possible the images you see.
Deep neural networks are the high art of artificial intelligence. At first we associated AI with machine learning, and then it turned out that this was only the first step. That its algorithms could be “harnessed” into far more powerful structures in which they work side by side in layers, complementing one another and sometimes “frying” graphics cards in the process. All of this made possible the progress we see today in so many fields.
And lo and behold, it also turned out that these very super-powerful combinations of algorithms… can dream. That is what Mordvintsev’s interrupted sleep showed us.
At the time he is a newly hired Google employee. Although he wants to work on deep neural networks, his employers decide that the teams doing that already have plenty of people and that he will be more useful on the SafeSearch project, whose goal is to filter pornographic content out of search results.
Muff, Half-Shoe and Mossy Beard
Google, however, has the unusual practice of letting its employees spend one day a week working on another company project, and Mordvintsev naturally turns to his favorite subject.
That is exactly what is on his mind on the night of the famous discovery. But he knows he lacks the experience to develop the idea, so two of his Google colleagues, impressed by his post, join the work. Chris Olah, then 22, is still an intern at the company, having arrived as a young talent. He has won the “20 Under 20” fellowship, which gives him money to drop out of university and focus on work.
The trio is completed by Mike Tyka from Google’s Seattle office. He spent many years as a biochemist before turning to computer science. He works on the kinds of artificial intelligence technologies that will help develop new drugs. But he is also a man of the arts—he has had solo exhibitions as a sculptor, with works inspired by his biochemical research.
The three never meet in person, yet in just a month they manage to put together the famous blog post that shows the world the algorithm’s art. At first they wonder whether to make it a scientific paper, but decide it is more important for the idea to reach more people quickly.
Troika at Dawn
In the photos the three process through deep dreaming, mountains turn into pagodas and leaves become birds. Exquisite angels and sinister demons appear against plain backgrounds. Chris Olah sees new animals too, and names them the pig-snail, the camel-fish, the dog-bird.
At first they call the field “inceptionism”—a play on the title of the film Inception and the word “interception.” At Google, however, it turns out there is already a project by that name, so the name is changed, and along comes “deep dreaming.”
The name may sound metaphorical, but it is quite close to reality. Deep neural networks are essentially a series of layers of algorithms that analyze the information fed into them. At the end, based on a precise analysis, they return answers. For example, they can determine who the person in a photo is.
It is no accident that they are called “neural networks”—the way they process information resembles what happens in our brains. Each layer receives information from the previous one and passes it on, assessing various parameters along the way. Finally comes the last layer, which simply has to output the verdict, making the classification based on everything so far.
The Endless Dream, When You Slip Away
The networks used here are convolutional neural networks, whose job is to process images. Pictures are turned into computer code, and the “little neurons” start processing them, looking for patterns. Often their task is classification—for example, judging what the image in a photo is, a person or a hamster.
For this to happen, they have to be trained on an enormous volume of data, on the basis of which they tune their fine parameters. After that they are ready for work—you can send them a photo and they will guess what is in it with astonishingly high accuracy.
Until recently, however, exactly how they work remained an enigma—what happens in the very communication between the layers. This is where the concept of deep dreaming came in. Mordvintsev’s idea is to amplify the parameters the algorithm has judged important and to visualize them before the classification itself takes place.
In recent years neural networks have proved an extraordinarily successful solution, for instance in facial recognition or natural language processing, but the truth is that so far we do not know exactly what goes on inside them. Mordvintsev’s goal is to “peek into the brain,” to see how the network works. And that is exactly what he manages to do. Though he does not suspect that it will take the form of the visions you see—dreams and nightmares.
No Memory Remains of the Dream
Mordvintsev interrupts the convolutional neural network before it has produced its result and shows us what it has seen in its processing so far. Then he amplifies some of the input parameters so that the things that caught its attention stand out.
In other words, Mordvintsev simply changes the network’s task—from telling us what is in a photo to generating something that may not exist in it at all. Quite a different paradigm…
If this happens in the first layers, only some general strokes are visible—shapes, colors. The deeper you go, the more complex the objects become. We see the network bringing out eyes, shapes. The first layers detect textures, then figures, and finally objects.
A Sleepy Wanderer in a Bottomless Forest
It is just before the network finishes its work that the most spectacular images appear, the ones that recall hallucinatory pictures.
The resemblance is no accident. Karl Friston, a professor of neuroscience in London, says these images “very accurately show something we imagine as the result of psychedelics or during a hallucination, and this is entirely tangible.”
The dream metaphor in the technology’s name is precisely aimed. The beautiful or frightening pictures we see while we rest and our brain processes the past day evidently resemble, even visually, what “passes” through the layers of a neural network.
Lucy in the Sky With Diamonds
Mordvintsev’s colleague Mike Tyka gives a rather good explanation. “There is a deep connection between the way visual neural networks and our brains work. That’s why I don’t find it surprising that similar results show up. If you use chemical stimulants, for example, that confuses the neurons and the transmission of synaptic signals, and you get strange over-interpretations and distortions. I think we have a complete analogy here. I believe that studying neural networks can really help us better understand biological ones too.”
Our brains “house” about 100 billion neurons, linked by 100 trillion connections that are still not fully understood. And whenever we see something, they work on a similar principle. There is initial processing of the information in the visual cortex, which identifies lines and angles. Then the data go to the parts of the brain that fill the meaning with images—faces, eyes, hands. Finally comes the assessment, which also draws on our memories and our knowledge of language, so that we can determine what we are seeing.
With neural networks the principles are similar. True, in the sheer volume of connections they are still far from our numbers, closer to much earlier forms of life in evolutionary terms. But they are striding forward fast. And besides, in some fields it turns out they already “work” better than the human brain.
The City Sleeps in Soundless Darkness
In the first days after its appearance, the subject became a hit. Anyone could play with their own images, have fun with the creepy little eyes popping up on a portrait of their mother-in-law… and then, as with every hit, interest faded somewhat. Objectively, though, deep dreaming has become a serious field within neural networks, and in the coming years we can certainly expect more interesting results.
It is also increasingly used when one needs to assess whether a neural network works well and whether its architecture is right.
At first the three at Google used a network already trained on ImageNet, a dataset widely used in the field. It contains 1.2 million images in 1,000 classes, no fewer than 118 of which are dog breeds. Naturally, that shows in the images themselves. Later the idea was applied to all sorts of datasets and neural networks. Even to ones that work with text.
For scientists in the field, what the processing looks like is also important—it gives them a lot of information about how and why the algorithm “sees.” How it differs from the way our brains work and how it resembles them.
For example, dumbbells often appear in deep dreaming images. One would assume that for the “computer brain” these would be metallic-looking, elongated objects, thinner in the middle with wide weights at the ends. It turns out, however, that what stands out instead are metal objects necessarily attached to a human hand.
Deep dreaming starts generating impressions and interesting images even when it is fed a single-color frame, random noise or a uniform texture. And might… this be how human imagination works too? The way all of us dream?
A Dream in C
The discovery opens many doors, technological but also philosophical. Perhaps generating images with neural networks can change our ideas about human perception and creativity.
Today all three heroes still work at Google. Shortly after his discovery, Mordvintsev was moved to the team of his dreams, where he worked on facial recognition. Later he specialized in searching for more and more algorithms connected to generating art, some of them also impressive. In 2021—six years after the “nightmare night”—he set out, in his own way, to recreate the algorithm.
The idea came after he was asked for an image to be included in the K21 collection, which gathers “the art of the next 100 years.” Mordvintsev wanted to run his original code; back then he had written a mere 30 or so lines that did all the work. But they “wouldn’t start,” since they relied on many external libraries that had changed in the meantime. So the programmer set about writing everything from scratch. He chose the classic language C. The new base algorithm now runs to 1,000 lines of code and, as its creator says, depends on no external factors. It will always be possible to run it as long as C works on at least one computer.
Fittingly, this program—called deepdream.c—was sold in the form of non-fungible tokens, NFTs.
Sweet Dreams
So, is this art? Something created from nothing? Something seen and interpreted?
As Mordvintsev’s colleague Mike Tyka puts it, “if we think about human creativity, one small component of it is the ability to gather impressions and recombine them in interesting, unexpected ways. It’s cool to see computers notice unexpected things that wouldn’t occur to people.”
Let us give the floor to Mordvintsev as well. “I think it was Picasso who said: ‘I have never made art, I only do research.’ Of course, that’s oversimplified. Scientists have to cite their sources and explain how they use them. To take the time to present what they do within the framework of what others did before them. In art it isn’t like that. Besides, for me the little piece of code I once uploaded to GitHub is a work of art too. Code has always been my art. Code is my paintbrush.”
Over Sleepy Lyulin, Veiled in Mist
Here you can see the GitHub repository where I, too, “played” with deep dreaming for some of the images you see. And if you’re into computer code, you can use it to have some fun yourself; you don’t need any particularly specialized software.
And even if you don’t program at all, you can still experiment here. I’d be happy if you shared the results afterward.
Artificial intelligence will go on being magic or simply a con in our daily lives. In it we have an ever more powerful weapon that we are only just learning to use. But, as deep dreaming shows, sometimes a single good idea is enough to show how, hand in hand, human and algorithm can create art. Can create a beauty of their own… and, as we remember from Mordvintsev’s good old compatriot, it is beauty, and nothing else, that will save the world.
