March 23, 2016, could have been an apocalyptic day in the history of the digital world, one unlikely ever to be repeated. Netflix, Facebook and Spotify were just a few of the countless sites that suddenly turned out to be… impossible to maintain and update.
Why? Twenty-eight-year-old programmer Azer Koçulu had simply deleted 11 fateful lines of computer code.
But how can an unknown programmer block the sites of global giants? Why, in today's strange online world, do enormous multibillion-dollar projects unexpectedly turn out to rest on the shoulders of volunteer enthusiasts? And what made Azer so angry that he unleashed the butterfly effect in its most terrifying form?
Sometimes the smallest, seemingly harmless lines of code change the whole program…
A Backup Never Hurt Anyone
The digital story of JavaScript—the programming language is a fascinating one: it was created in just 11 days and… sometimes it shows, even though it has come a long way since. It became the life force that paved the paths of today's internet as we know it. Where we once downloaded programs for just about everything and installed them through the operating system, today almost everything happens in the browser.
The change came more quickly and imperceptibly than we could ever have guessed… And everything can come crashing down just as quickly, as the story of today's hero shows.
The rise of web applications coincided with another important phenomenon in the software world—the mass arrival of open-source software. The pioneer was Linus Torvalds, the creator of Linux and Git, and over the past decades a huge part of today's software industry has come to rest on programs that are open and freely shared.
This is especially true of the JavaScript world. Its most popular platform is called NPM, and it holds literally millions of projects that a programmer in need can use. Everyone is welcome to leave their own bits of code, which, one way or another, can then be used by everyone else.
Haste Makes Bugs
This system works in a more than impressive way. New libraries keep appearing all the time, and the most successful ones are downloaded and used by developers millions of times a day.
What's more, projects of this kind gradually grow into enormous towers of Babel, because software libraries rest on one another, each new one using a little piece of code from the ones before…
Which, at some point, for all its convenience, can turn into a problem. Exactly as in the story of Azer Koçulu.
Many people joke that to create a new project with all the modules you have to use, you pile up gigabytes in the node_modules folder, which at some point weighs more than the Universe… but that, in the end, doesn't affect end users and is definitely the smaller problem.
A Word Spoken, a Bug Thrown
The bigger risk is surely security, which becomes ever harder to track when you rely on a sea of code. And that is exactly what the episode with the migrant from Bulgaria's southern neighbor, Turkey, reminds us of.
From an early age, Azer Koçulu was passionate about the ideas behind open-source software. Alongside his main job, he had a hobby of sorts—creating libraries and uploading them to NPM, giving them to other programmers.
Some of his projects enjoyed serious success, and he found it satisfying. By the infamous March 23, 2016, he had published no fewer than 273, some of which were used hundreds of thousands of times a week. And that's exactly when he received an email that made him furious…
It came from Bob Stratton, a patent specialist at a company called kik. Azer had created a small software library with the same name, and to avoid any confusion, the lawyers were asking him to rename his project.
Whispers Wreck Websites
You know how things in online communication sometimes escalate fast, right? Well, a few emails later, Azer was breathing fire and brimstone, with a vocabulary that a character in a Serbian film would envy. He flatly refused to rename the library, even when offered $30,000. He said his idea was different and that commercial interests should not lead the way in the field of open source.
Having read some interesting things about his close relatives, the patent lawyer realized this approach wasn't going to work. So he decided to go straight to the organization that could help him.
Although NPM is an entirely noncommercial platform, it has an owner company that looks after the rules and the infrastructure. That is who kik's patent lawyer wrote to, and he evidently managed to impress the recipients, because this time, instead of a colorful reply, he met with agreement.
Azer received a new email, this time from NPM, informing him that his library had been removed over copyright issues. Is that so? Outraged that the platform had given in to blackmail and trampled on its own rules, he deleted all 273 of his libraries in a single move.
Drop by Drop, a Bug
Which brings us to the afternoon when thousands of programmers around the world got the same error—some library almost none of them had ever heard of couldn't be loaded. That meant they were blocked in their work, and some who had the bad luck to be pushing their new code to users at that very moment found themselves with completely broken sites. And all of it because of a single one of Azer's projects.
But how could no one have heard of the missing little library when, in the month before, it had been downloaded by no fewer than 2,550,569 programmers?
How could everyone have used his code without ever hearing his name? How had companies like Netflix, Facebook and Spotify ended up trusting outside software that could even do unsuspected harm?
The answer, once again, lies in the pyramid of today's web software. Everyone relies on external libraries, which in turn rely on earlier ones… in a huge cavalcade of mutual dependencies.
Azer had simply pushed the first domino… “This just broke the internet,” reads the comment that collected the most cheerful reactions on the subject on GitHub, the platform for programmers.
Measure Thrice, Code Once
Well, the same thing happened with Azer's work. It turned out that a huge number of the most widely used applications relied on precisely the smallest of all his 273 projects—called Left-pad and consisting of… 11 lines of code:
If you've even dabbled in programming, you'll quickly see that this is an elementary function that literally anyone with a basic knowledge of JavaScript could write in minutes. And yet, instead of writing it themselves, some of the most popular libraries holding up the web had decided to use it ready-made.
Most of the trouble came from the fact that Left-pad was at the time part of Babel—a very important and widespread library used to “translate” code so that older browsers can read it too. Babel is used practically everywhere. The list included the sites of Facebook, Netflix, PayPal, Yahoo, Spotify, Reddit, Slack and LinkedIn.
And so… the “online apocalypse” was quickly averted. The first working fix appeared 42 minutes after the problem was first mentioned. Less than two hours later, a new package with the same name had been uploaded to NPM, which solved the problem at once.
The Fool Isn't the One Who Eats the Bugs…
Then came the post-match analysis. Azer, kik and NPM all went into explanation mode to tell the community what had happened, and the picture quickly became clear.
Azer said he had gotten angry because “this situation made me realize that NPM is someone's private land where corporate is more powerful than the people. And I do open source because ‘power to the people.’” He asked for forgiveness from everyone who had been affected, but said this was exactly the opportunity needed to talk more about how decisions in the community should be made by the community itself, not by centralized institutions.
The more important questions, however, remain. Had it been a far more complex piece of code, it could literally have blocked the work of millions of programmers around the world and threatened almost all of the biggest sites.
Does the idea of open-source software still sound so tempting?
Strike the Code While It's Hot
Fortunately, it turns out to be more magnetic and more effective than ever. And cases like this one are exactly what prove it.
Because the community, which includes an enormous number of specialists, can react quickly and solve a problem like this. Today Azer is a decade more experienced, he has racked up serious successes as a programmer, he has a simpler haircut, and maybe he has even forgotten that he once very nearly “broke the internet.” But the lessons of this case led to stricter rules about who can delete a code library and how. Today that would take much longer, and everyone using it would be warned in time to take action.
So when the commercial element is eliminated, miracles can sometimes happen.
Open-source software reigns as never before. Because it is one of those not-so-numerous examples that bring us back to the original idea of the internet. The chance to be united, equal, to help one another… If only we could learn to do the same on social media.
