Tonight I want to watch a movie, just not one as dumb as the last. I'm looking for recommendations. Once upon a time I would have dug and dug around the internet; now there's no need, artificial intelligence takes care of me.
Content recommendation algorithms are among the older and most successful representatives of machine learning. These days they aren't quite so fashionable to talk about, for two main reasons. First, they already work very well, and second, they are a rather awkward subject for those who use them most actively.
Why is that? They are precisely what lies at the root of one of the big problems of our time: the filter bubble. We're in a hurry, we only have time to read recommended information, we surround ourselves with comfortable points of view, we have no energy left for nuance, and soon enough we split into vaxxers and anti-vaxxers…
…but hold on, weren't we talking about movies at the start? Isn't that something harmless? Exactly. The algorithms are not to blame for how we use them. And the story of those searching for ways to offer the best content is interesting for anyone who cares about technology and the future it is preparing for us.
AI, Recommend Me a Movie!
A decade ago, the Netflix Prize was all the rage. The giant had promised a million dollars to the scientists who came up with the best solution in the field of content recommendation algorithms.
The last round of the prize, in 2010, was canceled, and Netflix gave up on organizing the competition any further. Apparently, they had already become good enough in this area…
Whether recommendation systems are a good or a bad thing is not the right question at all. We have always asked our friends for recommendations on the next books, movies and records to enjoy. Consulting people whose taste is similar to ours is certainly a pretty good approach. Obviously, the fact that a movie has become very popular, for instance, is no guarantee we'll like it. A much better idea is to compare our tastes with friends who have already seen it.
Today, though, we don't even have time for that. And why should it happen that way anyway, when we have sophisticated machine learning algorithms on our side? They are very good at matching data and finding patterns and similarities. At picking out, from a vast database, the people whose tastes resemble ours, organizing the information and, on that basis, giving us a specific recommendation with a very serious chance of hitting the mark.
So how do the recommendation systems that have proved themselves the best today actually work?
Divide and Hit the Target
What is machine learning? A way of creating “artificial intelligence” that helps in countless areas (and perhaps one day will even govern us). Like every other scientific discipline, if we approach it without prejudice, we can picture it as something simpler. The computer receives a huge body of information. Then the machine learning specialists take care to offer it the most accurate algorithms so it can try to find the patterns within. The possible solutions are endless: some are simpler and can be explained in a few lines, others are almost impossible to present in popular terms.
The engineers split the data they feed in into a set for the actual training and a set for testing how well it performs. Then they let the program try on its own to guess the variable parameters of the given algorithm using the main data, and compare the results against the test data set aside for that purpose.
At this point they apply various new methods to show how well the main algorithm has handled its task. If its score is high enough, they're sure they've done their job. And they can set it free…
Reading Minds
As I mentioned, recommendation algorithms are among the oldest pioneers of machine learning, and today we know them quite well.
There are two main approaches in this field, called “collaborative filtering” and “content-based filtering.” More modern solutions use various combinations of the two, so as to draw on the strengths of both.
Collaboration, or cooperation, means that people work together to achieve a given result. That is also the foundation of collaborative filtering. The same principle I mentioned earlier, the one we still use today. Or, in the language of sales: “People who bought… also bought….” The algorithm's goal in this case is to process the data so as to seek out, as far as possible, what the choices of different users have in common, and thereby draw out the trends.
Well, back in the day everyone liked Titanic, so how do we stay afloat in this ocean of choices?
Without Bias
Broadly speaking, the approach goes like this. A collaborative filtering algorithm groups together the people who have given ratings as close as possible to those of the user we're about to recommend something to. The data is then used to calculate how likely it is that people with similar taste will also like another product from the same group. Among the users who liked your favorite movies, we find the average scores for each movie and see which ones come out highest. Those are the ones we'll recommend. While, of course, trying to ignore things like movies with too few ratings, or users whose numbers are all over the place. Or the so-called “partiality,” which the tech world calls by the English term “bias.”
On the other side comes content-based filtering. The idea isn't very different; the perspective is simply flipped. The algorithms are based on the other units, which in this case are the movies rather than their viewers. Users' ratings are compared for every pair of movies, and on that basis a guess is made about what rating a user who hasn't yet seen the second movie would give it.
As I said, Netflix is definitely a pioneer and a proven force in this field, which is why we'll look at its approach in more detail. According to the company, 80 percent of views of its movies are the result of personalized suggestions. Just imagine! Only 1 in 5 movies is watched because the viewer opened it on their own; all the rest were suggested as the next best choice.
Collaborators
Netflix bases its system more on the collaborative filtering approach. Evidently, that proved the winning solution after the competition I told you about. And the story of the prize is more than interesting. Few people know it, but Netflix started out in the late '90s distributing movies by subscription, delivering the chosen titles on DVDs to viewers' homes.
In 2000 it unveiled its advanced recommendation system for the first time, and six years later it created the prize in question. It promised 1 million dollars to anyone who managed to build a system with better performance. There was, of course, a specific yardstick, too; 1 million isn't handed out for nothing. One of the most commonly used criteria was chosen: the root mean square error (RMSE). Netflix's algorithm achieved the exceptionally good value of 0.9525, and the prize was promised to anyone who beat it by at least 10 percent.
Fortunately, the prize didn't stay with those who announced it (“fortunately,” because they had plenty of money anyway). In 2007 it was won by an entry that used a combination of two more complex algorithms: matrix factorization (SVD) and restricted Boltzmann machines (RBM). The score in this case was 0.88, which met the requirements!
Prime Time
In 2009 the prize was won again, with a new combination of algorithms. Netflix dutifully paid out the promised sum once more, but this time decided not to use the proposed solutions in practice. It judged that very slightly better results would require far too heavy an investment. Its system had already proved to work extremely well, reaching the aforementioned 80 percent of recommended content.
One of the reasons the prizes were dropped afterward was the arrival of the streaming era. The company stopped simply selling movies and turned into a subscription platform, reaching more than 190 million subscribers by 2021!
At that point, recommending well was simply no longer such a priority, because its next sale was not a new movie but a subscription.
Fit to Read
And so recommendation systems kept moving forward, bringing us to today's era of the notorious filter bubble. Without a doubt, these very algorithms still underpin the personalized content and recommendations we get from social networks and search engines. It's just that these solutions aren't talked about as much anymore, because they are no longer only a source of pride.
Of course, everything described so far covers only the basic ideas. The recommendation algorithms that Facebook and Google work with are extremely complex and multilayered, and, naturally, little is known about them. Besides, the recent revelations made by a former senior employee of the former social network show clearly why it is not at all in the company's interest to reveal its systems.
As for Google and the ads it shows you, there's one small detail. Based on your everyday interactions online, the search engine determines key characteristics about you. You'll be surprised (or maybe not) by how accurate they are if you take a peek here. You can even limit them, or so they claim.
In any case, for more complex tasks such as facial recognition, machine translation and self-driving cars, the next generation of technology is used today. It is the high art of this field and the universal solution on which the greatest hopes for our technological future rest: neural networks.
It is deep neural networks that today lie at the heart of the big players' recommendation systems. They don't just use one or two specific algorithms to process the data, but many layers capable of searching for ever finer trends in it. Thanks to neural networks, computers can even… dream! But that is a long and fascinating topic we'll discuss in more detail soon.
Hands-On Algorithms
Here you'll find a file repository (oddly enough, in Bulgarian we simply borrow the English word) with a basic model for building a recommendation algorithm on the official dataset used for the Netflix Prize. It applies the principles of collaborative filtering.
So now we know a little more about the intelligent systems that pick our preferences for us, whether news, movies or books. And knowing how they work is certainly useful for keeping their downsides in mind, too. Algorithms are neither good nor bad. It's up to us how far we let them, through their recommendations, send us into the “bubble.” They already know us quite well. So it's a good idea for us to get to know them, too…
