I'm a week late to this story that hits right in my field of expertise! Apparently the Netflix prediction contest has been won!
The company’s challenge, begun in October 2006, was both geeky and formidable: come up with a recommendation software that could do a better job accurately predicting the movies customers would like than Netflix’s in-house software, Cinematch. To qualify for the prize, entries had to be at least 10 percent better than Cinematch.The winner, formally announced Monday morning, is a seven-person team of statisticians, machine-learning experts and computer engineers from the United States, Austria, Canada and Israel. The multinational team calls itself BellKor’s Pragmatic Chaos. The group — a merger of teams — was the longtime frontrunner in the contest, and in late June it finally surpassed the 10 percent barrier. Under the rules of the contest, that set off a 30-day period in which other teams could try to beat them.
That, in turn, prompted a wave of mergers among competing teams, who joined forces at the last minute to try to top the leader. In late July, Netflix declared the contest over and said two teams had passed the 10-percent threshold, BellKor and the Ensemble, a global alliance with some 30 members. Netflix publicly said the finish was too close to call. But Netflix officials at the time privately informed BellKor it had won. Though further review of the algorithms by expert judges was needed, it certainly seemed BellKor was the winner, as it turned out to be.
When the contest was announced I wasn't sure that a 10% improvement was possible given the dataset. Apparently I was wrong! Still, it was a close thing... if they had made the requirement 11% would they have still found a winner?
I love these sorts of contests. They're great for the issuer and for the winner because the cost is almost entirely borne by the many losers.
Yet the sort of sophisticated teamwork deployed in the Netflix contest, it seems, is a tricky business. Over three years, thousands of teams from 186 countries made submissions. Yet only two could breach the 10-percent hurdle. “Having these big collaborations may be great for innovation, but it’s very, very difficult,” said Greg McAlpin, a software consultant and a leader of the Ensemble. “Out of thousands, you have only two that succeeded. The big lesson for me was that most of those collaborations don’t work.”
Tens of thousands of man-years spent for Netflix's benefit, for a total cost to the company of under $2 million (including administrative costs). Slate runs some numbers.
Imagine if Netflix had paid all these math whizzes the prevailing wage—say, $100,000 a year. The company would have had to shell out more than $3 million for just one year of the top performers' time, and that's assuming it could've sussed out who the top performers were going to be. Of course, many of the programmers worked far longer than a year, some of them setting aside their primary occupations in order to work on the Netflix problem full-time. As Netflix CEO Reed Hastings admitted to the New York Times, "You look at the cumulative hours and you're getting Ph.D.s for a dollar an hour."But even that number discounts the contest's true benefits to Netflix. Had the company simply put out a help-wanted ad for software engineers, it probably wouldn't have been able to recruit many of the geniuses it found through the competition. That's because most of them already have other jobs. BellKor's members work for, among others, AT&T and Yahoo, and many members of the Ensemble are employed by the data-consulting firm Opera Solutions. The participants also spanned the globe. Netflix got submissions from people in more than 100 countries, and the winning team's members worked in New Jersey, Montreal, Israel, and Austria.
That's the main reason why I didn't enter the competition. The only way to really win is to be the one issuing the challenge.
