Close competition, innovative ideas, and a lot of determination were some of the highlights of the first ever Yahoo Labs Learning to Rank Challenge. The challenge, which ran from March 1 to May 31, drew a huge number of participants from the machine learning community. There were a whopping 4,736 submissions coming from 1,055 teams. Winners were awarded cash prizes and the opportunity to present their algorithms at a workshop at the 27th International Conference on Machine Learning (ICML 2010) in Haifa, Israel. “The number of entries for the challenge was greater than expected – by some of the most reputable researchers in the industry,” said Yahoo Labs Researcher Olivier Chapelle. “The intense competition is an example of the huge interest in the field of learning to rank, and its potential to improve user experience and search result relevance.” The goal was for participants to learn a ranking function based on the relevance judgments that were available for a training set. They had to predict the ranking of several URLs for each query in a validation set and in a test set. There were two tracks in the challenge: a standard learning to rank track and a transfer learning track where the goal was to learn a ranking function for a small market by leveraging the larger training set of another market. “We received a lot of positive feedback from the academic community for releasing very large datasets for the challenge,” said Chapelle. “Researchers now have the opportunity to benchmark their algorithms using our datasets.” Participants presented their solutions at a workshop at ICML 2010. Chapelle noticed that all top competitors used similar techniques in their approach: an ensemble of trees, bagging, boosting, and random forests, and often a combination of all of these techniques – a clear indication of the effectiveness of these techniques in learning to rank. Winners were scored according to the relevance of their predicted rankings. To celebrate their victories, winners got together for a dinner on June 25th at Rennstaurant, a restaurant near the center of Haifa that offered majestic views in a stone building. And the winners are Track 1:

• 1st place: Chris Burges, Krysta Svore, Ofer Dekel, Qiang Wu, Paul Bennett, Andrzej Pastusiak, John Platt [Microsoft Research]
• 2nd place: Eric Gottschalk [Activision Blizzard], David Vogel [Data Mining Solutions]
• 3rd place: Dmitry Pavlov, Cliff Brunk [Yandex Labs]
• 4th place: Daria Sorokina [Yandex Labs]

Track 2:

• 1st place: Igor Kuralenok [Yandex]
• 2nd place: Ping Li [Cornell University]
• 3rd place: Dmitry Pavlov, Cliff Brunk [Yandex Labs]
• 4th place: Pierre Geurts [Uuniversity of Liege]

Congratulations to all of the winners! Full datasets (including validation and test labels) are available under the Webscope program. (http://webscope.sandbox.yahoo.com).