Yahoo Labs Launches “Learning to Rank Challenge”

Dec 31, 1969

Yahoo Labs announces its first-ever online Learning to Rank (LTR) Challenge that will give academia and industry the unique opportunity to benchmark their algorithms against two datasets used by Yahoo for their learning to rank system. “Learning to rank has become a very exciting field over the last few years, and it has tremendous potential to improve user experience and result relevance, which are very important to Yahoo and the entire Internet industry,” said Olivier Chapelle, Yahoo senior scientist and technical lead for the competition. Unfortunately, most large-scale real-world datasets are not publicly available, which makes it difficult for academia and smaller companies to benchmark and improve their ranking algorithms. Yahoo is hoping to change that with the LTR Challenge. Participants will gain access to two datasets that Yahoo uses to train its search ranking function. All datasets have been reviewed to conform to Yahoo's data protection standards, including strict controls on privacy. “We are among the few owners of massive ranking datasets. We are willing to share this wealth of data in order to drive the sciences of the internet forward to ultimately build a better experience for users everywhere,” said Ben Shahshahani, head of search sciences. “Hosting the LTR Challenge will certainly lead to exciting new insights for the machine learning community as a whole.” The challenge will consist of two tracks: a standard learning to rank track and a transfer learning track. Both will be open to all research groups in academia and industry. Participants can upload multiple submissions over the course of the challenge, but not more than one submission every 8 hours. The LTR Challenge leader board will include the top 50 contestants, updated in real time. The challenge will run from March 1st through May 31st. After the contest closes on May 31st, Yahoo will select the winners of the two tracks and announce them at a workshop at the ICML conference in Haifa, Israel on June 25th. To learn more about the challenge details, how to register, and how to download the datasets, visit