HBase Goes Fast and Lean with the Accordion Algorithm

The Scalable Systems research team at Yahoo is contributing a significant enhancement to the forthcoming release of Apache HBase 2.0 called Accordion, an algorithm that simultaneously improves performance speed and disk space, two metrics typically considered at odds. Accordion optimizes RAM by re-organizing in-memory data in efficient data structures and reducing redundancies. The HBase server’s memory footprint then periodically expands and contracts (like an Accordion) to produce a longer lifetime of data in memory, less I/O, and overall faster performance.    

Researching the Definition of Good Online Conversations and How They Should Rank with the Yahoo News Annotated Comments Corpus

In recent statistical experiments looking at online comment threads, Yahoo Research shows that automatically identifying and ranking good conversations on top will cultivate a more civil and constructive atmosphere in online communities and potentially encourage participation from more users. In an effort to foster more respectful online discussions and encourage more research among academics surrounding comments, we present the Yahoo News Annotated Comments Corpus (YNACC) via our data sharing program, Webscope.

Adapting to the Evolution of Email with Machine-Generated Mail Mining

As a result of recent research, we know that machines account for 90% of all mail traffic. These machine-generated messages, whether they are purchase receipts, flight reservations, or something else, contain loads of personal information. At Yahoo Research, we have developed a new classifying technology that distinguishes between human- and machine-generated mail that has allowed us to unpack that data in a meaningful ways to advance the mail experience for our users.

Reinventing Mail Search and Solving the Catch 22 Dilemma Between Precision and Recall

How many times have you tried to search for an email and not been able to find it? You might think that getting the right email search result is as easy as getting a good search result on the Web, where you often find what you’re looking for on the first try. Unfortunately, experience tells us otherwise. The reason? Mail search is, perhaps surprisingly, an entirely different animal. At Yahoo Research, we’ve been working hard at changing this situation.

Introducing Similarity Search at Flickr

We're introducing large-scale visual similarity search on our photosharing community Flickr, a feature powered by deep neural networks. Our Computer Vision team uses a state-of-the-art approximate nearest neighbor algorithm, Locally Optimized Product Quantization (LOPQ), to compute distances on and store high-dimensional floating point feature vectors for an index of billions of photos.

Yoelle Maarek to Deliver Keynote at WWW 2017

Yoelle Maarek, Vice President of Research at Yahoo, will deliver a keynote address at the 16th International World Wide Web Conference taking place in Perth, Australia from 3-7 April 2017.

Researching the Future of Automated Question-Answering

Our Yahoo Research Text Mining team in Haifa is working to advance the state-of-the-art in question-answering, which, among other things, will lead to conversational bots appearing more human. Guess who came out on top in their most recent experiment: humans or algorithms?

Presenting an Open Source Toolkit for Lightweight Multilingual Entity Linking

We just open sourced Fast Entity Linker, our unsupervised, accurate, and extensible multilingual named entity recognition and linking system, along with datapacks for English, Spanish, and Chinese. Fast Entity Linker is one of only three freely-available multilingual named entity recognition and linking systems. The system achieves a low memory footprint and fast execution times by using compressed data structures and aggressive hashing functions.

10 Years of Hadoop and its Israeli Pioneering Researchers

The Yahoo Research Scalable Platforms team in Haifa collectively adds many years of experience to Yahoo and the Hadoop community in distributed computing research and development. The team specializes in scalability and high availability, arguably the biggest challenges in big data platforms. We profile the team and their impact in the Israeli research and engineering community.

Open Sourcing a Deep Learning Solution for Detecting NSFW Images

To the best of our knowledge, there is no open source model or algorithm for identifying NSFW images. In the spirit of collaboration and with the hope of advancing this endeavor, we are releasing our deep learning model that will allow developers to experiment with a classifier for NSFW detection, and provide feedback to us on ways to improve the classifier.