Posted as part of our April Fool's Day edition.
Scientists at Yahoo! today released Ideological Search, allowing users to control the ideology of their search results for the first time in search technology history. Until now, many Web search users were offended by the facts, pages, articles, and blogs in their search results that contradicted their own personal beliefs and values. Furthermore, search engines were often accused of being biased in one direction or another. Rather than try to comply with a hard-to-define “search fairness doctrine”, Yahoo! Ideological Search will allow its users to personally control the ideological perspective of their search. Users can now search with confidence, knowing that their search results will perfectly match their ideology, and no results will offend them. “This is a gigantic step in the personalization of search and we are excited to bring this key technology to market ahead of our competitors, " said Dr. Prabhakar Raghavan, Chief Strategist of Yahoo! Search and head of the Yahoo! Labs organization that pioneered the technology.
The team applied the latest research from the fields of sentiment analysis, intent detection, eye tracking, clustering, and empathetic reasoning to create Ideological Search. Current research directions stop at simple social search, faceted search, safe search, contextual search, while Ideological Search goes much further to give users only the information they want.
Every person in Yahoo Research! was involved in developing Ideological Search. For example, Pig, the large scale data processing environment had to be specifically adapted to Ideological Search. "Pig was used to build the index for Ideological Search; however, not all ideologies were compatible with Pig, especially those against the exploitation of animals, " said Benjamin Reed from the Pig project. "We created a sister project, called Tofu, with the same Pig codebase to process some ideologies. Tofu was less optimized, but more flexible and gelatinous."
The motivation for Ideological Search originated largely from the Microeconomics group. "It is not just deep science, it is also a strategic move to accelerate the fractionalization of the search marketplace which can only benefit us," said Preston McAfee, head of the group.
According to Andrei Broder from Computational Advertising, "Our customer is king! We should believe what they believe." This made the advertising infrastructure much more complicated. "Switching between ideologies millions of times per second was extremely challenging, " he said. "Although it is difficult to keep up, the accuracy of our targeted advertising has greatly increased."
Dr. Raghavan explained that dedicated members of each core ideology were brought into the lab through a judicious and ideologically appropriate choice of financial incentives and appeals to their strong-held convictions. The subjects were injected with a mildly radioactive dye, which crossed the blood-brain barrier, allowing Yahoo! scientists to monitor emotional responses in the mid cortex in response to results present in search results. Based on this data, the search team trained IMLR (Ideological Machine Learned Ranking) functions on a per-ideology basis were then applied as appropriate to incoming searches.
The Search Technologies group took a different approach to detecting ideology. They found that when they took on the belief systems of Yahoo! users with more extreme ideologies to process their searches, they could hack into election offices and gather voting records. The group indexed and clustered these records using their proprietary Ideological Detection Indexing Of Terms (IDIOT) technology. With IDIOT, users can enter their name, birthday, and last voting location and to have their ideology determined. According to Pras Sarkar, Lead Engineer on the project, "Building a smarter IDIOT technology was no easy feat. We created a new framework called Clorocks - an asynchronous platform for harvesting vital ideological information through users' social networks, inbox reading habits, TIVO programming, etc. Clorocks is built on previous Yahoo! Research advancements like Getoutameway (queue processing) and BigBird (high performance database system)."
One side benefit of the ideology detection is that users no longer need to wonder about their world view. Now that the ideology of user can be identified, the team will be developing Yahoo! Me, a new service will tell users what they should see, where they should work and live, what they should buy, who they should date, and how they should vote. The anticipated outcome is that the service will allow users to spend less time worrying about these stressful topics and more time patronizing advertisers. However, Ricardo Baeza-Yates of Yahoo! Europe was disappointed: “This is yet another instance of search results being US-centric; we hope to bring these ideas to European politics in the near future”.