Ben Shahshahani Returns to Yahoo Labs as VP of Advertising Sciences

By Ron Brachman

I am pleased to announce the return of Ben Shahshahani to Yahoo Labs as our new Vice President of Advertising Sciences in the United States.

Yahoo Labs is home to the company’s most forward-looking thinkers, providing deep technical expertise on scientific and technical topics of critical importance to Yahoo’s future. Advertising sciences is a crucial area for Yahoo, and Ben will lead our efforts to understand fundamental principles and create innovative technology essential to connecting advertisers to the right audiences at the right time. With Ben’s guidance, the team will focus on many key advertising-related scientific subjects, including, for example, efficiency, relevancy, engagement, ad effectiveness, marketplaces, and increasing advertiser ROI.

An accomplished engineer and researcher, Ben has years of experience that will serve him well in his new role. Most recently he served as an Engineering Director in Google’s Display Advertising team. There he managed the Display Campaign Optimization team.

Until mid-2012, Ben held the position of Vice President of Search and Media Sciences within Yahoo Labs. Among his many other responsibilities, Ben was at the helm of the development of algorithms that powered Yahoo’s search and media products including user modeling/profiling, data mining and recommendation systems, query and content processing, relevance ranking for vertical search, search assist, and page layout optimization. Ben was a long-time Yahoo, starting back in 2006.

Before Yahoo, Ben was a Research Scientist and Director of Natural Language Processing at Nuance, and part of the Speech Processing Group at IBM. Ben holds a PhD in Electrical Engineering from Purdue University and has over a dozen issued and filed patents related to online advertising, search, natural language, and speech processing.

On a personal note, I am thrilled Ben is coming home to Yahoo. Those who worked with Ben share my enthusiasm, and our many new faces ready to greet him will benefit from his leadership and expertise. We have had an incredibly exciting year at Yahoo Labs, and I couldn’t be happier to continue that momentum with Ben’s arrival.

Tackling Natural Language Generation… at Scale


By Amanda Stent

If you have used a smartphone personal assistant then you would probably agree a computer has talked to you in a “natural language” like English or Spanish.  However, it may surprise you to learn that the same is true if you have checked your email, used a shopping website, checked the weather online, tweeted with a company, or looked up directions on the Web.  In fact, the Internet today is full of a mishmash of human- and computer-generated language.

How do computers generate language?  Modern natural language generation (NLG) systems operate over raw numerical data, structured databases, or text input.  They generate language for a great variety of useful applications, including weather forecasting, financial and healthcare report generation123, and review summarization4.  They produce output using one of three basic methods.  The first, and by far most widely used, is template-based generation:  a human writes natural language text with gaps, and the computer fills the gaps in from dictionaries.  If you’ve received a form letter from a company, that was template based generation.  The second type of natural language generation is grammar based: a human writes a set of rules covering the structure of a natural language, and the computer processes the rules to produce natural language.  Example grammar-based NLG systems are the open source SimpleNLG and OpenCCG systems.  The third approach to natural language generation is statistical: the computer “reads” a lot of text (e.g. from the Web) and learns the patterns with which people write or speak.  Then it can produce those patterns.  A variation on statistical natural language generation that allows for more control uses a simple set of rules specifying the structure of the language to produce many possible outputs, and then a statistical model of text to rank those outputs so the most “human like” one can be selected.

Now let’s imagine that you wanted to make a system that talked with human users using natural language.  For example, you might want to make a mobile app that recommended restaurants, that helped users change their bad habits, that compared the stats of football players for a fantasy football league, or that played a character in a mobile game.  What would you want the NLG system to do in each case?  At a minimum, you would probably want the system to produce correct, grammatical and natural prompts and responses, in an efficient manner; that is, you would want the system’s output to capture the content of the input accurately, to be easily understandable by a human, and to appear in a reasonable amount of time. These are standard NLG evaluation metrics.

One could argue that there is more than enough natural language on the Web to give any computer a correct, grammatical and natural output for almost any input, i.e. if we can learn the mappings from language inputs to knowledge representations, we never have to build an NLG system again.  However, if you wanted to use the NLG system in an interactive context, such as encouraging users while they exercise or playing a character in a game, you would probably also want several other, less obvious, things from your NLG system.  For example, you might want the system to exhibit controlled variation.  Specifically, you might want the system to adapt its output to the context and to the user (e.g. not keep saying ‘Peyton Manning, the quarterback’ when ‘Manning’ would work for a football fan, or not say ‘Rob’s Bistro, 234 Main Street, Madison’ when the user is right across the street and it could just say ‘Rob’s Bistro, in front of you’).  In addition, if the system is representing a company or a character in a game, you might want it to exhibit personality; a villain interacts differently than a hero, and different companies have different corporate personalities. And finally, if the system is very interactive, you would want it to have good ways to manage the interaction – for example, good ways to handle errors and ambiguities. Several of these new metrics arise directly from the interactive nature of the application – essentially, you want users to be sufficiently engaged with the system that they continue the interaction. The problem is that these additional desiderata are easy for humans to understand but hard to quantify and model in a computer program, and especially so in the absence of user feedback.  We need methods for NLG that allow us to model the complexities of interaction as well as take advantage of the many sources of language data on the Web.

What is the big goal for NLG systems for interaction? What would allow us to say this AI problem had been ‘solved’? And what about the science of NLG - how can we use NLG systems to further understand human intelligence?  The famous Turing test is a test of an interactive NLG system, but in some ways the test is oddly limited – the system and human are not co-present and can interact only through text, so the interaction does not take into account physical context or the user’s history; the task is a sort of trivia quiz, so the user may not care deeply about success; and there is no social or emotional engagement element, so only a small aspect of human intelligence is examined.  At the same time, the famous experiments with the Eliza chatbot showed how easily humans can be fooled about human intelligence. What if we proposed new tests, e.g. a computer system that could convince a user to buy a product, or a virtual standup comedian?  Both of these applications involve task-related intelligence, conversational intelligence and social intelligence.  Or how about an interactive system that could be so helpful and engaging that a user would choose it over a human personal assistant?

At Yahoo, we are all about creating fun and personalized interactions to support users’ daily habits, and consequently we care deeply about issues of adaptation and engagement.  Our applications run the gamut from asynchronous interaction (e.g. Yahoo Answers, Yahoo Groups) to situated interaction (e.g. Yahoo mobile search, Aviate).  Furthermore, at Yahoo Labs we have the ability to run experiments at scale, allowing us to automatically identify the subtle features of language use that correspond, for example, to ‘helpful’ adaptation, to ‘informative’ answers or to a ‘fun’ personality.  If you are a graduate student or faculty researcher interested in questions around NLG for interaction, we invite you to contact us – we would love to collaborate. Help us design interactive systems for the future that are engaging (e.g. fun, dramatic, beautiful) as well as useful.


Di Fabbrizio, G., Stent, A., & Gaizauskas, (2013) Summarizing opinion-related information for mobile devices. In Neustein, A. & Markowitz, J. (eds). Mobile Speech and Advanced Natural Language Solutions. Springer.

Dr. Ben Shneiderman Engages With Data Visualization In Big Thinkers Talk


Last week we were honored to have had Dr. Ben Shneiderman, Professor of Computer Science and Founding Director of the Human-Computer Interaction Laboratory at the University of Maryland, present a Big Thinkers talk at Yahoo entitled, “Information Visualization for Knowledge Discovery: Big Insights from Big Data.” During his presentation, Dr. Shneiderman focused on the importance of visualization tools in answering Big Data questions and solving Big Data problems. Shneiderman enthusiastically stated that “visualization is a way of engaging people,” and that “visualizations give you answers to questions you didn’t know you had.”

Professor Shneiderman also covered his “8 Golden Rules of Data Science”:

  • Choose actionable problems and compelling theories
  • Open your mind: domain experts and statisticians
  • If you don’t have questions, you’re not ready
  • Clean, clean, clean… your data (gently on the screen)
  • Know thy data: ranges, patterns, clusters, gaps, outliers, missing values, uncertainty
  • Evaluate your efficacy, refine your theory
  • Take responsibility, reveal your failures
  • Work is complex, proceed with humility

The event was broadcast live on our homepage and viewers had the opportunity to ask questions and comment on our Twitter stream @YahooLabs as well as our Facebook page.

You can view Dr. Shneiderman’s full presentation here:

Machine Learning for (Smart) Dummies

By Aryeh Kontorovich

So how do you tell a cat from a dog? It’s something a three-year-old does with near-perfect accuracy, but not something we can formalize in simple rules or easily write code for.

When searching for “cats that look like dogs,” here’s what comes up:


Traditional Artificial Intelligence (AI) attempts to produce clean, interpretable decision rules. Modern machine learning takes this a step further: Rather than trying to feed computers man-made rules, we hope computers will discover their own rules based on examples so that many tasks requiring human input will become fully automated. In other words, we need machines to learn.

Yahoo scientists and engineers are faced with solving numerous learning-related problems. As a visiting scientist from Ben Gurion University who has, by now, spent some time working alongside Yahoos, I have come to respect the amount of hands-on experience my colleagues have with standard machine learning algorithms: SVM, boosting, nearest neighbors, decision trees…. Of course, many of these algorithms are simple and intuitive (such as, “Given a test point, predict the label of the closest training point”). But, their mathematical underpinning is not always well understood.

Using my background in theoretical machine learning research, I instructed a recent seven-week course at Yahoo with the aim of providing a theoretical foundation on which the aforementioned algorithms are based. Why does a large margin guarantee good generalization? How does one avoid overfitting? What are the “no free lunch” results in learning? What is the best learning rate one could hope for? Using rigorous mathematical tools, the course provides answers to these questions.

I am firmly of the conviction that “there is nothing so practical as a good theory.” My hope is that deep insight into common learning algorithms will give practitioners a better sense of which ones are more applicable in any given situation, and perhaps even guide other scientists, engineers, etc. in designing novel approaches.

One of the benefits of the open academic collaborations that Yahoo Labs encourages, including mine, is the knowledge transfer each party brings to the table. It is in the same spirit of collaboration and open discourse that we are offering all of the seven classes below for your professional and/or personal enrichment. I hope you find them useful.

Week 1:

Week 2:

Week 3:

Week 4:

Week 5:

Week 6:

Week 7:

Are you ready for some Tumblr data-driven football?

by Nemanja Djuric, Vladan Radosavljevic, and Mihajlo Grbovic

The summer of soccer is behind us, and sports fans across the U.S. can finally turn their attention to real football (that is, American football). After more than seven months of silence, the NFL stage is set for a new season of blood, sweat, and data. Yes, data.

Everybody hopes to see his or her favorite team clinch the coveted Vince Lombardi Trophy, but data-driven predictions are another matter. And predictions are all the more fun when you add social media to the mix.

Following the success of our World Cup predictor where we correctly forecasted three out of four semifinalists using specific Tumblr chatter, Yahoo Labs is once again using the power of data science to bring you an answer to the only question that really matters this season: Who will win? Our statistical analysis includes Tumblr posts from May through August, which we used to create a machine learning predictor based on the popularity of each team and its players according to Tumblr’s 200+ million blogs.


The first step in creating our predictor was to isolate NFL-related Tumblr posts using NFL-related hashtags, including #nfl, #american_football, #offseason, and #football, found through state-of-the-art tag-clustering technology. Then, we counted the number of team mentions in those posts using only their short names (e.g., Eagles or 49ers) as a measure of popularity of the given team on the social network. In addition, we searched all the Tumblr content for full team names (e.g., Philadelphia Eagles or San Francisco 49ers). The popularity of the teams computed in this way is represented by the following two graphs: image

Further, we took the players from each team and computed each player’s individual popularity on Tumblr. Finally, we combined the aforementioned calculations with NFL game outcomes from 2013 and trained two statistical models that separately predicted the number of touchdowns and fields goals each team would score against its opponent, factoring in whether a team plays at home or away. For more details about the mathematics behind our approach, please see “Goalr! The Science of Predicting the World Cup on Tumblr” and our associated technical paper.

When we put this plethora of data together, we were able to calculate the winner of every game in the 2014 season, as well as the overall Super Bowl champion. And, in answer to the initial question, we determined the Tumblr community believes the Denver Broncos will reign victorious. Don’t agree? Then make your voice heard on Tumblr and you could change the outcome. Let the games begin!


Week 1 schedule and predicted results:


The Shortest Path to Happiness

By Daniele Quercia

We all know how busy the world is today. People race around from place to place trying to shave off minutes from their commutes in order to squeeze in more time for other things. But what if you had a happier, more pleasant journey?

In a previous Tumblr post called “Can Cities Make us Happy?”, we summarized our preliminary work on which urban elements make people happy. We found that in London, for example, people associate public gardens and Victorian and red brick houses with beauty and happiness, and that cars and fortress-like buildings are associated with sadness.

In our latest research we put those insights to practical use in the form of maps and routes. Consider that existing mapping technologies return shortest directions. Now, imagine a new mapping tool that, instead of suggesting the shortest walking course from A to B, is able to suggest a route that is both short and pleasant. Based on our previous work, we were able to design algorithms that automatically map out the most beautiful, quiet, and happy routes between two points. Taking into account an average of the results of the three algorithms, our study showed that despite being 12% longer in length and roughly 7 and a half minutes longer in time, respondents preferred the option of taking more scenic, quiet, and happy routes.

More interestingly perhaps, our study participants in London and Boston loved to attach memories to places: both personal memories (e.g., “This is the street I gave my first kiss.”) and shared memories (e.g., “That’s where the old BBC building was.”) In “Remembrance of Things Past,” French novelist Marcel Proust described how a small bite of a madeleine cake unleashed a cascade of memories from childhood. In a similar way, our participants found places to be pleasant (or not) and memorable depending on the way they smelled and sounded. It turns out that these smells and sounds also play a role in the paths people take from one place to another. This point begs a new question with fascinating implications for the research community: What if we had a mapping tool that suggested pleasant routes based not only on aesthetics, but also on memories, smells, and sounds?

Our study produced one other compelling point worth mentioning — participants pointed out that the experience of a place changes during the course of a day. For example, one of our London participants commented, “Fleet street is beautiful because of its history. However, depending on the time of day, it can be colorless and busy leading to the opposite results.” The idea that the pleasantness of routes differs depending on the daily course of the sun, variance in temperature, and noise level is extremely insightful and nuanced.

As we continue to research the shortest path to happiness, we’re thinking about all these questions. If you find this concept as interesting as we do and live in Berlin, Boston, London, or Turin, then we’d love for you to share your memories around a few paths in your city here. You’ll be helping us with our research, and hopefully making people’s paths happier.

Yahoo Labs to Host Big Thinker Ben Shneiderman

Mark your calendars! Dr. Ben Shneiderman, Professor of Computer Science and Founding Director of the Human-Computer Interaction Laboratory at the University of Maryland, is coming to Yahoo - and your computer screen - on September 18.

His talk is entitled: Information Visualization for Knowledge Discovery: Big Insights from Big Data”

Watch LIVE on the Yahoo Labs website homepage and ask questions or comment on the Yahoo Labs Facebook and Twitter (#BigThinkers) pages. 


Interactive information visualization tools provide researchers with remarkable capabilities to support discovery from Big Data resources. Users can begin with an overview, zoom in on areas of interest, filter out unwanted items, and then click for details-on-demand. The Big Data initiatives and commercial success stories such as Spotfire and Tableau, plus widespread use by prominent sites such as The New York Times have made visualization a key technology.

The central theme is the integration of statistics with visualization as applied for time series data, temporal event sequences such as electronic health records (, and social network data (  By temporal pattern search & replace and network motif simplification, complex data streams can be analyzed to find meaningful patterns and important exceptions.  The talk closes with 8 Golden Rules for Big Data. 


Ben Shneiderman ( is a Distinguished University Professor in the Department of Computer Science and Founding Director (1983-2000) of the Human-Computer Interaction Laboratory ( at the University of Maryland.  He is a Fellow of the AAAS, ACM, and IEEE, and a Member of the National Academy of Engineering, in recognition of his pioneering contributions to human-computer interaction and information visualization. His contributions include the direct manipulation concept, clickable web-link, touchscreen keyboards, dynamic query sliders for Spotfire, development of treemaps, innovative network visualization strategies for NodeXL, and temporal event sequence analysis for electronic health records.

Ben is the co-author with Catherine Plaisant of Designing the User Interface: Strategies for Effective Human-Computer Interaction (5th ed., 2010)  With Stu Card and Jock Mackinlay, he co-authored Readings in Information Visualization: Using Vision to Think (1999).  His book Leonardo’s Laptop appeared in October 2002 (MIT Press) and won the IEEE book award for Distinguished Literary Contribution.  His latest book, with Derek Hansen and Marc Smith, is Analyzing Social Media Networks with NodeXL(, 2010). 


Yahoo Labs is proud to bring you its 2014 Big Thinkers Speaker Series. Each year, some of the most influential, accomplished experts from the research community visit our campus to share their insights on topics that are significant to Yahoo. These distinctive speakers are shaping the future of the new sciences underlying the Web and are guaranteed to inform, enlighten, and inspire.

Shaky Research: Yahoo Labs Contributes to New Study on Servers and Seismic Activity

By Don McGillen

Have you ever wondered what happens to computer servers if you shake them really hard? The loss of functionality of data and telecommunication centers could have a disastrous impact on emergency operations and in the ability of communities to respond and recover when an earthquake hits. That’s why Howard University Associate Professor Claudia Marin-Artieda thinks about it all the time. In fact Dr. Marin-Artieda, who works in Howard’s Civil and Environmental Engineering Department, received a National Science Foundation (NSF) Career Award to study seismic protection systems for equipment and components in multi-story facilities that include data centers and computer-based communication centers. And now we’re helping her find out the answer to that question.  

Our Academic Relations (AR) team has been working hard to develop a strong and rich relationship with Howard University. Of the Historically Black Colleges and Universities (HBCUs), Howard is arguably the most elite in Computer Science, and the only HBCU to offer a PhD in the subject. Over the past year our AR team – along with Yahoo colleagues in our Washington, D.C. office – has partnered with Howard on a number of exciting initiatives, including recently hosting 25 young future leaders from various African nations as part of the Obama administration’s Young African Leaders Initiative.

As part of another program called Yahoo Servers to Academic Researchers (YSTAR), we donated 125 servers from our data centers to Howard last fall. The gift is enabling education initiatives never before possible at the university, and is spurring research with partnering institutions like the State University of New York at Buffalo through the Network for Earthquake Engineering Simulation (NEES) sponsorship. 


 YSTAR donation to Howard University

It is at the University at Buffalo that full-scale laboratory tests are currently being conducted on a frame and 40 servers donated by Yahoo. The seismic performance of the Yahoo frame will be tested on its own and supported on seismic isolated platforms under three-directional earthquake shaking. Dr. Marin-Artieda says, “The studies will provide valuable information regarding the validation of seismic solutions to achieve a desired protection level in essential facilities that are currently lacking. These studies are relevant since they will provide data on 1) deformation levels under severe earthquake shaking that are imposed to equipment-systems in essential facilities to achieve functionality requirements, 2) experimental data on systems characterization that is currently lacking, 3) validation of seismic solutions to achieve a desired protection level, etc.”


Left: NEES facility and shake tables, Right: Yahoo-donated servers and frame

Marin adds that, “Implementing the seismic protective options emerging from this research will reduce the vulnerabilities during and after an earthquake of data centers and telecommunication centers. The research is directly addressing critical needs of the earthquake engineering community by validating high-performance options to protect equipment and components of essential facilities.”

At Yahoo Labs, through the engagement of our Academic Relations team, we are thrilled to support such crucial research with such high-stakes, real-world impact. And since our headquarters is in one of the world’s most earthquake-prone locations, this study holds a special place in our heart!

For more on Dr. Claudia Marin-Artieda’s work, please see