Opening (8:55 - 9:00)

Invited Talk: Jeff Bigham (9:00 - 9:40)

“Crowd Agents: Interactive Crowd-Powered Systems in the Real World”


In this talk, I'll discuss several interactive crowd-powered systems that help people address real-world problems. For instance, VizWiz sends questions blind people have about their visual environment to the crowd, Legion allows outsourcing of desktop tasks to the crowd, and Scribe allows the crowd to caption audio in real-time. The thousands of people have engaged with these systems, providing an interesting look at how end users want to interact with crowd work.

Collectively, these systems illustrate a new approach to human computation in which the dynamic crowd is provided the computational support needed to act as a single, high-quality agent. The classic advantage of the crowd has been its wisdom, but our systems are beginning to show how crowd agents can surpass even expert individuals on motor and cognitive performance tasks.


Jeffrey P. Bigham is an Assistant Professor in the Department of Computer Science at the University of Rochester where he heads the ROC HCI Group. His work is at the intersection of human-computer interaction, human computation, and artificial intelligence, with a focus on developing innovative technology that serves people with disabilities in their everyday lives.  Jeffrey received his B.S.E degree in Computer Science fromPrinceton University in 2003. He received his M.Sc. degree in 2005 and his Ph.D. in 2009, both in Computer Science and Engineering from the University of Washington working with Richard E. Ladner. For his innovative research, Dr. Bigham has won a number of awards, including the Microsoft Imagine Cup Accessible Technology Award, the Andrew W. Mellon Foundation Award for Technology Collaboration, the MIT Technology Review Top 35 Innovators Under 35 Award, and Best Paper Awards at UIST, WSDM, and ASSETS. In 2012, he received the National Science Foundation CAREER Award.

Session 1: Games (9:40 - 10:20)

9:40 - 10:00      Systematic Analysis of Output Agreement Games: Effects of Gaming Environment,

                         Social Interaction, and Feedback

                         Shih-Wen Huang, UIUC

                         Wai-Tat Fu, University of Illinois at Urbana-Champaign

10:00 - 10:20    Doodling: A Gaming Paradigm for Generating Language Data

                         A Kumaran, Microsoft Research India

                         Sujay Jauhar, University of Wolverhampton

                         Sumit Basu, Microsoft Research

1st Coffee Break / Poster Session (10:20 - 11:00)

CAPTCHAs with a Purpose

Suhas Aggarwal, IIT Guwahati

Crowd-Sourcing Design: Sketch Minimization using Crowds for Feedback

David Engel, MIT

Verena Kottler, Max Planck Institute for Developmental Biology

Christoph Malisi, Max Planck Institute for Developmental Biology

Marc Röttig, University of Tuebingen Center for Bioinformatics

Eva Willing, Max Planck Institute for Plant~Breeding Research

Sebastian Schultheiss, Max Planck Institute

To Crowdsource or Not to Crowdsource?

Gireeja Ranade, UC Berkeley

Lav R. Varshney, IBM Thomas J. Watson Research Center

Learning from Crowds and Experts

Hiroshi Kajino, The University of Tokyo

Yuta Tsuboi, IBM Research - Tokyo

Issei Sato, The University of Tokyo

Hisashi Kashima, The University of Tokyo

Squaring and Scripting the ESP Game

François Bry, Ludwig-Maximilian University

Christoph Wieser, Ludwig-Maximilian University of Munich

Automatically providing action plans helps people complete tasks

Nicolas Kokkalis, Stanford

Scott Klemmer, Stanford

Thomas Koehn, Stanford

The Role of Super Agents in Mobile Crowdsourcing

Mohamed Musthag, Univ. of Massachusetts, Amhers

Deepak Ganesan, University of Massachusetts

Detecting Deceptive Opinion Spam using Human Computation

Christopher Harris, The University of Iowa

Improving Quality of Crowdsourced Labels via Probabilistic Matrix Factorization

Hyun Joon Jung, University of Texas at Austin

Matthew Lease, University of Texas at Austin

Towards Social Norm Design for Crowdsourcing Markets

Chien-Ju Ho, UCLA

Yu Zhang, UCLA

Jennifer Wortman Vaughan, UCLA

Mihaela van der Schaar, UCLA

Session 2: Machine Learning (11:00 - 11:40)

11:00 - 11:20    Crowdclustering with Sparse Pairwise Labels: A Matrix Completion Approach

                         Jinfeng Yi, Michigan State University

                         Rong Jin, Michigan State University

                         Anil Jain, Michigan State University

                         Shaili Jain, Yale University

11:20 - 11:40    Crowdsourcing Control: Moving Beyond Multiple Choice

                         Christopher Lin, University of Washington

                         Mausam, University of Washington

                         Daniel Weld, University of Washington

Session 3: Platforms (11:40 - 12:20)

11:40 - 12:00    TurkServer: Enabling Synchronous and Longitudinal Online Experiments

                         Andrew Mao, Harvard University

                         Yiling Chen, Harvard University

                         Krzysztof Gajos, Harvard University

                         David Parkes, Harvard University

                         Ariel Procaccia, Carnegie Mellon University

                         Haoqi Zhang, Harvard University

12:00 - 12:20    MobileWorks: A Non-Marketplace Architecture for Accurate Human Computation

                         Anand Kulkarni*, MobileWorks / UC Berkeley

                         Philipp Gutheim, MobileWorks, UC Berkeley

                         Prayag Narula, MobileWorks, UC Berkeley

                         David Rolnitzky, MobileWorks

                         Tapan Parikh, University of California, Berkeley

                         Bjoern Hartmann, University of California, Berkeley

Lunch (12:20 - 1:40)

Invited Talk: Adam Kalai (1:40 - 2:20)

“Programming by Example Revisted”

Abstract:  An old dream is to program computers through examples rather than by writing code.  One particularly common domain is text-processing: automating tasks such as extracting author names from a bibliography, removing duplicate entries in a log file.  As in many human computation problems, a key aspect of the problem is breaking a task into small steps, each of which is easy to understand.  Over the years, several systems have been built with this aim, yet none have been widely adopted.   We discuss why this is the case, surveying the earlier systems, and describe what a handful of researchers are doing to address the problem. 
Joint work with: Sumit Gulwani, Butler Lampson, Aditya Menon, Rob Miller, Omer Tamuz, Shubham Tulsiani, and Kuat Yessenov

Bio: Adam Tauman Kalai received his BA (1996) from Harvard, and MA (1998) and PhD (2001) under the supervision of Avrim Blum from CMU. After an NSF postdoctoral fellowship at M.I.T. with Santosh Vempala, he served as an assistant professor at the Toyota Technological institute at Chicago and then at Georgia Tech. He is now a Senior Researcher at Microsoft Research New England. His honors include an NSF CAREER award, and an Alfred P. Sloan fellowship. His research focuses on computational learning theory, human computation, and algorithms.

Session 4: Applications (2:20 - 3:40)

2:20 - 2:40        Crowdsourcing Annotations for Visual Object Detection

                         Hao Su, Stanford University

                         Jia Deng, Stanford University

                         Fei-Fei Li, Stanford University

2:40 - 3:00        Part Annotations via Pairwise Correspondence

                         Subhransu Maji, TTIC

                         Greg Shakhnarovich, TTIC

3:00 - 3:20        Contextual Commonsense Knowledge Acquisition from Social Content by        

                         Crowdsourcing Explanations

                         Yen-Ling Kuo, National Taiwan University

                         Jane Yung-jen Hsu, National Taiwan University

                         Fuming Shih,MIT

3:20 - 3:40        Hallucination: a mixed-initiative approach for efficient document reconstruction

                         Haoqi Zhang, Harvard University

                         John Lai, Harvard University

                         Moritz Baecher, Harvard University

2nd Coffee Break / Poster Session (3:40 - 4:20)

Social Choice for Human Computation

Andrew Mao, Harvard University

Ariel Procaccia, Carnegie Mellon University

Yiling Chen, Harvard University

Predicting Crowd-based Translation Quality with Language-independent Feature Vectors

Markus Krause, University of Bremen

Jan Smeddinck, University of Bremen

Niklas Kilian,

Nina Runge, Uni Bremen

Machine-learning for Spammer Detection in Crowd-sourcing

Harry Halpin, MIT

Roi Blanco , Yahoo! Research

Crowdsourcing: Dynamically Switching between Synergistic Workflows

Christopher Lin, University of Washington

Mausam, University of Washington

Daniel Weld, University of Washington

Learning Sociocultural Knowledge via Crowdsourced Examples

Mark Riedl, Georgia Institute of Technolog

Boyang Li, Georgia Institute of Technology

Stephen Lee-Urban, Georgia Institute of Technology

Darren Appling, Georgia Institute of Technology

Playful Surveys: Easing Challenges of Human Subject Research with Online Crowds

Markus Krause, University of Bremen

Jan Smeddinck, University of Bremen

Aneta Takhtamysheva, University of Bremen

Velislav Markov, University of Bremen

Nina Runge, Uni Bremen

Personalized Online Education—A Crowdsourcing Challenge

Daniel Weld, University of Washington

Eric Horvitz, Microsoft Research

Raphael Hoffmann, University of Washington

Eytan Adar, University of Michigan

Lydia Chilton, Unviersity of Washington

Mitchell Koch, Unviersity of Washington

Christopher Lin, University of Washington

Mausam, University of Washington

Using the Crowd to Do Natural Language Programming

Mehdi Manshadi, University of Rochester

Carolyn Keenan, University of Rochester

James Allen, University of Rochester

Diamonds From the Rough: Improving Drawing, Painting, and Singing via Crowdsourcing

Yotam Gingold, Rutgers / Columbia

Etienne Vouga, Columbia University

Eitan Grinspun, Columbia University

Haym Hirsh, Rutgers University

Collecting Representative Pictures for Words: A Human Computation Approach based on Draw Something Game

Jun Wang, Syracuse University

Bei Yu, Syracuse University

Business Meeting: 1st AAAI Human Computation Conference 2013 (4:20 - 5:20)

led by Eric Horvtiz

Closing (5:20-5:30)

Workshop Dinner (6:30 - 9:00)