8:45 am - 9:00 am
9:00 am - 9:30 am
9:30 am - 9:45 am
9:45 am - 10:00 am
10:00 am - 10:15 am
10:15 am - 10:30 am
10:30 am -- 11:15 am
11:15 am - 11:30 am
11:30 am -- 11:45 am
11:45 am -- 12:00 pm
12:00 pm - 12:15 pm
12:15 pm - 12:30 pm
12:30 pm - 2:00 pm
2:00 pm - 2:30 pm
2:30 pm - 2:45 pm
2:45 pm - 3:00 pm
3:00 pm - 3:15 pm
3:15 pm - 3:30 pm
3:30 pm - 4:15 pm
4:15 pm - 4:30 pm
4:30 pm - 4:45 pm
4:45 pm - 5:00 pm
5:00 pm - 5:30 pm
5:30 pm - 5:45 pm
7:30 pm - onward
Opening
Invited Talk: Eric Horvitz
SESSION 1: AI / Machine Learning
Large-Scale Live Active Learning: Training Object Detectors with Crawled Data and Crowds
Sudheendra Vijayanarasimhan, Kristen Grauman (UT Austin)
Robust Active Learning using Crowdsourced Annotations for Activity Recognition
Liyue Zhao, Gita Sukthankar (UCF); Rahul Sukthankar (Google Research/CMU)
Beat the Machine: Challenging workers to find the unknown unknowns
Josh Attenberg, Panos Ipeirotis, Foster Provost (NYU)
Human Intelligence Needs Artificial Intelligence
Daniel Weld, Mausam, Peng Dai (University of Washington)
Coffee Break / Poster Session 1
Towards Task Recommendation in Micro-Task Markets
Vamsi Ambati, Stephan Vogel, Jaime Carbonell (CMU)
On Quality Control and Machine Learning in Crowdsourcing
Matthew Lease (UT Austin)
CollabMap: Augmenting Maps using the Wisdom of Crowds
Ruben Stranders, Sarvapali Ramchurn, Bing Shi, Nicholas Jennings (University of Southampton)
Improving Consensus Accuracy via Z-score and Weighted Voting
Hyun Joon Jung, Matthew Lease (UT Austin)
Making Searchable Melodies: Human vs. Machine
Mark Cartwright, Zafar Rafii, Jinyu Han, Bryan Pardo (Northwestern University)
PulaCloud: Using Human Computation to Enable Development at the Bottom of the Economic Ladder
Andrew Schriner (University of Cincinnati); Daniel Oerther (Missouri University of Science and Technology); James Uber (University of Cincinnati)
Towards Large-Scale Processing of Simple Tasks with Mechanical Turk
Paul Wais, Shivaram Lingamneni, Duncan Cook, Jason Fennell, Benjamin Goldenberg, Daniel Lubarov, David Marin, Hari Simons (Yelp, inc.)
Learning to Rank From a Noisy Crowd
Abhimanu Kumar, Matthew Lease (UT Austin)
SESSION 2: The “Humans” in the Loop
Worker Motivation in Crowdsourcing and Human Computation
Nicolas Kaufmann; Thimo Schulze (University of Mannheim)
Honesty in an Online Labor Market
Winter Mason, Siddharth Suri, Daniel Goldstein (Yahoo! Research)
Building a Persistent Workforce on Mechanical Turk for Multilingual Data Collection
David Chen (UT Austin); William Dolan (Microsoft Research)
SESSION 3: Tools / Applications
CrowdSight: Rapidly Prototyping Intelligent Visual Processing Apps
Mario Rodriguez (UCSC); James Davis
Digitalkoot: Making Old Archives Accessible Using Crowdsourcing
Otto Chrons, Sami Sundell (Microtask)
LUNCH
Invited Talk: Jenn Wortman Vaughan
SESSION 4: Quality Control
Error Detection and Correction in Human Computation: Lessons from the WPA
David Alan Grier (GWU)
Programmatic gold: targeted and scalable quality assurance in crowdsourcing
Dave Oleson, Vaughn Hester, Alex Sorokin, Greg Laughlin, John Le, Lukas Biewald (CrowdFlower)
An Iterative Dual Pathway Structure for Speech-to-Text Transcription
Beatrice Liem, Haoqi Zhang, Yiling Chen (Harvard University)
An Extendable Toolkit for Managing Quality of Human-based Electronic Services
David Bermbach, Robert Kern, Pascal Wichmann, Sandra Rath, Christian Zirpins (KIT)
Coffee Break / Poster Session 2
Developing Scripts to Teach Social Skills: Can the Crowd Assist the Author?
Fatima Boujarwah, Jennifer Kim, Gregory Abowd, Rosa Arriaga (Georgia Tech)
CrowdLang - First Steps Towards Programmable Human Computers for General Computation
Patrick Minder, Abraham Bernstein (University of Zurich)
Ranking Images on Semantic Attributes using CollaboRank
Jeroen Janssens, Eric Postma, Jaap Van den Herik (Tilburg University)
Artificial Intelligence for Artificial Artificial Intelligence
Peng Dai, Mausam, Daniel Weld (University of Washington)
One Step beyond Independent Agreement: A Tournament Selection Approach for Quality Assurance of Human Computation Tasks
Yu-An Sun, Shourya Roy (Xerox); Greg Little (MIT CSAIL)
Turkomatic: Automatic, Recursive Task and Workflow Design for Mechanical Turk
Anand Kulkarni, Matthew Can, Bjoern Hartmann (UC Berkeley)
MuSweeper: Collect Mutual Exclusions with Extensive Game
Tao-Hsuan Chang, Cheng-wei Chan, Jane Yung-jen Hsu (National Taiwan University)
MobileWorks: A Mobile Crowdsourcing Platform for Workers at the Bottom of the Pyramid
Prayag Narula, Philipp Gutheim, David Rolnitzky, Anand Kulkarni, Bjoern Hartmann (UC Berkeley)
SESSION 5: Pricing and Allocation
What’s the Right Price? Pricing Tasks for Finishing on Time
Siamak Faridani, Bjoern Hartmann (UC Berkeley); Panos Ipeirotis (NYU)
Pricing Mechanisms for Online Labor Market
Yaron Singer, Manas Mittal (UC Berkeley EECS)
Labor Allocation in Paid Crowdsourcing: Experimental Evidence on Positioning, Nudges and Prices
John Horton (ODesk); Dana Chandler (MIT)
SESSION 6: Discussions
Closing Remarks
Workshop Dinner
