Overview

AAAI HCOMP is the premier venue for disseminating the latest research findings on crowdsourcing and human computation. While artificial intelligence (AI) and human-computer interaction (HCI) represent traditional mainstays of the conference, HCOMP believes strongly in inviting, fostering, and promoting broad, interdisciplinary research. This field is particularly unique in the diversity of disciplines it draws upon, and contributes to, ranging from human-centered qualitative studies and HCI design, to computer science and artificial intelligence, economics and the social sciences, all the way to policy and ethics. We promote the exchange of scientific advances in human computation and crowdsourcing not only among researchers, but also engineers and practitioners, to encourage dialogue across disciplines and communities of practice.

Past meetings include eleven AAAI HCOMP conferences (2013-2023) and four earlier workshops, held at the AAAI Conference on Artificial Intelligence (2011-2012), and the ACM SIGKDD Conference on Knowledge Discovery and Data Mining (2009-2010).

An overview of the history, goals, and peer review procedures of the conference can be found in the preface to the HCOMP-13 proceedings. Additional background on the founding of the conference is discussed in a Computing Research News story.

Past Meetings


HCOMP 2013 Photo Album

Past Proceedings

Past Paper Awards

  • 2023

Best Paper Awards
Where Does My Model Underperform? A Human Evaluation of Slice Discovery Algorithms
Nari Johnson, Ángel Alexander Cabrera, Gregory Plumb and Ameet Talwalkar

Confidence Contours: Uncertainty-Aware Annotation for Medical Semantic Segmentation
Andre Ye, Quan Ze Chen and Amy Zhang

Honorable Mention Paper Awards
Collect, Measure, Repeat: Reliability Factors for Responsible AI Data Collection
Oana Inel, Tim Draws and Lora Aroyo

A Taxonomy of Human and ML Strengths in Decision-Making to Investigate Human-ML Complementarity
Charvi Rastogi, Liu Leqi, Kenneth Holstein and Hoda Heidari

  • 2022

Best Paper Award
It Is like Finding a Polar Bear in the Savannah! Concept-Level AI Explanations with Analogical Inference from Commonsense Knowledge
Gaole He, Agathe Balayn, Stefan Buijsman, Jie Yang, and Ujwal Gadiraju

Honorable Mention Paper Awards
Connecting Algorithmic Research and Usage Contexts: A Perspective of Contextualized Evaluation for Explainable AI
Q. Vera Liao, Yunfeng Zhang, Ronny Luss, Finale Doshi-Velez, and Amit Dhurandhar

Near-Optimal Reviewer Splitting in Two-Phase Paper Reviewing and Conference Experiment Design
Steven Jecmen, Hanrui Zhang, Ryan Liu, Fei Fang, Vincent Conitzer, and Nihar B. Shah

Best Poster/Demo Award
Human-in-the-loop mixup
Katherine Collins, Umang Bhatt, Weiyang Liu, Bradley Love, and Adrian Weller

  • 2021

Best Paper Award
A Checklist to Combat Cognitive Biases in Crowdsourcing
Tim Draws, Alisa Rieger, Oana Inel, Ujwal Gadiraju and Nava Tintarev

Honorable Mention Paper Awards
On the Bayesian Rational Assumption in Information Design
Wei Tang and Chien-Ju Ho

Rapid Instance-level Knowledge Acquisition from Class-level Common Sense
Chris Welty, Lora Aroyo, Flip Korn, Sara Marie Mc Carthy and Shubin Zhao

Best Blue Sky Ideas
First Place: The Science of Rejection: A Research Area for Human Computation
Burcu Sayin, Jie Yang, Andrea Passerini and Fabio Casati

Second Place: Human Computation and Crowdsourcing for Earth
Yasaman Rohanifar, Syed Ishtiaque Ahmed, Sharifa Sultana, Prateek Chanda and Malay Bhattacharyya

Third Place: Human in the Loop for Machine Creativity
Neo Christopher Chung

Best Poster/Demo Award
FindItOut: A Multiplayer GWAP for Collecting Plural Knowledge
Agathe Balayn, Gaole He, Andrea Hu, Jie Yang and Ujwal Gadiraju

  • 2020

Best Paper Award
Motivating Novice Crowd Workers through Goal Setting: An Investigation into the Effects on Complex Crowdsourcing Task Training
Amy Rechkemmer and Ming Yin

Best Student Paper
Impact of Algorithmic Decision Making on Human Behavior: Evidence from Ultimatum Bargaining
Alexander Erlei, Franck Awounang Nekdem, Lukas Meub, Avishek Anand and Ujwal Gadiraju

Best Blue Sky Ideas
First Place: Using Human Cognitive Limitations to Enable New Systems
Vincent Conitzer

Second Place: Group-Assign: Type Theoretic Framework for Human AI Orchestration
Aik Beng Ng, Zhangsheng Lai, Simon See and Shaowei Lin

Best Demo Award
OpenTag: Understanding Human Perceptions of Image Tagging Algorithms
Kyriakos Kyriakou, Pınar Barlas, Styliani Kleanthous and Jahna Otterbacher

Best Work-in-Progress Award
Assessing Political Bias using Crowdsourced Pairwise Comparisons
Tzu-Sheng Kuo, Mcardle Hankin, Miranda Li, Andrew Ying and Cathy Wang

  • 2019

Best Paper Award
A Large-Scale Study of the "Wisdom of Crowds"
Camelia Simoiu, Chiraag Sumanth, Alok Shankar and Sharad Goel

Best Paper Finalists
Human Evaluation of Models Built for Interpretability
Isaac Lage, Emily Chen, Jeffrey He, Menaka Narayanan, Been Kim, Samuel Gershman and Finale Doshi-Velez

Fair Work: Crowd Work Minimum Wage with One Line of Code
Mark Whiting, Grant Hugh and Michael Bernstein

Best Poster / Demo Presentation
PairWise: Mitigating Political Bias in Crowdsourced Content Moderation
Jacob Thebault-Spieker, Sukrit Venkatagiri, David Mitchell, Chris Hurt and Kurt Luther

  • 2018

Best Paper Award
All That Glitters is Gold -- An Attack Scheme on Gold Questions in Crowdsourcing
Alessandro Checco, Jo Bates and Gianluca Demartini

Best Paper Finalists
Towards Accountable AI: Hybrid Human-Machine Analyses for Characterizing System Failure
Besmira Nushi, Ece Kamar and Eric Horvitz

Best Poster / Demo Presentation
Are 1,000 Features Worth A Picture? Combining Crowdsourcing and Face Recognition to Identify Civil War Soldiers
Vikram Mohanty, David Thames, Kurt Luther

  • 2017

Best Paper Award
Toward Scalable Social Alt Text: Conversational Crowdsourcing as a Tool for Refining Vision-to-Language Technology for the Blind
Elliot Salisbury, Ece Kamar, and Meredith Ringel Morris

Best Paper Finalists
Supporting Image Geolocation with Diagramming and Crowdsourcing
Rachel Kohler, John Purviance, and Kurt Luther

  • 2016

Best Paper Award
Why Is That Relevant? Collecting Annotator Rationales for Relevance Judgments
Tyler Mcdonnell, Matthew Lease, Mucahid Kutlu and Tamer Elsayed

Best Paper Finalists
Efficient Techniques for Crowdsourced Top-k Lists
Luca de Alfaro, Vassilis Polychronopoulos and Neoklis Polyzotis

Extending Workers' Attention Span Through Dummy Events
Avshalom Elmalech, David Sarne, Esther David and Chen Hajaj

  • 2015

Best Paper Award
Crowdsourcing in the Field: A Case Study Using Local Crowds for Event Reporting
Elena Agapie, Jaime Teevan, and Andres Monroy-Hernandez

Best Paper Finalists
Tropel: Crowdsourcing Detectors with Minimal Training
Genevieve Patterson, Grant Van Horn, Serge Belongie, Pietro Perona, and James Hays

Crowdsourcing from Scratch: A Pragmatic Experiment in Data Collection by Novice Requesters
Alexandra Papoutsaki, Hua Guo, Metaxa-Kakavouli, Connor Gramazio, Jeff Rasley, Wenting Xie, Guan Wang, and Jeff Huang

  • 2014

Best Paper Award
STEP: A Scalable Testing and Evaluation Platform
Maria Christoforaki and Panos Ipeirotis

Best Paper Finalists
A Crowd of Your Own: Crowdsourcing for On-Demand Personalization
Peter Organisciak, Jaime Teevan, Susan Dumais, Robert Miller, and Adam Tauman Kalai

Crowdsourcing for Participatory Democracies: Efficient Elicitation of Social Choice Functions
David Lee, Ashish Goel, Tanja Aitamurto, and Helene Landemore

  • 2013

Best Paper Award
Crowdsourcing Multi-Label Classification for Taxonomy Creation
Jonathan Bragg, Mausam, and Daniel S. Weld

Best Paper Finalists
nEmesis: Which Restaurants Should You Avoid Today?
Adam Sadilek, Sean Brennan, Henry Kautz, and Vincent Silenzio

Community Clustering: Leveraging an Academic Crowd to Form Coherent Conference Sessions
Paul Andre, Haoqi Zhang, Juho Kim, Lydia Chilton, Steven P. Dow, and Robert C. Miller

Library

  • Past Reports

Workshops Held at the First AAAI Conference on Human Computation and Crowdsourcing: A Report. Tatiana Josephy, Matthew Lease, Praveen Paritosh, Markus Krause, Mihai Georgescu, Michael Tjalve, and Daniela Braga. AI Magazine, 35(2), 75-78, 2014.

Shar Steed. Harnessing Human Intellect for Computing. Computing Research Association (CRA) Computing Research News, Vol. 25 No. 2, Feburary 2013.

A report on the human computation workshop (HCOMP 2009). Panagiotis G. Ipeirotis, Raman Chandrasekar, and Paul N. Bennett. SIGKDD Explorations 11, no. 2 (2009): 80-83.

  • Stay CONNECTED: HCOMP COMMUNITY

We welcome everyone who is interested in crowdsourcing and human computation to:

  • Join crowd-hcomp Google Group (mailing list) to post and receive crowdsourcing and human computation email announcements (e.g., calls-for-papers, job openings, etc.) including updates here about the conference. To subscribe send an email to crowd-hcomp+subscribe@googlegroups.com.
  • Check our Google Group webpage to view the archive of past communications on the HCOMP mailing list.
  • Keep track of our twitter hashtag #HCOMP2024.
  • Join the HCOMP Slack Community to be in touch with researchers, industry players, practitioners, and crowd workers around Human Computation and relevant topics.