Program

  • Keynotes

Find detailed information about the keynotes.

  • Accepted Papers

View a list of accepted papers.

View the conference proceedings.

  • Graduate Consortium

View a list of accepted papers and videos.

  • Works-in-Progress and Demonstrations

View a list of accepted papers.

  • Blue Sky Ideas Papers

View a list of accepted papers.

  • CrowdCamp

Read more about the CrowdCamp here.

  • Social Events

We will have many social events to help you have fun at HCOMP2021! Please click on the Virtual Chair event link in the schedule below to attend these events and other sessions.


  • HCOMP-21 SCHEDULE

   To attend sessions, please visit this Virtual Chair event link. To attend, you will first need to register for HCOMP and you will receive instructions on how to join each session.


   All times are EST (Eastern Standard Time). You can check what that means where you live by clicking here.


Sunday, Nov 14, 2021

8:00 am - 12:00 noon EST CrowdCamp (Session 1)

Chairs: Ting-Hao ‘Kenneth’ Huang and Amy Pavel

 Seminar Room
10:00 am - 3:45 pm EST Graduate Consortium

Chairs: Saiph Savage and David Sarne

 Plenary Room
8:00 pm - 12:00 midnight EST CrowdCamp (Session 2)

Chairs: Ting-Hao ‘Kenneth’ Huang and Amy Pavel

 Seminar Room

Monday, Nov 15, 2021

10:30 am - 11:00 am EST HCOMP Opening

Chairs: Ece Kamar and Kurt Luther 		Plenary Room
11:00 am - 12:00 pm EST Keynote 1: Chris Lintott

Chair: Lora Arroyo

 Plenary Room
12:00 pm - 12:30 pm EST

Break

 Lobby
12:30 pm - 1:30 pm EST

Paper Session 1: Understanding Crowds

Chair: Chris Welty

Iterative Quality Control Strategies for Expert Medical Image Labeling Beverly Freeman, Naama Hammel, Sonia Phene, Abigail Huang, Rebecca Ackermann, Olga Kanzheleva, Miles Hutson, Caitlin Taggart, Quang Duong and Rory Sayres
A Checklist to Combat Cognitive Biases in Crowdsourcing
Amazon Best Paper 		Tim Draws, Alisa Rieger, Oana Inel, Ujwal Gadiraju and Nava Tintarev
Exploring the Music Perception Skills of Crowd Workers Ioannis Petros Samiotis, Sihang Qiu, Christoph Lofi, Jie Yang, Ujwal Gadiraju and Alessandro Bozzon
On the Bayesian Rational Assumption in Information Design
Honorable Mention Paper 		Wei Tang and Chien-Ju Ho
Plenary Room
1:30 pm - 2:30 pm EST Works-in-Progress and Demonstrations (Group A) Poster Room
2:30 pm - 3:00 pm EST Break Lobby
3:00 pm - 4:00 pm EST Keynote 2: Kumar Chellapilla (sponsored by Amazon)

Chair: Saiph Savage

 Plenary Room
4:00 pm - 5:00 pm EST Social Event and Best Paper Award (sponsored by Amazon) Seminar Room

Tuesday Nov 16, 2021

11:00 am - 12:00 pm EST Keynote 3: Tanu Mitra

Chair: Lydia Manikonda

 Plenary Room
12:00 pm - 12:30 pm EST Break Lobby
12:30 pm - 1:30 pm EST

Paper Session 2: Collaborating with and Explaining AI

Chair: Joel Chan

StreamCollab: A Streaming Crowd-AI Collaborative System to Smart Urban Infrastructure Monitoring in Social Sensing Yang Zhang, Lanyu Shang, Ruohan Zong, Zeng Wang, Ziyi Kou and Dong Wang
Pterodactyl: Two-step Redaction Of Images For Robust Face Deidentification Abdullah Alshaibani and Alexander Quinn
Explaining Autonomous Decisions in Swarms of Human-on-the-Loop Small Unmanned Aerial Systems Ankit Agrawal and Jane Cleland-Huang
From Human Explanation to Model Interpretability: A Framework Based on Weight of Evidence David Alvarez Melis, Harmanpreet Kaur, Hal Daume, Hanna Wallach and Jennifer Wortman Vaughan
Plenary Room
1:30 pm - 2:30 pm EST Works-in-Progress and Demonstrations (Group B) Poster Room
2:30 pm - 3:00 pm EST Break Lobby
3:00 pm - 4:00 pm EST

Industry Panel: The Role of Human Computation in a Changing Technology Landscape

Moderator: Ece Kamar

Panelists: Olga Megorskaya (Toloka), Kumar Chellapilla (Amazon), Markus Krause (Brainworks), Besmira Nushi (Microsoft), Chris Welty (Alphabet)

 Plenary Room
4:00 pm - 5:00 pm EST Town Hall

Chairs: Ece Kamar and Kurt Luther 		Plenary Room

Wednesday Nov 17, 2021

11:00 am - 12:00 pm EST Keynote 4: Jaime Teevan

Chair: Alex Williams

 Plenary Room
12:00 pm - 12:30 pm EST Break Lobby
12:30 pm - 1:30 pm EST

Paper Session 3: Learning from Crowds

Chair: Ece Kamar

Rapid Instance-level Knowledge Acquisition from Class-level Common Sense
Honorable Mention Paper 		Chris Welty, Lora Aroyo, Flip Korn, Sara Marie Mc Carthy and Shubin Zhao
Modeling Simultaneous Preferences for Age, Gender, Ethnic, and Professional Profiles in Government-expense Spending: A Conjoint Analysis Lujain Ibrahim, Mohammad M. Ghassemi and Tuka Alhanai
Iterative Human-in-the-Loop Discovery of Unknown Unknowns in Image Datasets Lei Han, Xiao Dong and Gianluca Demartini
Utility of Crowdsourced User Experiments for Measuring the Central Tendency of User Performance to Evaluate Error-rate Models on GUIs Shota Yamanaka
Plenary Room
1:30 pm - 2:30 pm EST Works-in-Progress and Demonstrations (Group A) Poster Room
2:30 pm - 3:00 pm EST Break Lobby
3:00 pm - 4:00 pm EST

Blue Sky Ideas

Chair: Joel Chan

Human in the Loop for Machine Creativity
3rd Place Blue Sky Idea 		Neo Christopher Chung
The Science of Rejection: A Research Area for Human Computation
1st Place Blue Sky Idea 		Burcu Sayin, Jie Yang, Andrea Passerini and Fabio Casati
Human Computation and Crowdsourcing for Earth
2nd Place Blue Sky Idea 		Yasaman Rohanifar, Syed Ishtiaque Ahmed, Sharifa Sultana, Prateek Chanda and Malay Bhattacharyya
Plenary Room
4:00 pm - 5:00 pm EST

Social Event & Best Work-in-Progress/Demo Award (sponsored by Toloka)

FindItOut: A Multiplayer GWAP for Collecting Plural Knowledge
Toloka Best Poster/Demo 		Agathe Balayn, Gaole He, Andrea Hu, Jie Yang and Ujwal Gadiraju
Seminar Room and Lobby

Thursday Nov 18, 2021

11:00 am - 12:00 pm EST Keynote 5: Juho Kim

Chair: Shayan Doroudi

 Plenary Room
12:00 pm - 12:30 pm EST Break Lobby
12:30 pm - 1:30 pm EST

Paper Session 4: Improving Crowd Workflows and Aggregation

Chair: Kurt Luther

Making Time Fly: Using Fillers to Improve Perceived Latency in Crowd-Powered Conversational Systems Tahir Abbas, Ujwal Gadiraju, Vassilis-Javed Khan and Panos Markopoulos
Human+AI Crowd Task Assignment Considering Result Quality Requirements Masaki Kobayashi, Kei Wakabayashi and Atsuyuki Morishima
Enhancing Image Classification Capabilities of Crowdsourcing-based Methods through Expanded Input Elicitation Romena Yasmin, Joshua Grassel, Md Mahmudulla Hassan, Adolfo Escobedo and Olac Fuentes
Forecast Aggregation via Peer Prediction Juntao Wang, Yang Liu and Yiling Chen
Plenary Room
1:30 pm - 2:30 pm EST Works-in-Progress and Demonstrations (Group B) Poster Room
2:30 pm - 3:00 pm EST Break Lobby
3:00 pm - 4:00 pm EST Keynote 6: Olga Megorskaya (sponsored by Toloka)

Chair: Chris Welty

 Plenary Room
4:00 pm - 4:30 pm EST HCOMP Closing

Chairs: Ece Kamar and Kurt Luther 		Plenary Room

  • Stay CONNECTED: HCOMP COMMUNITY

We welcome everyone who is interested in crowdsourcing and human computation to:

  • Join crowd-hcomp Google Group (mailing list) to post and receive crowdsourcing and human computation email announcements (e.g., calls-for-papers, job openings, etc.) including updates here about the conference. To subscribe send an email to crowd-hcomp+subscribe@googlegroups.com.
  • Check our Google Group webpage to view the archive of past communications on the HCOMP mailing list.
  • Keep track of our twitter hashtag #HCOMP2024.
  • Join the HCOMP Slack Community to be in touch with researchers, industry players, practitioners, and crowd workers around Human Computation and relevant topics.