Program

  • Virtual Co-host

HCOMP2020 will be virtually co-hosted by the Dutch institute for media culture Netherlands Institute for Sound and Vision. Read the full announcement here.

  • Keynotes

Find detailed information about the keynotes.

  • Accepted Papers

View a list of accepted papers.

View the conference proceedings.

  • Work in Progress and Demonstrations

View a list of accepted papers.

  • Blue Sky Ideas Track

View a list of accepted papers.

  • Workshops

View a list of the workshops.

  • CrowdCamp

   JOIN NOW! CrowdCamp2020 introduced an exciting online data challenge - CATS4ML: Crowdsourcing Adverse Test Sets for Machine Learning. The challenge is open for participation until 30 April, 2021.

By participating in this challenge you will join the global research community in using the power of crowdsourcing to discover AI blindspots in the Open Images Dataset.

Winning contributions will be presented at CrowdCamp 2021 and invited to co-author a paper describing the research breakthroughs.


Read more about the CrowdCamp here.

  • Social Events

We will have many social events to help you have fun at HCOMP2020! Please check here for more information.

  • HCOMP Visuals

Visual summaries of talks and sessions are now available here.


  • HCOMP-20 Program

   The program is produced on underline.io. To attend you need first to register to HCOMP and you will receive instructions on how to join each session.

   All times are CET (Central European Time). You can check what that means where you live by clicking here.


Sunday, Oct 25, 2020

2:00 pm - 6:20 pm CET Workshop 2: Rigorous Evaluation of AI Systems (REAIS 2020)

Monday, Oct 26, 2020

2:00 pm - 6:20 pm CET Workshop 1: Data Excellence Workshop (DEW 2020)
5:00 pm - 6:20 pm CET

Posters and Demos Session 1: Short Presentations

Chairs: Anna De Liddo and Alessandro Checco

Demos

OpenTag: Understanding Human Perceptions of Image Tagging Algorithms
Best Demo 		Kyriakos Kyriakou, Pınar Barlas, Styliani Kleanthous and Jahna Otterbacher

Work in Progress

Exploring Effectiveness of Inter-Microtask Qualification Tests in Crowdsourcing Masaya Morinaga, Susumu Saito, Teppei Nakano, Tetsunori Kobayashi and Tetsuji Ogawa
Crowdwork as a Snapshot in Time: Image Annotation Tasks during a Pandemic Evgenia Christoforou, Pinar Barlas and Jahna Otterbacher
Assessing Political Bias using Crowdsourced Pairwise Comparisons
Best WiP 		Tzu-Sheng Kuo, Mcardle Hankin, Miranda Li, Andrew Ying and Cathy Wang
Dynamic Worker-Task Assignment for High-Quality Task Results with ML Workers Yu Yamashita, Masaki Kobayashi, Kei Wakabayashi and Atsuyuki Morishima
A Survey of Visually Impaired Workers in Japanese and US Crowdsourcing Platforms Ying Zhong, Makoto Kobayashi, Masaki Matsubara and Atsuyuki Morishima
A TOPSIS-based Multi-objective Model for Constrained Crowd Judgment Analysis Sujoy Chatterjee and Sunghoon Lim
Socially Augmented Crowdsourced Collection of Folk Theories Jonas Oppenlaender
Turkish Judge: A Peer Evaluation Framework for Crowd Work Appeals Mukund Venkateswaran, Edward Cohen, Nivedita Sankar and Chris Callison-Burch
Qualification Labour: A Fair Wage Isn’t Enough if Workers Need to Do 5,000 Low Paid Tasks to Qualify for Your Task Jonathan Kummerfeld
Towards a Crowdsourcing Platform for Low Resource Languages -- A Collectivist Approach Sarah Luger, Christopher M. Homan and Allashera Tapo
6:30 pm - 7:30 pm CET HCOMP Opening
7:40 pm - 8:30 pm CET Keynote 1: Mounia Lalmas (Spotify)

Tuesday Oct 27, 2020

3:00 pm - 4:00 pm CET

Session 1: Socio-Technical Aspects

Chair: Ujwal Gadiraju

Full Papers

Privacy-Preserving Face Redaction using Crowdsourcing Abdullah Alshaibani, Sylvia Carrell, Li-Hsin Tseng, Jungmin Shin and Alexander Quinns
Fast, Accurate, and Healthier: Interactive Blurring Helps Moderators Reduce Exposure to Harmful Content Anubrata Das, Brandon Uyvu Dang and Matthew Lease
CrowDEA: Multi-view Idea Prioritization with Crowds Yukino Baba, Jiyi Li and Hisashi Kashima

Short Papers

Does Exposure to Diverse Perspectives Mitigate Biases in Crowdwork? An Explorative Study Xiaoni Duan, Chien-Ju Ho and Ming Yin
Modeling Annotator Perspective and Polarized Opinions to Improve Hate Speech Detection Sohail Akhtar, Valerio Basile and Viviana Patti
Schema and Metadata Guide the Collective Generation of Relevant and Diverse Work Xiaotong Tone Xu, Judith Fan and Steven Dow
3:00 pm - 4:00 pm CET Doctoral Consortium Session 1

Chairs: Maribel Acosta and Markus Krause
4:10 pm - 4:55 pm CET

Session 2: Socio-Technical Aspects

Chair: David Lee

Full Papers

Soliciting Human-in-the-Loop User Feedback for Interactive Machine Learning Reduces User Trust and Impressions of Model Accuracy Donald Honeycutt, Mahsan Nourani and Eric Ragan
The Role of Domain Expertise in User Trust and the Impact of First Impressions with Intelligent Systems Mahsan Nourani, Joanie T. King and Eric Ragan
Impact of Algorithmic Decision Making on Human Behavior: Evidence from Ultimatum Bargaining
Best Student Paper 		Alexander Erlei, Franck Awounang Nekdem, Lukas Meub, Avishek Anand and Ujwal Gadiraju
4:10 pm - 4:55 pm CET Doctoral Consortium Session 2

Chairs: Maribel Acosta and Markus Krause
5:00 pm - 5:50 pm CET Keynote 2: Anna Ridler (Artist)
6:00 pm - 6:50 pm CET

Session 3: Human Computation Applications

Chair: Anna De Liddo

Full Papers

Effective Operator Summaries Extraction Ido Nimni and David Sarne
Predicting Crowdworkers' Performance as Human-Sensors for Robot Navigation Nir Machlev and David Sarne
Understanding the Effects of Explanation Types and User Motivations on Movie Recommender System Use Qing Li, Sharon Lynn Chu, Nanjie Rao and Mahsan Nourani

Short Papers

The Challenges of Crowd Workers in Rural and Urban America Claudia Flores-Saviaga, Yuwen Li, Benjamin Hanrahan, Jeffrey Bigham and Saiph Savage
6:00 pm - 6:50 pm CET Doctoral Consortium Session 3

Chairs: Maribel Acosta and Markus Krause
7:00 pm - 8:00 pm CET HCOMP Social and Awards

Wednesday Oct 28, 2020

3:00 pm - 4:00 pm CET

Session 4: Methods and Platforms

Chair: Marta Sabou

Full Papers

Trainbot: A Conversational Interface to Train Crowd Workers for Delivering On-Demand Therapy Tahir Abbas, Vassilis-Javed Khan, Ujwal Gadiraju and Panos Markopoulos
Motivating Novice Crowd Workers through Goal Setting: An Investigation into the Effects on Complex Crowdsourcing Task Training
Best Paper 		Amy Rechkemmer and Ming Yin
Enhancing Collective Estimates by Aggregating Cardinal and Ordinal Inputs Ryan Kemmer, Yeawon Yoo, Adolfo Escobedo and Ross Maciejewski

Short Papers

How Useful Are the Machine-Generated Interpretations? A Human Evaluation on Guessing the Wrongly Predicted Labels Hua Shen and Ting-Hao 'Kenneth' Huang
Batch Prioritization of Data Labeling Tasks for Training Classifiers Masanari Kimura, Kei Wakabayashi and Atsuyuki Morishima
A Case for Soft Loss Functions Alexandra Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank and Massimo Poesio
3:00 pm - 5:30 pm CET CrowdCamp Session 1

Chairs: Anna Lisa Gentile and Igor Karpov
4:10 pm - 4:55 pm CET

Session 5: Methods and Platforms

Chair: Gabriella Kazai

Full Papers

Analyzing Workers Performance in Online Mapping Tasks Across Web, Mobile, and Virtual Reality Platforms Gerard van Alphen, Sihang Qiu, Alessandro Bozzon and Geert-Jan Houben
How Context Influences Cross-Device Task Acceptance in Crowd Work Danula Hettiachchi, Senuri Wijenayake, Simo Hosio, Vassilis Kostakos and Jorge Goncalves
Verifying Extended Entity Relationship Diagrams with Open Tasks Marta Sabou, Klemens Käsznar, Dietmar Winkler, Markus Zlabinger and Stefan Biffl
5:00 pm - 5:50 pm CET

Posters and Demos Session 2: Posters in gather town

Chairs: Anna De Liddo and Alessandro Checco

Same posters and demos as in session 1
6:00 pm - 6:50 pm CET Keynote 3: Chris Welty (Google Research)
7:00 pm - 8:00 pm CET Townhall Meeting

Thursday Oct 29, 2020

3:00 pm - 3:50 pm CET

Blue Sky Session

Chair: Ujwal Gadiraju

Using Human Cognitive Limitations to Enable New Systems
Best Blue Sky Idea 		Vincent Conitzer
Group-Assign: Type Theoretic Framework for Human AI Orchestration
Second Place Best Blue Sky Idea 		Aik Beng Ng, Zhangsheng Lai, Simon See and Shaowei Lin
A Technology-Assisted Social Computing Framework for Solving Complex Social Problems Kevin Kells
COGNET: The Planetary Cognition Delivery Network Simo Hosio and Niels van Berkel
3:00 pm - 3:50 pm CET

Posters and Demos Session 3: Posters in gather town

Chairs: Anna De Liddo and Alessandro Checco

Same posters and demos as in session 1
4:00 pm - 4:50 pm CET Keynote 4: Julia Noordegraaf (University of Amsterdam)
5:00 pm - 5:50 pm CET Keynote 5: Pietro Perona (Amazon)
6:00 pm - 7:00 pm CET HCOMP Closing

  • Stay CONNECTED: HCOMP COMMUNITY

We welcome everyone who is interested in crowdsourcing and human computation to:

  • Join crowd-hcomp Google Group (mailing list) to post and receive crowdsourcing and human computation email announcements (e.g., calls-for-papers, job openings, etc.) including updates here about the conference. To subscribe send an email to crowd-hcomp+subscribe@googlegroups.com.
  • Check our Google Group webpage to view the archive of past communications on the HCOMP mailing list.
  • Keep track of our twitter hashtag #HCOMP2024.
  • Join the HCOMP Slack Community to be in touch with researchers, industry players, practitioners, and crowd workers around Human Computation and relevant topics.