• Tracks

Accepted papers are grouped by track, with a link to each track group provided below.

• Proceedings
• Papers
• Graduate Consortium
• Works-in-Progress and Demonstrations
• Blue Sky Ideas Papers

Proceedings

View Conference Proceedings

Papers

A Checklist to Combat Cognitive Biases in Crowdsourcing (video)

• Tim Draws, Alisa Rieger, Oana Inel, Ujwal Gadiraju and Nava Tintarev

Enhancing Image Classification Capabilities of Crowdsourcing-based Methods through Expanded Input Elicitation (video)

• Romena Yasmin, Joshua Grassel, Md Mahmudulla Hassan, Adolfo Escobedo and Olac Fuentes

Explaining Autonomous Decisions in Swarms of Human-on-the-Loop Small Unmanned Aerial Systems (video)

• Ankit Agrawal and Jane Cleland-Huang

Exploring the Music Perception Skills of Crowd Workers (video)

• Ioannis Petros Samiotis, Sihang Qiu, Christoph Lofi, Jie Yang, Ujwal Gadiraju and Alessandro Bozzon

Forecast Aggregation via Peer Prediction (video)

• Juntao Wang, Yang Liu and Yiling Chen

From Human Explanation to Model Interpretability: A Framework Based on Weight of Evidence (video)

• David Alvarez Melis, Harmanpreet Kaur, Hal Daume, Hanna Wallach and Jennifer Wortman Vaughan

Human+AI Crowd Task Assignment Considering Result Quality Requirements (video)

• Masaki Kobayashi, Kei Wakabayashi and Atsuyuki Morishima

Iterative Human-in-the-Loop Discovery of Unknown Unknowns in Image Datasets (video)

• Lei Han, Xiao Dong and Gianluca Demartini

Iterative Quality Control Strategies for Expert Medical Image Labeling

• Beverly Freeman, Naama Hammel, Sonia Phene, Abigail Huang, Rebecca Ackermann, Olga Kanzheleva, Miles Hutson, Caitlin Taggart, Quang Duong and Rory Sayres

Making Time Fly: Using Fillers to Improve Perceived Latency in Crowd-Powered Conversational Systems (video)

• Tahir Abbas, Ujwal Gadiraju, Vassilis-Javed Khan and Panos Markopoulos

Modeling Simultaneous Preferences for Age, Gender, Ethnic, and Professional Profiles in Government-expense Spending: A Conjoint Analysis (video)

• Lujain Ibrahim, Mohammad M. Ghassemi and Tuka Alhanai

On the Bayesian Rational Assumption in Information Design (video)

• Wei Tang and Chien-Ju Ho

Pterodactyl: Two-step Redaction Of Images For Robust Face Deidentification (video)

• Abdullah Alshaibani and Alexander Quinn

Rapid Instance-level Knowledge Acquisition from Class-level Common Sense (video)

• Chris Welty, Lora Aroyo, Flip Korn, Sara Marie Mc Carthy and Shubin Zhao

StreamCollab: A Streaming Crowd-AI Collaborative System to Smart Urban Infrastructure Monitoring in Social Sensing (video)

• Yang Zhang, Lanyu Shang, Ruohan Zong, Zeng Wang, Ziyi Kou and Dong Wang

Utility of Crowdsourced User Experiments for Measuring the Central Tendency of User Performance to Evaluate Error-rate Models on GUIs (video)

• Shota Yamanaka

Graduate Consortium

Designing Human-AI Collaborative Systems for Historical Photo Identification (video)

• Vikram Mohanty

Disentangling Inherent Ambiguity and Disagreement in Crowdsourced Annotation (video)

• Quanze Chen

Empowering Workers with Artificial Intelligence Tools (video)

• Carlos Toxtli-Hernandez

Human-UAV Collaborations in the Human-On-the-Loop Emergency Response Systems (video)

• Ankit Agrawal

Incorporating Human In The Loop For Voice Assistants (video)

• Shih-Hong Huang

Investigating Impacts of the Intra-Rural Digital Divide on Rural and Super-Rural Crowd Workers (video)

• Rebecca Jonas

Supporting High-Stakes Investigations with Expert-Led Crowdsourcing

• Sukrit Venkatagiri

Unlocking the Potential of the Crowd by Challenging its Assumptions (video)

• Amy Rechkemmer

What is the Next Dataset Creation Pipeline? (video)

• Yuan-Hong (Andrew) Liao

Works-in-Progress and Demonstrations

Group A

A General-Purpose Crowdsourcing Computational Quality Control Toolkit for Python (video)

• Dmitry Ustalov, Nikita Pavlichenko, Vladimir Losev, Iulian Giliazev and Evgeny Tulin

A Human Computation Approach for Ontology Restrictions Verification (video)

• Stefani Tsaneva and Marta Sabou

A Paradigm Shift towards Crowd-based Healthcare System (video)

• Sujoy Chatterjee, Thipendra P Singh, Amitava Choudhury and Anirban Mukhopadhyay

Crowdsourcing Diverse Paraphrases for Training Task-oriented Bots (video)

• Jorge Ramírez, Auday Berro, Marcos Baez, Boualem Benatallah and Fabio Casati

Crowdsourcing Exploratory Cues for Idea Browsing and Inspiration Discovery (video)

• Xiaotong Tone Xu, Khuyen Le, Christopher Cha, Samuel Fleet and Steven Dow

Crowdsourcing Human Oversight on Image Tagging Algorithms: An initial study of image diversity (video)

• Kyriakos Kyriakou, Pınar Barlas, Styliani Kleanthous, Evgenia Christoforou and Jahna Otterbacher

Double-Checking History: Designing Assessable Systems for Historical Photo Identification (video)

• Vikram Mohanty and Kurt Luther

Exploring the Role of Domain Experts in Characterizing and Mitigating Machine Learning Errors (video)

• Pavel Hoogland, Oana Inel, Jie Yang, Margje Schuur, Jasper van Vliet and Piet van Hoffen

FindItOut: A Multiplayer GWAP for Collecting Plural Knowledge (video)

• Agathe Balayn, Gaole He, Andrea Hu, Jie Yang and Ujwal Gadiraju

Morphological Matrices as a Tool for Crowdsourced Ideation (video)

• Jonas Oppenlaender

Group B

HAEM: Obtaining Higher-Quality Classification Task Results with AI Co-Workers (video)

• Yu Yamashita, Masaki Kobayashi, Kei Wakabayashi, Hiroyoshi Ito and Atsuyuki Morishima

Hate2Explain: Crowdsourced Explanations as a Cultural Bridge in Understanding Hateful Memes (video)

• Chao-Chun Han, Yi-Ching Huang, Nanyi Bi and Jane Hsu

How Much Do Platform Workers Value Reviews? An Experimental Method (video)

• David Holtz, Liane Scult and Siddharth Suri

Rsourcer: Scaling Feedback on Research Drafts (video)

• Yuchao Jiang, Boualem Benatallah and Marcos Báez

Supporting Dynamic Construction of Datasets with Annotator Suggestions (video)

• Jeongeon Park, Eunyeong Ko, Donghoon Han, Jinyeong Yim and Juho Kim

The Role of Anthropomorphic Visual Cues in Human Interactions with Conversational Agents (video)

• Emilija Zlatkutė, Sihang Qiu, Jie Yang and Ujwal Gadiraju

Towards a Requester-centered Study on the Use of ‘Bots’ for Completing Tasks (video)

• Jonas Oppenlaender and Simo Hosio

When Content Decides Where You Belong: Investigating Micro Communities on VSCO (video)

• Vikram Mohanty and Geneviève Patterson

iClarify - A Tool to Help Requesters Iteratively Improve Task Descriptions in Crowdsourcing (video)

• Zahra Nouri, Nikhil Prakash, Ujwal Gadiraju and Henning Wachsmuth

Blue Sky Ideas Papers

Human Computation and Crowdsourcing for Earth (video)

• Yasaman Rohanifar, Syed Ishtiaque Ahmed, Sharifa Sultana, Prateek Chanda and Malay Bhattacharyya

Human in the Loop for Machine Creativity (video)

• Neo Christopher Chung

The Science of Rejection: A Research Area for Human Computation (video)

• Burcu Sayin, Jie Yang, Andrea Passerini and Fabio Casati