Workshops

W1: Data Excellence

Date and Time October 26, 2020 2:00pm - 6:20pm CET

Organizers: Praveen Paritosh, Google; Matt Lease, UT Austin and Amazon; Mike Schaekermann, Amazon and University of Waterloo; Lora Aroyo, Google

Overview: Human annotated data is crucial for operationalizing empirical ways for evaluating, comparing, and assessing the progress of ML/AI research. As human annotated data represents the compass that the entire ML/AI community relies on, the human computation (HCOMP) research community has a multiplicative effect on the progress of the field. Optimizing the cost, size, and speed of collecting data has attracted significant attention by HCOMP and related research communities. In the first to market rush with data, aspects of maintainability, reliability, validity, and fidelity of datasets are often overlooked. We want to turn this way of thinking on its head and highlight examples, case-studies, methodologies for excellence in data collection. Data excellence happens organically due to appropriate support, expertise, diligence, commitment, pride, community, etc. We will invite speakers and submissions exploring such case studies in data excellence, focusing on empirical and theoretical methodologies for reliability, validity, maintainability, and fidelity of data. Goals of the workshop:

• Gather case studies of data excellence;
• Help define data excellence;
• Build a catalog of best practices for data excellence;
• Empirical and theoretical methodologies for reliability, validity, maintainability, fidelity of data;
• How do we invest more, not less, in data? How do we justify that investment?

W2: Rigorous Evaluation of AI Systems

Date and Time October 25, 2020 2:00pm - 6:20pm CET

Organizers: Bernease Herman, Sarah Luger, Maria Stone, Kurt Bollacker

Overview: Human annotated datasets have emerged as the primary mechanism to operationalize empirical ways for evaluating, comparing, and assessing progress in machine learning, AI, and related fields. Crowdsourcing has helped solve the issue of scale in human annotated datasets while additionally has increased the impact of the variability of humans as instruments to provide these annotations. In recent research and applications where these evaluations rely on human annotated datasets or methodologies, we are interested in the meta-questions around characterization of those methodologies. Some of the expected activities in the workshop include:

• Invited and contributed presentations from the evaluation, crowdsourcing, HCI, and AI communities;
• Group discussion on human-centered evaluation metrics as well as similarities and differences from dataset annotation;
• A focus on metascience, which aims to overtly, iteratively make science better through a reflective investment on evaluation, measurement, replicability, and contribution, thus attempting to build a substantial foundation of science in order to make long term fundamental improvements to the scientific method;
• Collecting, examining and sharing current evaluation and replication efforts, comprehensive of one system or competitive of multiple systems with the goal of critically evaluating the evaluations themselves;
• Developing an open repository of existing evaluations with relevant methodology fully documented and raw data and outcomes available for public scrutiny.