HCOMP is the home of the human computation and crowdsourcing community. It’s the premier venue for presenting latest findings from research and practice into frameworks, methods and systems that bring together people and machine intelligence to achieve better results.
While artificial intelligence (AI) and human-computer interaction (HCI) represent traditional mainstays of the conference, HCOMP believes strongly in fostering and promoting broad, interdisciplinary research. Our field is particularly unique in the diversity of disciplines it draws upon and contributes to, including human-centered qualitative studies and HCI design, social computing, artificial intelligence, economics, computational social science, digital humanities, policy, and ethics. We promote the exchange of advances in human computation and crowdsourcing not only among researchers, but also engineers and practitioners, to encourage dialogue across disciplines and communities of practice.
Submissions may hence cover theory, studies, tools and applications that present novel, interesting, impactful interactions between people and computational systems. These cover a broad range of scenarios, from classical human computation, wisdom of the crowds, and all forms of crowdsourcing to people-centric AI methods, systems and applications.
Topics of interest include:
Authors are invited to submit papers of up to 8 pages, plus any number of additional pages containing references only. Note the page limits for the proceedings below.
All submitted papers must represent original work, not previously published or under simultaneous peer-review for any other peer-reviewed, archival conference or journal.
Papers must be formatted in AAAI two-column, camera-ready style; please refer to the AAAI 2020 Author Kit for details. The AAAI copyright block is not required on submissions, but must be included on final accepted versions.
Electronic abstract and paper submission through the HCOMP-20 EasyChair paper submission site is required on or before the deadlines listed above. Authors will receive confirmation of receipt of their abstracts or papers, including an ID number, shortly after submission. You can make updates to all information submitted before the deadline. We will consider only the final version received before the call closes.
All papers must be anonymized (include no information identifying the authors or their institutions) for double-blind peer-review. To ensure fairness, authors should declare any conflicts-of-interest with PC members by selecting the “Declare Conflicts” link on the upper-right of your EasyChair submission page.
Authors are strongly encouraged to include supplemental materials such as executables and data files so that reviewers can reproduce results in the paper, images, additional videos, related papers, more detailed explanations, derivations, or results. These materials will be viewed only at the discretion of the reviewers, who are only obligated to read your paper itself.
Each paper will be reviewed by at least two members of the programme committee and one AC. Reviewers will be instructed to evaluate paper submissions according to specific review criteria. We encourage authors to review them before submission.
To ensure relevance, authors should consider including research questions and contributions of broad interest to crowdsourcing and human computation, as well as discuss relevant open problems and prior work in the field. When evaluation is conducted entirely within a specific domain, authors are encouraged to discuss how findings might generalize to other communities and application areas using crowdsourcing and human computation.
Accepted papers will be listed on the conference website.
To be included in the proceedings and in the conference programme at least one author must register for the main conference. The registration needs to occur by the camera-ready deadline.
Accepted full papers will be allocated eight (8) pages in the conference proceedings; up to two (2) additional pages may be used at a cost to the authors of $275 per page. Final papers found to exceed page limits and or otherwise violating the instructions to authors will not be included in the proceedings. Authors will be required to transfer copyright of their paper to AAAI. Accepted full papers will be published in the HCOMP conference proceedings and included in the AAAI Digital Library.
If your paper is accepted, you will be invited to present it at HCOMP2020.
As noted earlier, at least one author of each accepted paper must register for the main conference to present the work or acceptance will be withdrawn. The deadline for that is the same as the camera-ready deadline.
HCOMP 2020 will recognize a best-paper and two runner ups. New to HCOMP is a best student-paper.
A best student paper is a paper where major parts of the work have been completed by an undergrad or postgrad student.
Reviewers will be asked to flag papers they deem worthy of a prize. The general chairs will set up a small panel who will read the papers, consider the comments of the reviewers and assess the talk to determine the winners.
HCOMP workshops are a gathering place for people interested in human computation or crowdsourcing to meet in the context of a focused and interactive discussion. They are an opportunity to move the field forward and build community. Workshops might address basic or applied research, human computation or crowdsourcing in industry, new methodologies, emerging application areas, or other topics of interest. Workshop schedules should encourage lively debates and discussion. Each workshop should generate ideas that will give the HCOMP community a fresh way of thinking about the topic or that suggest promising directions for future work.
If you would like to organize a workshop, please reach out to hcompconference@gmail.com by June 26 with a description of your proposed topic, intended audience, list of potential speakers, and a one-day or half-day schedule, including some time for contributed talks. Workshops are expected to be inclusive, and priority will be given to workshops that include a diversity of speakers, organizers, and viewpoints. Organizers also have the possibility to publish an overview article about their workshop (up to 10 pages, including abstracts of selected articles) to AI Magazine.
Questions? Please contact the Workshop Chairs.
Alessandro Bozzon, at A.Bozzon@tudelft.nl
Anca Dumitrache, at anca.dumitrache89@gmail.com
The Works-in-Progress and Demonstration track focuses on recent findings or other types of innovative or thought-provoking work, hands-on demonstration, novel interactive technologies and experiences relevant to the HCOMP community.
We encourage practitioners and researchers to submit Works-in-Progress & Demo Track as it provides a unique opportunity for sharing valuable ideas, eliciting useful feedback on early-stage work, and fostering discussions and collaborations among colleagues. To maximise visibility this year accepted Works-in-Progress and Demonstration submissions will also be presented with a 1-2 min PechaKucha presentation dedicated to the Works-in-Progress & Demonstration Track within the HCOMP plenary programme.
Submissions are welcome from any area of computer science, artificial intelligence, and human-computer interaction, to economics and the social sciences, all the way to digital humanities, policy, and ethics. This year, we especially encourage work that generate new insights into the “human computation” side of HCOMP, such as new understandings about human cognition, human-in-the-loop intelligence systems, human-AI interaction and collaboration, algorithmic and interface techniques for augmenting human abilities to perform tasks, and other issues that affect how humans collaborate with AI systems (such as bias, fairness, and interpretability).
A Work-in-Progress is a concise report of recent findings or other types of innovative or thought-provoking work relevant to the HCOMP community. The difference between Works-in-Progress and other contribution types is that Work-in-Progress submissions represent work that has not reached a level of completion that would warrant the full Refereed selection process. That said, appropriate submissions should make some contribution to the body of HCOMP knowledge, whether realized or promised. A significant benefit of a Work-in-Progress derives from the discussion between the author and conference attendees that will be fostered by the synchronous virtual presentation of the work. Work-in-Progress submissions are in the form of a ~2 page extended abstract, see Submission section.
A demonstration is a high-visibility, high-impact forum of the HCOMP program that allows you to present your hands-on demonstration, share novel interactive technologies, and stage interactive experiences. Demonstrations will showcase this year’s most exciting human computation and collaborative human-AI prototypes and systems. If you have an interesting prototype, system, exhibit or installation, we want to know about it. Sharing hands-on experiences of your work is often the best way to communicate what you have created. The demonstration submission should describe the nature of the system as well as the expected form of interaction with the user and the audience. We advise preparing a demo video to address connectivity issues during the virtual demo session. Demonstrations submissions are in the form of a ~2 page extended abstract, see Submission section.
All submitted Works-in-Progress and Demos must represent original work, not previously published or under simultaneous peer-review for any other peer-reviewed, archival conference or journal.
Submissions must be done in the form of up to 2 pages Extended Abstract (references can extend beyond the 2 pages) via the HCOMP-20 EasyChair Website. Submissions to Easychair is required on or before the deadline listed above. We cannot accept submissions by e-mail or fax. Authors will receive confirmation of receipt of their submissions, including an ID number, shortly after submission. HCOMP will contact authors again only if problems are encountered with papers.
Extended Abstracts must be formatted in AAAI two-column, camera-ready style; please refer to the AAAI 2020 Author Kit for details. Papers must be in trouble-free, high-resolution PDF format, formatted for US Letter (8.5″ x 11″) paper, using Type 1 or TrueType fonts. The AAAI copyright block is not required for works-in-progress or demo submissions, as they are not included in the formal proceedings. Please see below for information about publication.
Authors are invited, but not required, to include supplemental materials such as executables and data files so that reviewers can reproduce results in the paper, images, additional videos, related papers, more detailed explanations, derivations, or results. These materials will be viewed only at the discretion of the reviewers, who are only obligated to read your paper itself.
Authors should include information identifying themselves and their institutions for single-blind review.
Submissions to the Works-in-Progress & Demonstration Track will be curated. This means that submissions will only be evaluated by the Works-in-Progress & Demonstration chairs (in coordination with other relevant conference chairs). Authors should expect to receive only very light feedback on their submissions.
Accepted papers will NOT be included in official conference proceedings, and so may be submitted later to other conferences or journals for official publication. Accepted papers will be made available online on the conference website, and a summary of the WiP and Demo session will be provided in the AAAI AI Magazine.
Accepted submissions will be presented at the conference in two main modalities: 1) with a 1-2 min PechaKucha presentation in a plenary session dedicated to the Works-in-Progress & Demonstration Track within the HCOMP plenary programme, and 2) with an dedicated streaming session in which authors can present their work in their preferred presentation format (PPT presentation, open discussion, virtual poster, live demo, video demo etc). For organisation purposes authors are invited to mention their desired presentation format(s) at the time of submission, including (but not limited to) virtual poster, interactive demo, short video, short presentation.
HCOMP 2020 will recognize one best Works-in-Progress and one best Demonstration. Announcement of the winners will be done at the conference in a virtual plenary session together with the other conference award. The Works-in-Progress and Demonstration chairs will assess the Works-in-Progress and Demonstrations submissions to determine the winners (considering both quality of paper submission and presentation at the conference).
Please contact the Works-in-progress and Demonstration Co-Chairs if you have any questions.
Anna De Liddo, at anna.deliddo@open.ac.uk
Alessandro Checco, at a.checco@sheffield.ac.uk
HCOMP’s annual Doctoral Consortium provides doctoral students with a unique opportunity to meet each other and experienced researchers in the field. Students will be mentored by a group of faculty who are leaders in the diverse specialties that make up the HCOMP field. The objectives of the Doctoral Consortium are to provide students with an opportunity:
HCOMP is unique in the diversity of disciplines it draws upon, and contributes to, ranging from computer science, artificial intelligence, and human-computer interaction, to economics and the social sciences, all the way to digital humanities, policy, and ethics. This year, we especially encourage works that generate new insights into the quality (or qualities) of human-annotated datasets, including elements such as reliability and replicability of human computation and crowdsourcing experiments, novel metrics for aggregation of results, and holistic approaches to deal with bias, fairness, and interpretability.
Applicants must be currently enrolled in a full-time PhD program and have written, or be close to completing, a thesis proposal (or equivalent). We will give preference to students who have proposed or are about to propose but are far enough from completing their thesis that the feedback they receive at the event can impact their work. Before submitting, students should discuss this criterion with their advisor or supervisor.
Those accepted are required to present their work at the Doctoral Consortium. Participants will also likely be required to present a poster on their work during the poster session at the main conference.
Submissions will be reviewed by a Program Committee of Doctoral Committee Mentors, with selection based upon the expected potential of both the student and their proposed work, as well as the expected benefit to the student from participation. Priority will be given to students whose research goes beyond locally available expertise at their home institutions.
More information about financial support coming soon.
Applicants must submit a solely-authored paper in English containing: 1) a Doctoral Research Overview; and 2) a Supplemental Paragraph.
Please summarize your doctoral research, including the following sections:
Please write a paragraph explaining:
Your paper should be no more than 4 pages in total: 3 pages for the Doctoral Research Overview (including all figures and references), and the 4th page being the Supplemental Paragraph. The first page must contain the title of the paper, full author name, affiliation and contact details, an abstract of up to 250 words, and up to 3 keywords describing the research topic areas.
Papers must be formatted in AAAI two-column, camera-ready style; please refer to the AAAI 2019 Author Kit for details. Papers must be in trouble-free, high-resolution PDF format, formatted for US Letter (8.5” x 11”) paper, using Type 1 or TrueType fonts. It is the responsibility of the student to ensure that their submission uses no unusual formatting and is printable on a standard printer. The AAAI copyright block is not required for works-in-progress or demo submissions, as they are not included in the formal proceedings. Please see below for information regarding dissemination.
Electronic submission through the HCOMP-20 EasyChair (https://easychair.org/conferences/?conf=hcomp2020) paper submission site is required on or before the deadline listed above. We cannot accept submissions by e-mail or fax. Authors will receive confirmation of receipt of their submissions, including an ID number, shortly after submission. HCOMP will contact authors again only if problems are encountered with papers.
Submissions will be distributed only to mentors and other attendees of the doctoral consortium. Proceedings of the Doctoral Consortium will NOT be archived. As such, students may freely submit their research contributions for official publication in other venues. Participant names and university affiliations, as well as paper titles and abstracts, will be publicized on the conference website and in the conference program for the poster session.
Please contact the Doctoral Consortium Chairs.
Maribel Acosta, at maribel.acosta@kit.edu
Markus Krause, at public.markus.krause@gmail.com
In cooperation with the Computing Community Consortium (CCC), HCOMP 2020 invites submissions to a special “Blue Sky Ideas” track. The emphasis of this track will be on visionary ideas, long term challenges, and opportunities in research that are outside of the current mainstream topics of the field.
Submissions will be judged on the extent to which they expand the possibilities and horizons of the field or challenge existing assumptions prevalent in the field.
Submissions to this track should follow the same formatting guidelines as submissions to the research track, but are limited to at most four pages in length for both submission and final publication. Accepted papers will be published in the conference proceedings and on the web.
To encourage researchers to present truly visionary concepts, the CCC is offering prizes for up to three top papers in this special track: first prize $1000, second prize $750, and third prize $500, to be awarded.
Note that paper submissions in this track will not receive full-fledged reviews as in the other tracks at HCOMP 2020. All submissions however will be screened to ensure that they meet the required criteria of being visionary in nature. A committee of experts will adjudicate the top-3 submissions to this track and nominate them for awards sponsored by CCC.
The submissions are done electronicaly through the HCOMP-20 EasyChair paper submission site is required on or before the deadline listed above. We cannot accept submissions by e-mail or fax. Authors will receive confirmation of receipt of their submissions, including an ID number, shortly after submission. HCOMP will contact authors again only if problems are encountered with papers.
Questions? Please contact the Blue Sky Ideas track Chairs.
Ujwal Gadiraju, at u.k.gadiraju@tudelft.nl
CrowdCamp started as a one-day hackathon for researchers and practitioners with interests in crowdsourcing, human computation, social media, and collective intelligence. CrowdCamp has been going on since 2011, at first co-located with CHI, then with CSCW, and since 2014 with HCOMP. Historically, CrowdCamp’s focus for participants has been aimed at creating deliverable prototypes or study designs during the workshop. You can find details of previous editions here. Prior CrowdCamp projects have resulted in top-tier conference publications, blog posts, and on-going research. For the past several years the primary theme of CrowdCamp has been the Human-AI Collaboration.
This year, while keeping the same focus, we would like to draw attention to data - how to leverage human computation for the purpose of generating, acquiring, annotating reliable datasets for AI. Going even further, during CrowdCamp 2020 we would like to focus on projects that explore what are characteristics of reliable datasets, how can we measure and improve the overall quality as well as the quality of various aspects of such datasets, e.g. their reliability, reproducibility, fairness, etc., and finally what influences the quality of human-labelled datasets in terms of the collection, labeling, aggregation, sampling, etc practices. Thus, we are considering transforming CrowdCamp into an online data challenge, where participants will work on their innovative solutions prior to the CrowdCamp event, and during the event we will focus on presentations and discussion of these solutions.
All participants are encouraged to embrace the mutual interests of HCI and AI research within their work on the CrowdCamp projects, touching on different methodologies for the same area of research. CrowdCamp will also be leveraged as an opportunity to better understand the challenges and opportunities for research collaborations at the intersection of artificial intelligence and human-computer interaction.
We invite students, faculty, industry researchers, and anyone else who is interested to participate in CrowdCamp. We are looking for people with different expertise, ranging from social scientists to programmers, ethnographers to designers, and anyone else interested in the broader topic of Human-AI Collaboration.
In order to prepare an event that has a good coverage of interests and topics across the HCOMP community, we would like you to fill this application form with your expression of interest to participate. The application takes about 5 minutes to complete, and asks for:
Once we have collected all the ideas for research topics and datasets, we will ask you to vote on them so that we can form teams prior to the day of the workshop. The CrowdCamp day will focus on working on a research project (whether that’s a study design, an idea for a new algorithm, a system architecture, or something else entirely) within your interdisciplinary HCI-AI teams. Each team will also have the opportunity to get feedback on their research project from a group of experts (in both HCI and AI methodologies). Our goal is to foster interdisciplinary collaborations and enable this for your particular domain of interest and research ideas.
We welcome everyone who is interested in crowdsourcing and human computation to: