AAAI HCOMP is the premier venue for disseminating the latest research findings on human computation and crowdsourcing. Its focus is on research and practice into frameworks, methods, and systems that bring together people and machine intelligence to achieve better results. HCOMP 2024 will be held as an in-person conference in Pittsburgh, Pennsylvania from October 16-19, 2024.
All times are midnight AoE
Submit to HCOMP 2024 using Easychair here.
While artificial intelligence (AI) and human-computer interaction (HCI) represent traditional mainstays of the conference, HCOMP believes strongly in fostering and promoting broad, interdisciplinary research. Our field is particularly unique in the diversity of disciplines it draws upon and contributes to, including human-centered qualitative studies, HCI design, social computing, machine learning, natural language processing, the broader realms of artificial intelligence (including LLMs and generative AI), economics, computational social science, digital humanities, policy, and ethics. We promote the exchange of advances in human computation and crowdsourcing not only among researchers but also engineers and practitioners, to encourage dialogue across disciplines and communities of practice.
With the unprecedented proliferation of AI systems across all domains and the adoption of machine learning across disciplines, there is a renewed focus on how complex AI systems are built, machine learning models are trained, and relevant data pipelines are set up to ensure responsible practices throughout these lifecycles. The role of human input and intelligence is being widely discussed in the age of LLMs, and generative AI. Human input serves a multitude of important purposes in these contexts, ranging from generation of training data to validation, evaluation, and facilitating oversight. Ensuring that data work is carried out in a fair, ethical, unbiased, and responsible manner every step of the way can help create better AI systems. It is in this spirit that the AAAI HCOMP 2024 theme focuses on “Responsible Crowd Work for Better AI.”
Topics of interest include:
Submissions may, therefore, cover theory, user studies, tools, and applications that present novel, interesting, impactful interactions between people and computational systems. These cover a broad range of scenarios, from classical human computation, the wisdom of crowds, and all forms of crowdsourcing to people-centric AI methods, systems, and applications.
Authors are invited to submit anonymized full papers of variable length up to a maximum of 8 pages (including all content, figures, and tables). Additional pages may contain references only (i.e., max. 8 pages + references). Shorter and focussed submissions are welcome and reviewers will assess the contributions of the work accordingly. Papers must be formatted in AAAI two-column, camera-ready style; please refer to the AAAI 2023 Author Kit for details (available templates: AAAI 2023 Author Kit on Overleaf or AAAI 2024 Author Kit.zip [Word | LaTeX]). The AAAI copyright block is not required on submissions but must be included in the final accepted versions.
Authors are invited, but not required, to include supplemental materials such as executables and data files so that reviewers can reproduce results in the paper, images, additional videos, related papers, more detailed explanations, derivations, or results. These materials will be viewed only at the reviewers’ discretion, who are only obligated to read the submitted papers.
Accepted full papers will be published in the HCOMP conference proceedings and included in the AAAI Digital Library. AAAI enables authors to use Open Responsible AI Licenses (Open RAIL), licenses designed to permit free and open access, re-use, and downstream distribution of derivatives of AI artifacts as long as the behavioral-use restrictions always apply (including to derivative works). Need Help Deciding if an AI Pubs License is Right for You? Take a look here: https://www.licenses.ai/blog/2023/3/3/ai-pubs-rail-licenses. Please consider this as a way to share your work with the community. If you have questions, comments, or feedback, please contact the RAIL team here: https://www.licenses.ai/contact.
If your paper is accepted, you will be required to present it in person at the AAAI HCOMP 2024 conference in Pittsburgh. At least one author of each accepted paper must register for the main conference to present the work or acceptance will be withdrawn.
Anonymity.
HCOMP 2024 will adopt a double-blind review process. Submissions should omit any author names, affiliations, or other identifying information. Submissions not complying with this guidance will be desk-rejected.
Preprint servers.
We do not have a policy against uploading preprints to SSRN or arXiv before they are submitted for review at the conference. Nevertheless, to ensure the integrity of the peer review process, we ask that no authors publicize the work until that process is complete. Please do not share confidential info specific to a current review process on social media or in similarly public forums.
Conflicts of interest.
To ensure fairness, authors should declare any conflicts of interest with PC members by selecting the “Declare Conflicts” link on the upper-right of your EasyChair submission page.
Double Submission Policy.
Papers submitted to the HCOMP conference must represent original work that has not been previously published or that is not under simultaneous peer-review for any other peer-reviewed archival conference or journal. Note that:
Review Criteria.
Reviewers will be instructed to evaluate paper submissions according to specific review criteria, some of which is unique to HCOMP. Our intent in posting these review criteria online is to further improve transparency of the conference’s peer-review process and to provide additional guidance to authors in preparing their submissions (especially for young researchers, as well as researchers from diverse disciplines).
We encourage authors to review these criteria and contact us with any questions or feedback. Tweet us @hcomp_conf or email us at hcomp24@easychair.org.
Reviews.
Each paper will be reviewed by at least three members of the program committee and one SPC member. Reviewers will be instructed to evaluate submissions according to specific review criteria. We encourage authors to review them before submission.
To ensure relevance, authors should consider including research questions and contributions of broad interest to crowdsourcing and human computation, as well as discuss relevant open problems and prior work in the field. When the evaluation is conducted within a specific domain, authors are encouraged to discuss how findings might generalize to other communities and application areas using crowdsourcing and human computation.
Attendance.
To present a talk at HCOMP 2024, authors must submit a full paper for review by the program committee. Accepted papers will be granted a presentation at the conference. Therefore, submitting a full paper is a mandatory requirement for authors who wish to present their work at the conference.
To be included in the proceedings and the conference program, at least one author must register in person for the main conference. The registration needs to occur by the camera-ready deadline.
Presentation.
If your paper is accepted, we are delighted to invite you to present your work at the conference. Please note that at least one author of each accepted paper must register for the main conference to present their work in person. Failure to do so will result in the withdrawal of acceptance. Remote presentation of accepted papers is not permitted except in the case of unforeseen circumstances. The deadline for registering to present your paper is the same as the camera-ready deadline.
HCOMP 2024 will recognize one best paper, one best student paper, and up to two runner-ups. Reviewers will be asked to flag papers they deem worthy of a prize. The general chairs will set up a small committee that will read the nominated papers, consider the comments of the reviewers, and assess the presentation to determine the winners.
All times are midnight AoE
The Works-in-Progress and Demonstration track focuses on recent findings or other types of innovative or thought-provoking work, hands-on demonstration, novel methods, technologies, and experiences relevant to the HCOMP communities. We also welcome work published in other venues to encourage sharing work with HCOMP’s interdisciplinary audience. We encourage practitioners and researchers to submit to the Works-in-Progress & Demo Track as it provides a unique opportunity for sharing valuable insights and ideas, eliciting useful feedback on early-stage work, and fostering discussions and collaborations among colleagues. Submissions are welcome from multiple fields, ranging from computer science, artificial intelligence, and human-computer interaction, to economics, business, and the social sciences, all the way to digital humanities, policy, and ethics.
Accepted papers in this track will be non-archival and they will not be included in the official proceedings of the HCOMP conference. They will be made available online on the conference website. Authors of accepted papers can thus benefit from exchanging insights on their work, while maintaining the option to further develop their ideas and submit the outcome to other venues.
A Work-in-Progress is a concise report of recent findings or other types of innovative or thought-provoking work relevant to the HCOMP community. The difference between Works-in-Progress and other types of contribution is that Work-in-Progress submissions represent work that has not yet reached a level of completion that would warrant the full Refereed selection process. That said, appropriate submissions should make some contribution to the body of HCOMP knowledge, whether realized or promised. A significant benefit of a Work-in-Progress derives from the discussion between the author and the conference attendees. Work-in-Progress submissions are in the form of a 2-3 page paper; see Submission section. Authors of accepted Works-In-Progress will present their papers in-person at the conference in the form of a poster.
A demonstration is a high-visibility, high-impact forum of the HCOMP program that allows you to present your hands-on demonstration, share novel technologies and tools, and stage interactive experiences. Demonstrations will showcase this year’s most exciting human computation, collaborative human-AI, and collective intelligence prototypes and systems. If you have an interesting prototype, system, exhibit, or installation, we want to know about it. Sharing hands-on experiences of your work is often the best way to communicate what you have created. The demonstration submission should describe the nature of the system as well as the expected form of interaction with the user and the audience. Demonstration submissions are in the form of a 2-3 page paper; see Submission section.
Submissions for HCOMP Works-In-Progress and Demo papers must be done via the HCOMP 2024 EasyChair Website under the Works-in-Progress and Demonstration track. Submissions to Easychair are required on or before the deadline listed above. We cannot accept submissions by email. Authors will receive confirmation of receipt of their submissions, including an ID number, shortly after submission. HCOMP will contact authors again only if problems are encountered with their papers.
In an effort to have as inclusive a program as possible, please reach out if you have any special circumstances that we should be aware of (e.g., visa requirements that would require an early review of your submission).
Works-in-Progress and Demonstration papers must be between 2 and 3 pages of content. References can extend beyond the page limit.
HCOMP papers must be formatted in AAAI two-column, camera-ready style; please refer to the AAAI 2023 Author Kit for details (available templates: AAAI 2023 Author Kit on Overleaf or AAAI 2024 Author Kit.zip [Word | LaTeX]). Papers must be in trouble-free, high-resolution PDF format, formatted for US Letter (8.5″ x 11″) paper, using Type 1 or TrueType fonts. The AAAI copyright block is not required for works-in-progress or demo submissions, as they are not included in the formal proceedings. Please see below for information about publication.
Authors are invited, but not required, to include supplemental materials such as executables and data files so that the results of the work can be reproduced.
Authors should include information identifying themselves and their institutions for single-blind review.
Accepted papers will NOT be included in official conference proceedings, and therefore they could be submitted later to other conferences or journals for official publication. Accepted papers will be made available online on the conference website.
Please contact the Works-in-Progress and Demonstration Co-Chairs if you have any questions.
The Human Computation (HCOMP) conference Doctoral Consortium (DC) aims to provide doctoral students working in areas of interest to our community with a unique experience to engage with one another, as well as with experienced researchers in the field. Students will be mentored by senior researchers who are leaders in the diverse specialities of interest to HCOMP.
The objectives of the DC are to provide students with the opportunity to:
Participation in the HCOMP DC is a great opportunity to increase the visibility of a doctoral researcher’s work and to build connections with members of the HCOMP community.
All times are midnight AoE
HCOMP is unique in the diversity of disciplines that it draws upon and contributes to, ranging from computer science (especially artificial intelligence and human-computer interaction) to economics, organizational behavior, and the social sciences, all the way to digital humanities, policy, healthcare, digital health, and ethics. Any DC submissions fitting the topics of interest in the conference Call for Submissions are applicable.
Applicants must currently be enrolled in a PhD program. The Doctoral Consortium accepts doctoral researchers from any stage in their candidature but will prioritize candidates who have established a clear topic and research approach and have made some progress, but who are not so far along that they can no longer make changes. Before submitting, candidates should discuss their criteria with their advisor.
Those accepted are required to present their work at the DC, in-person (no virtual participation). Participants will also present a poster on their work during the poster session at the main conference.
Submissions will be reviewed based upon the expected potential of both the student and their proposed work, as well as the expected benefit to the applicant from participation. Priority will be given to applicants whose research goes beyond locally available expertise at their home institutions.
We expect to be able to support a limited number of DC participants with free registration to the conference. Should you require such financial assistance, please indicate it in your application as outlined below.
Applicants must submit a solely-authored paper in English containing: 1) a Doctoral Research Overview; and 2) a Supplemental Paragraph, as described below.
Please summarize your doctoral research, including the following sections:
Please write a paragraph explaining: Why you want to participate in the consortium at this point in your doctoral studies and how you expect to benefit from the consortium; Your expected (approximate) completion date; If you require financial assistance in participating the conference, and the justification (note: we cannot support everyone, so please consider this option if you really need it).
Your paper should be no more than 4 pages in total: 3 pages for the Doctoral Research Overview (including all figures and references), and the 4th page being the Supplemental Paragraph. The first page must contain the title of the paper, full author name, affiliation and contact details, an abstract of up to 250 words, and up to 3 keywords describing the research topic areas.
For consistency, we ask that all submissions to the HCOMP DC be formatted in the same way as described in the HCOMP 2024 Call for Papers.
For simplicity, we ask that all submissions to the HCOMP Doctoral Consortium be submitted through the same portal using the HCOMP 2024 EasyChair paper submission site. Submissions must be made on or before the deadlines listed above. We cannot accept submissions by e-mail or fax. Authors will receive confirmation of receipt of their submissions, including an ID number, shortly after submission. HCOMP will contact authors again only if problems are encountered with papers.
Submissions will be distributed only to mentors and other attendees of the doctoral consortium. Proceedings of the Doctoral Consortium will NOT be archived. As such, students may freely submit their research contributions for official publication in other venues. Participant names and university affiliations, as well as paper titles and abstracts, will be publicized on the conference website and in the conference program for the poster session.
Please direct all questions to the Doctoral Consortium Co-Chairs:
We invite proposals for workshops at HCOMP 2024. Workshops bring together communities with common research interests and agendas to discuss ongoing work and initiate new collaborations. Workshops may address research, best practices, tools, education, emerging themes and applications, and critical thinking about existing methodologies and frameworks. Topics should be within the general scope of HCOMP but may include emerging research directions that have not yet been fully explored and may even be seen as controversial. We welcome submissions from both academia and industry.
Note that workshops this year will be entirely in-person.
All times are midnight AoE
The workshop format can include discussions of contributions from attendees (e.g., posters, lightning talks) as well as providing space for lively, informal debates (e.g., breakout groups, panel discussions), and any other group activities (e.g., collaborative brainstorming exercises, competitions etc.).
We look forward to new and exciting formats that can use the workshop timeslot and format for creative and non-traditional “workshops.”
Keep everyone involved! We especially encourage workshop formats with activities that keep attendees engaged throughout the workshop. Get creative! Consider incorporating demos, feedback on innovative task designs, or short-term collaborations. What can attendees create? … or solve?
Workshops should be entirely in-person and should last a maximum of 4 hours.
Link to the submission site: https://easychair.org/conferences/?conf=hcomp24
If you would like to organize a workshop at HCOMP 2024, please submit a PDF (2-4 pages) by the submission deadline above.
Submissions must be in PDF format up to a maximum of 4 pages (including all content, figures, and tables) and may contain references in additional pages. Submissions must be formatted in AAAI two-column, camera-ready style; please refer to the AAAI 2023 Author Kit for details (available templates: AAAI 2023 Author Kit on Overleaf or AAAI 2024 Author Kit.zip [Word | LaTeX]). The AAAI copyright block is not required on submissions but must be included in the final accepted versions.
Workshops will be selected through evaluation by a small jury and the workshop chairs. Jurors will provide insight to the workshop chairs considering fit and contribution to the HCOMP community as well as relation to other submitted workshops.
Priority will be given to workshops that include a diverse group of organizers across different dimensions, including but not limited to affiliation, affiliation type (e.g., academia, industry), geographical location, and the area of research.
Organizers will be notified whether their workshop has been accepted on 15th July 2024. Due to the multifaceted criteria for creating a workshop program, reviews will not be sent to potential workshop organizers.
Organizers of accepted workshops will be asked to provide details of the workshop for publication on the conference website containing the workshop title, names and affiliations of organizers, 200-word abstract, workshop website link (if any), and any participation requirements.
A final workshop description will be published in the HCOMP conference proceedings and included in the AAAI Digital Library. This final tutorial description should use the AAAI two-column, camera-ready Word/Latex templates and must be submitted by the 25th of July 2024.
Please contact HCOMP 2024 Workshop Co-Chairs for any questions concerning workshop proposals.
We welcome tutorials focusing on the application, use, and deployment of tools and frameworks proposed by HCOMP scholarship in real-world practice and/or research settings. Tutorials should focus on tools and frameworks that have already been deployed/evaluated in actual practice or in the wild, not merely at a ‘proof-of-concept’ stage. Tutorials related to research methodologies are also welcome.
Tutorials can also consist of specialized courses with significant depth in specific established and/or emerging areas of research and practice, including the various subfields relevant to the HCOMP community.
We particularly encourage submissions by practitioners who work on the ground and are concerned with human-computation systems in socially consequential domains.
The tutorial should provide an overview of the core tool/framework/area for the general HCOMP audience, situate it in the context where it is deployed, and used, and if applicable, discuss the consequences and ramifications of the tool after deployment (if applicable).
Tutorials can last 45min or 90min.
All times are midnight AoE
Link to the submission site: https://easychair.org/conferences/?conf=hcomp24
The proposal should consist of a maximum of 2 pages not including references. Proposals should include:
Submissions must be in PDF format up to a maximum of 2 pages (including all content, figures, and tables) and may contain references in additional pages. Submissions must be formatted in AAAI two-column, camera-ready style; please refer to the AAAI 2023 Author Kit for details (available templates: AAAI 2023 Author Kit on Overleaf or AAAI 2024 Author Kit.zip [Word | LaTeX]). The AAAI copyright block is not required on submissions but must be included in the final accepted versions.
Tutorial proposals will be assessed by the Tutorial Chairs, with selections governed by quality and the need for a balanced and diverse program of interest to the HCOMP community.
A final tutorial description will be published in the HCOMP conference proceedings and included in the AAAI Digital Library. This final tutorial description should use the AAAI two-column, camera-ready Word/Latex templates and must be submitted by July 25, 2024.
Please contact HCOMP 2024 Tutorials Co-Chairs for any questions concerning workshop proposals.
All times are midnight AoE
Issues of data quality and the provenance of crowdsourcing data have long been a focus within human computation research. Practitioners have typically relied on techniques such as redundancy, accuracy testing and attention checks to ensure a diverse and reliable set of responses from participants. In recent years, however, large language models (LLMs) like ChatGPT have changed the rules of the game in crowdsourcing, making it easy to produce human-sounding text and solve various human intelligence tasks (HITs) at the level of professional annotators. This technology thus threatens the reliability of crowdsourced data labelling as a means to collect manually curated datasets, which are critical for testing and evaluation of AI models and applications. This track solicits research pertaining to the provenance and reliability of crowdsourcing task data in the age of large language models.
We welcome submissions that seek to replicate crowdsourcing studies. Studies should re-run past crowdsourcing experiments first completed prior to the release of modern LLMs and tools (e.g., ChatGPT). Wherever possible, HIT features such as task design, reward structures and policies should be the same as in the original study. Studies should also compare their results with those of the original study, while also including mechanisms for capturing and/or estimating the use of large language models and other AI tools (either through self-reported usage or by analysing submission traces).
Experiments should report on and discuss how the quality and quantity of crowdworker contributions compare across experiments through metrics such as:
We particularly welcome submissions related to crowdsourcing tasks which are currently difficult for LLMs. These may include (but are not limited to):
Submissions should include the following:
Successful submissions will be invited to present at HCOMP 2024. Authors of successful submissions who are unable to attend will be invited to submit a pre-recorded video presentation. Please note that while the track will be archival, proceedings from this track will not be published alongside the main conference proceedings and will instead be published in a separate proceedings series published by CEUR.
CrowdCamp is a hackathon for researchers and practitioners interested in crowdsourcing, human computation, collective intelligence, and AI. The focus is on creating a prototype, a study design, an idea for a new algorithm, a pilot study, or something else entirely within the workshop itself. Prior CrowdCamp projects have resulted in top-tier conference publications, blog posts, and ongoing research.
This year’s CrowdCamp will focus on the theme “Responsible Crowd Work for Better AI.” We leave this up to participants to decide how to interpret this theme. Projects could involve topics such as mitigating bias in the crowd work pipeline, facilitating greater inclusion and designing safer work practices for crowd workers, and developing novel systems for more efficient use of crowd labor. Projects may include generative AI in their solution, though it is by no means a requirement.
The workshop will bring together researchers and practitioners across disciplines to explore innovative ideas and challenges that the fields of human computation and crowdsourcing are currently facing. We also welcome any other ideas that participants want to work on.
We invite students, faculty, and industry researchers to participate, from social scientists to programmers, ethnographers to designers, and anyone else interested. Everyone is welcome!
To apply to participate, you must complete the CrowdCamp 2024 application form: CrowdCamp 2024 Application (google.com). The application takes about 10 minutes to complete and asks for:
This year, CrowdCamp will be an all-day, in-person experience. Accepted participants are expected to attend on-site. While most teams will be formed on the day of the CrowdCamp, accepted participants can create teams before the convening if desired. Key Dates (all times are midnight AoE)
We welcome everyone who is interested in crowdsourcing and human computation to: