This year, the Human Computation (HCOMP) and Collective Intelligence (CI) conferences are being jointly organized. We are excited about this opportunity to bring our two communities closer together, increase the visibility of work in this area, promote more multidisciplinary research, and create a diverse and engaging program. We look forward to your submissions and welcoming you to our joint conference. We hope to foster a fruitful and diverse exchange of ideas, research, and experiences across the HCOMP and CI communities.
All times are midnight AoE
We are thrilled to announce that the joint organization of CI and HCOMP will provide authors with the opportunity to access the best aspects of both conferences. With this collaboration, HCOMP will now allow authors to publish their accepted papers as extended abstracts if desired so that they can submit their work elsewhere for archival publication, while retaining a presentation at the conference. This change encourages social science researchers to participate and share their work. Meanwhile, CI will now enable authors to publish their papers archivally, thereby enhancing the conference's academic value. We believe this collaboration will enable authors to showcase their work more effectively and contribute to advancing the fields of human computation, crowdsourcing, and computational intelligence.
To present a talk at the CI or HCOMP conference, authors must submit a full paper for review by the program committee. Accepted papers will be granted a presentation at the conference. Therefore, submitting a full paper is a mandatory requirement for authors who wish to present their work at the conference.
Extended abstract option.
For some researchers and fields (e.g., social scientists or economists), it is beneficial to retain the option to publish the final paper outside of a conference venue. For other researchers, there is benefit from sharing work that was published in other venues with HCOMP-CI’s interdisciplinary audience. To support these researchers, there is an extended abstract option: publish only an extended abstract in the proceedings, should the paper be accepted. Extended abstracts still must submit a full paper for review. Electing for this extended abstraction option is a commitment to replacing the submitted full paper with a two page extended abstract at publication time. Submissions using this extended abstract option may be previously published elsewhere, or may be submitted for publication elsewhere later. Authors opting for the extended abstract option will have their papers juried rather than peer-reviewed; in other words, these articles will still be considered fully by the program committee, but authors will not receive reviews back—only a decision. Extended abstract papers will be treated like full papers in the conference schedule: they will receive a presentation slot. If the work is previously published, authors must clearly and visibly indicate the original paper and citation in their manuscript.
Forward-to-journal option.
Typically, accepted conference papers will be published only in the proceedings, and if authors later want to extend or publish the work in another venue, they need to start a new review process. This year, however, we are excited to share that authors will have the opportunity to forward their papers, alongside their existing reviews and reviewers, for consideration to the ACM + SAGE Collective Intelligence journal. The journal will, at its discretion, keep or expand the existing set of reviewers for the article and consider the work for publication in the Collective Intelligence journal, accelerating the journal publication process. This option must be selected at submission time. Papers submitting under the extended abstract option will have reviewers forwarded, but since extended abstract option submissions do not receive written reviews, reviews will not be forwarded. Authors should bear in mind that, if the paper is also published archivally at the conference, the journal may ask for expanded content.
The conference is organized with one paper presentation track and two separate program subcommittees for paper submission and review. Authors may choose to submit to either the HCOMP or Collective Intelligence subcommittee. Each subcommittee reflects different areas of interest, as well as slightly different submission formats and instructions due to HCOMP being sponsored by AAAI and Collective Intelligence being sponsored by ACM.
Submit to HCOMP using Easychair here.
Topics
AAAI HCOMP publishes research on human computation and crowdsourcing. Its focus is on research and practice into frameworks, methods and systems that bring together people and machine intelligence to achieve better results.
While artificial intelligence (AI) and human-computer interaction (HCI) represent traditional mainstays of the conference, HCOMP believes strongly in fostering and promoting broad, interdisciplinary research. Our field is particularly unique in the diversity of disciplines it draws upon and contributes to, including human-centered qualitative studies and HCI design, social computing, artificial intelligence, economics, computational social science, digital humanities, policy, and ethics. We promote the exchange of advances in human computation and crowdsourcing not only among researchers, but also engineers and practitioners, to encourage dialogue across disciplines and communities of practice. Submissions may hence cover theory, studies, tools and applications that present novel, interesting, impactful interactions between people and computational systems. These cover a broad range of scenarios, from classical human computation, wisdom of the crowds, and all forms of crowdsourcing to people-centric AI methods, systems and applications.
Topics of interest include:
Crowdsourcing applications and techniques, including but not limited to: citizen science, collective action, collective intelligence, wisdom of the crowds, crowdsourcing contests, crowd creativity, crowdfunding, paid microtasks, crowd ideation, crowd sensing, prediction markets.
Techniques that enable and enhance human-in-the-loop systems, making them more efficient, accurate, and human-friendly, including task design, quality assurance, answer inference, biases and subjectivity, incentives, gamification, task allocation, complex workflows, real-time crowdsourcing etc.
Approaches to make crowd science FAIR (Findable, Accessible, Interoperable, Reproducible) and studies assessing and commenting on the FAIRness of human computation and crowdsourcing practice.
Techniques that enable and enhance human-in-the-loop systems, making them more efficient, accurate, and human-friendly, including task design, quality assurance, answer inference, biases and subjectivity, incentives, gamification, task allocation, complex workflows, real-time crowdsourcing, etc.
Studies about how people perform tasks individually, in groups, or as a crowd, including those drawing on techniques from human-computer interaction, social computing, computer-supported cooperative work, design, cognitive sciences, behavioral sciences, economics, etc.
Studies into the reliability and other quality aspects of human-annotated and -curated datasets.
Studies into replicability of crowdsourcing and human computation experiments.
Methods that use human computation and crowdsourcing to build people-centric AI systems and applications, including topics such as explainability and interpretability.
Studies into fairness, accountability, transparency, ethics, and policy implications for crowdsourcing and human computation.
Studies about how people and intelligent systems interact and collaborate with each other and studies revealing the influences and impact of intelligent systems on society.
Studies that inform our understanding about the future of work, distributed work, the freelancer economy, open innovation and citizen-led innovation.
Format
Authors are invited to submit papers of up to 10 pages, plus any number of additional pages containing references only. Papers must be formatted in AAAI two-column, camera-ready style; please refer to the AAAI 2023 Author Kit for details. The AAAI copyright block is not required on submissions but must be included on final accepted versions. Accepted full papers will be allocated ten (10) pages in the conference proceedings. Final papers found to exceed page limits and or otherwise violating the instructions to authors will not be included in the proceedings.
Authors are invited, but not required, to include supplemental materials such as executables and data files so that reviewers can reproduce results in the paper, images, additional videos, related papers, more detailed explanations, derivations, or results. These materials will be viewed only at the discretion of the reviewers, who are only obligated to read your paper itself.
Authors will be required to transfer copyright of their paper to AAAI. Accepted full papers will be published in the HCOMP conference proceedings and included in the AAAI Digital Library.
AAAI enables authors to use Open Responsible AI Licenses (Open RAIL), licenses designed to permit free and open access, re-use, and downstream distribution of derivatives of AI artifacts as long as the behavioral-use restrictions always apply (including to derivative works). Need Help Deciding if an AI Pubs License is Right for You? – review this section. Please consider this as a new way to share your work with the community. If you have questions, comments, or feedback, please contact the RAIL team here: https://www.licenses.ai/contact
Submit to Collective Intelligence using Easychair here.
Topics
ACM Collective Intelligence has a transdisciplinary focus devoted to advancing the theoretical and empirical understanding of collective performance in diverse systems, including human organizations, hybrid AI-human teams, computer networks, adaptive matter, cellular systems, neural circuits, animal societies, nanobot swarms, and others. Contributions may include principles that apply across scales, as well as new ways of harnessing the collective to improve social, ecological, and economic outcomes.
Collective intelligence has typically published a broad range of work across the social and behavioral sciences and computing, but also welcomes work in areas such as the natural sciences and humanities. We seek perspectives that emphasize both traditional views of intelligence as well as optimality, satisficing, robustness, adaptability, and wisdom. In more technical terms, this includes issues related to collective output quality and assessment, aggregation of information and related topics (e.g., network structure and dynamics, higher-order vs. pairwise interactions, spatial and temporal synchronization, diversity, etc.), accumulation of information by individuals or components, environmental complexity, evolutionary considerations, and design of systems and platforms fostering collective intelligence. Methods may range from behavioral, to theoretical, to engineering.
Topics of interest include:
Collective decision-making studies and systems, focused on topics such as citizen engagement, collective creativity, collaborative problem-solving, extended and distributed cognition, crowdsourcing, incentive mechanisms, innovation, knowledge creation, organizational design, participatory decision-making, prediction markets, social computing, strategy formation, forecasting, team structures, wisdom of crowds, and voting design
Artificial intelligence techniques that amplify or exhibit collective intelligence, such as adaptive dynamic systems, robotics, and multi-agent technology
Data science investigations of collectively intelligent systems, including biological systems, the emergence of intelligence, evolutionary intelligence
Ethics and values, including societal norms
Format
CI papers are of variable length. Paper length must be based on the weight of the contribution. A new idea presented in a compact format is more likely to be accepted than the same idea in a long format, and shorter, more focused papers are encouraged. As a guideline, please consider papers in the range of 10,000 words.
Collective Intelligence will adopt the ACM TAPS Workflow. For LaTeX authors, submissions should be made using the double-column format (example PDF) using \documentclass[sigconf,review,anonymous]{acmart}. For Word authors, submissions should be made using either the double-column format (submission template, example PDF) or the single-column template (submission template, example PDF). Please ensure that you and your co-authors obtain an ORCID ID, so you can complete the publishing process for your accepted paper. ACM has been involved in ORCID from the start and we have recently made a commitment to collect ORCID IDs from all of our published authors. The collection process has started and will roll out as a requirement throughout 2022. We are committed to improve author discoverability, ensure proper attribution and contribute to ongoing community efforts around name normalization; your ORCID ID will help in these efforts.
Authors are invited, but not required, to include supplemental materials such as executables and data files so that reviewers can reproduce results in the paper, images, additional videos, related papers, more detailed explanations, derivations, or results. These materials will be viewed only at the discretion of the reviewers, who are only obligated to read your paper itself.
By submitting your article to an ACM Publication, you are hereby acknowledging that you and your co-authors are subject to all [ACM Publications Policies](https://www.acm.org/publications/policies), including ACM's new [Publications Policy on Research Involving Human Participants and Subjects](https://www.acm.org/publications/policies/research-involving-human-participants-and-subjects). Alleged violations of this policy or any ACM Publications Policy will be investigated by ACM and may result in a full retraction of your paper, in addition to other potential penalties, as per ACM Publications Policy.
Anonymity.
All papers must be anonymized (including no information identifying the authors or their institutions) for double-blind peer review.
Preprint servers.
We do not have a policy against uploading preprints to SSRN or arXiv before they are submitted for review at the conference. Nevertheless, to ensure the integrity of the peer review process, we ask that no authors publicize the work until that process is complete. Please do not share confidential info specific to a current review process on social media or in similarly public forums.
Conflicts of interest.
To ensure fairness, authors should declare any conflicts of interest with PC members by selecting the “Declare Conflicts” link on the upper-right of your EasyChair submission page.
Double Submission Policy:
Excepting papers submitting under the extended abstract option, all papers submitted to the conference must represent original work that has not been previously published or under simultaneous peer-review for any other peer-reviewed archival conference or journal. Specifically:
Papers that have appeared at a conference with published proceedings constitute previously published work.
Papers overlapping papers that have appeared at a conference with published proceedings must contain significant new results.
Papers appearing at a workshop do not constitute previously published work as long as the paper submitted to HCOMP is an extension of the workshop paper. Extensions might include new results, more in-depth analysis, an evaluation not part of the workshop paper, or further experiments.
Reviews.
Each full paper submission will be reviewed by at least two members of the program committee and one AC. Reviewers will be instructed to evaluate paper submissions according to specific review criteria. We encourage authors to review them before submission.
To ensure relevance, authors should consider including research questions and contributions of broad interest to crowdsourcing and human computation, as well as discuss relevant open problems and prior work in the field. When the evaluation is conducted within a specific domain, authors are encouraged to discuss how findings might generalize to other communities and application areas using crowdsourcing and human computation.
Attendance.
To be included in the proceedings and the conference program, at least one author must register in person for the main conference. The registration needs to occur by the camera-ready deadline.
Presentation.
If your paper is accepted, we are delighted to invite you to present your work at the conference. Please note that at least one author of each accepted paper must register for the main conference to present their work in person. Failure to do so will result in the withdrawal of acceptance. Remote presentation of accepted papers is not permitted except in the case of unforeseen circumstances. The deadline for registering to present your paper is the same as the camera-ready deadline.
Collective Intelligence and HCOMP 2023 will recognize the best paper winner and honorable mention(s). Reviewers will be asked to flag papers they deem worthy of a prize. The general chairs will set up a small panel that will read the papers, consider the reviewers' comments and assess the talk to determine the winners.
The Works-in-Progress and Demonstration track focuses on recent findings or other types of innovative or thought-provoking work, hands-on demonstration, novel methods, technologies and experiences relevant to the HCOMP and CI communities. We encourage practitioners and researchers to submit to the Works-in-Progress & Demo Track as it provides a unique opportunity for sharing valuable insights and ideas, eliciting useful feedback on early-stage work, and fostering discussions and collaborations among colleagues. Submissions are welcome from multiple fields, ranging from computer science, artificial intelligence, and human-computer interaction, to economics, business, and the social sciences, all the way to digital humanities, policy, and ethics.
Accepted papers in this track will be non-archival and they will not be included in the official proceedings of the HCOMP/CI conference. They will be made available online on the conference website. Authors of accepted papers can thus benefit from exchanging insights on their work, while maintaining the option to further develop their idea and submit the outcome to other venues.
A Work-in-Progress is a concise report of recent findings or other types of innovative or thought-provoking work relevant to the HCOMP and/or CI community. The difference between Works-in-Progress and other types of contribution is that Work-in-Progress submissions represent work that has not yet reached a level of completion that would warrant the full Refereed selection process. That said, appropriate submissions should make some contribution to the body of HCOMP/CI knowledge, whether realized or promised. A significant benefit of a Work-in-Progress derives from the discussion between the author and the conference attendees. Work-in-Progress submissions are in the form of a 2-3 page paper; see Submission section. Authors of accepted Works-In-Progress will present their paper in-person at the conference in the form of a poster.
A demonstration is a high-visibility, high-impact forum of the HCOMP/CI program that allows you to present your hands-on demonstration, share novel technologies and tools, and stage interactive experiences. Demonstrations will showcase this year’s most exciting human computation, collaborative human-AI, and collective intelligence prototypes and systems. If you have an interesting prototype, system, exhibit or installation, we want to know about it. Sharing hands-on experiences of your work is often the best way to communicate what you have created. The demonstration submission should describe the nature of the system as well as the expected form of interaction with the user and the audience. Demonstrations submissions are in the form of a 2-3 page paper; see Submission section.
Submissions for both HCOMP and CI Works-In-Progress and Demo papers must be done via the HCOMP/CI 2023 EasyChair Website under the Works-in-Progress and Demonstration track. Submissions to Easychair are required on or before the deadline listed above. We cannot accept submissions by e-mail. Authors will receive confirmation of receipt of their submissions, including an ID number, shortly after submission. HCOMP/CI will contact authors again only if problems are encountered with their papers.
Works-in-Progress and Demonstration papers must be between 2 and 3 pages of content. References can extend beyond the page limit.
Papers must be formatted in AAAI two-column, camera-ready style; please refer to the AAAI 2023 Author Kit for details. Papers must be in trouble-free, high-resolution PDF format, formatted for US Letter (8.5″ x 11″) paper, using Type 1 or TrueType fonts. The AAAI copyright block is not required for works-in-progress or demo submissions, as they are not included in the formal proceedings. Please see below for information about publication.
Authors are invited, but not required, to include supplemental materials such as executables and data files so that the results of the work can be reproduced.
Authors should include information identifying themselves and their institutions for single-blind review.
Accepted papers will NOT be included in official conference proceedings, and therefore they could be submitted later to other conferences or journals for official publication. Accepted papers will be made available online on the conference website.
Please contact the Works-in-Progress and Demonstration Co-Chairs if you have any questions.
This year, the Collective Intelligence (CI) and Human Computation (HCOMP) conferences are being jointly organized and will have a joint Doctoral Consortium to provide doctoral students across both communities with a unique opportunity to meet each other and experienced researchers in the field. Students will be mentored by a group of faculty who are leaders in the diverse specialties that make up the CI and HCOMP fields. The objectives of the Doctoral Consortium are to provide students with an opportunity:
The joint organization of CI and HCOMP this year is also a great opportunity to increase the visibility of doctoral student work and to build connections between the two communities.
CI and HCOMP are similarly unique in the diversity of disciplines they draw upon and contribute to, ranging from computer science, artificial intelligence, and human-computer interaction, to economics, organizational behavior, and the social, all the way to digital humanities, policy, ethics, and biology. Any DC submissions fitting the topics of interest in the overall Call for Submissions are applicable. For example:
Topics include collective decision-making studies and systems, artificial techniques that amplify or exhibit collective intelligence, data science investigations of collectively intelligent systems, ethics and values, including societal norms, crowdsourcing applications and techniques, techniques that enable and enhance human-in-the-loop systems, approaches to make crowd science FAIR (findable, accessible, interoperable, reproducible), studies about how people perform tasks individually, in groups, or as a crowd, studies into the reliability and other quality aspects of human-annotated and -curated datasets, studies into replicability of crowdsourcing and human computation experiments, methods that use human computation and crowdsourcing to build people-centric AI systems and applications, studies into fairness, accountability, transparency, ethics, and policy implications for crowdsourcing and human computation, studies about how people and intelligent systems interact and collaborate with each other, and studies that inform our understanding about the future of work.
Applicants must be currently enrolled in a PhD program. The Doctoral Consortium accepts students from any stage in their candidature, but will prioritize candidates who have a clear topic and research approach and have made some progress, but are not so far along that they can no longer make changes. Before submitting, students should discuss this criterion with their advisor or supervisor.
Those accepted are required to present their work at the Doctoral Consortium. Participants will also likely be required to present a poster on their work during the poster session at the main conference.
Submissions will be reviewed based upon the expected potential of both the student and their proposed work, as well as the expected benefit to the student from participation. Priority will be given to students whose research goes beyond locally available expertise at their home institutions.
We encourage applicants to explore the Gary Marsden Travel Awards for financial support. Deadlines for applications are the 9th of July and the 9th of September. More information can be found on the SIGCHI website. We note that the Gary Marsden Travel Award only applies to SIGCHI conferences. Since CI is a SIGCHI conference, and since the Doctoral Consortium is being held as a joint CI+HCOMP event this year, it should be eligible for that program.
Applicants must submit a solely-authored paper in English containing: 1) a Doctoral Research Overview; and 2) a Supplemental Paragraph.
Please summarize your doctoral research, including the following sections:
Please write a paragraph explaining: Why you want to participate in the consortium at this point in your doctoral studies and how you expect to benefit from the consortium; Your expected (approximate) completion date.
Your paper should be no more than 4 pages in total: 3 pages for the Doctoral Research Overview (including all figures and references), and the 4th page being the Supplemental Paragraph. The first page must contain the title of the paper, full author name, affiliation and contact details, an abstract of up to 250 words, and up to 3 keywords describing the research topic areas.
For consistency, we ask that all submissions to the CI+HCOMP Doctoral Consortium be formatted in the same way using the ACM template and process described for the CoIlective Intelligence subcommittee in the CI+HCOMP 2023 Call for Papers.
For simplicity, we ask that all submissions to the CI+HCOMP Doctoral Consortium be submitted through the same portal using the CI 2023 EasyChair paper submission site. Submissions must be made on or before the deadlines listed above. We cannot accept submissions by e-mail or fax. Authors will receive confirmation of receipt of their submissions, including an ID number, shortly after submission. CI+HCOMP will contact authors again only if problems are encountered with papers.
Submissions will be distributed only to mentors and other attendees of the doctoral consortium. Proceedings of the Doctoral Consortium will NOT be archived. As such, students may freely submit their research contributions for official publication in other venues. Participant names and university affiliations, as well as paper titles and abstracts, will be publicized on the conference website and in the conference program for the poster session.
Please feel free to direct all questions or comments to the following email addresses: anna.cox@ucl.ac.uk or dlee105@ucsc.edu.
We invite proposals for workshops at HCOMP 2023 / CI 2023. Workshops bring together communities with common research interests and agendas to discuss ongoing work and initiate new collaborations. Workshops may address research, best practices, tools, education, emerging themes and applications, and critical thinking about existing methodologies and frameworks. Topics should be within the general scope of HCOMP or CI (or both), but may include emerging research directions that have not yet been fully explored and may even be seen as controversial. Workshops may also be a forum for other types of topic-specific activities, such as tutorials or roundtables.
Workshop format can include discussions of contributions from attendees (e.g., posters, lightning talks) as well as providing space for lively, informal debates (e.g., breakout groups, panel discussions).
Keep everyone involved! We especially encourage workshop formats with activities that keep attendees engaged throughout the workshop. Get creative! Consider incorporating demos, feedback on innovative task designs, or short-term collaborations. What can attendees create? … or solve?
If you would like to organize a workshop at HCOMP 2023 / CI 2023, please submit a PDF (2-4 pages) by June 16 via EasyChair (see below).
Note: Priority will be given to workshops that include a diverse group of organizers across different dimensions including but not limited to affiliation, affiliation type (e.g., academia, industry), geographical location, and area of research.
https://easychair.org/conferences/?conf=hcomp23 Track: workshops.
Workshop day(s):
Sign-up Deadline: October 1, 2023
CrowdCamp is a one-day hack-a-thon for researchers and practitioners with interests in crowdsourcing, human computation, collective intelligence, and AI. The focus is on creating a first prototype, study design, an idea for a new algorithm, a pilot study, or something else entirely within the workshop itself. Prior CrowdCamp projects have resulted in top-tier conference publications, blog posts, and on-going research.
As the theme for this year’s CrowdCamp, we propose generative AI which is revolutionizing various fields, including art, entertainment, and education. However, the full potential of generative AI is yet to be realized, and the role of human intelligence and crowd computing in shaping generative AI is critical for AI alignment. Similarly, there is a need for a timely response to systematically identify and mitigate safety concerns in all generative models that are currently available (see, for instance, the Adversarial Nibbler data challenge).
The workshop will bring together researchers and practitioners from both the crowdsourcing and collective intelligence disciplines to explore innovative ideas as well as the challenges associated with generative AI models. We also welcome any other ideas that participants want to work on.
We invite students, faculty, and industry researchers to participate, from social scientists to programmers, ethnographers to designers, and anyone else interested. Everyone is welcome!
To apply to participate, we ask that you complete the CrowdCamp 2023 application form. The application takes about 10 minutes to complete, and asks for:
This year, CrowdCamp will be a hybrid experience. Accepted participants may choose to attend on-site or remotely. Throughout the day, there will be touchpoints where virtual and on-site groups can connect and showcase their work. We'll try to integrate the virtual teams via live streaming and shared sessions to create an inclusive environment for all attendees. More details to be shared shortly.
Past CrowdCamps have shown a strong track-record of bringing together diverse teams of researchers, engineers, and students who work passionately together on well-scoped project ideas. Many participants have continued working on their projects beyond CrowdCamp, generating impactful publications and fruitful collaborations. A few examples include:
1. What does the co-location mean to authors?
The co-location means that you'll have the unique opportunity to reach a much broader audience. In addition to the typical HCOMP community, your work will also be accessible to the CI community, which includes researchers from diverse fields such as citizen science, complex networks, and political science. Your paper presentation will have the potential to make an impact across multiple fields simultaneously. Furthermore, the co-location presents an excellent opportunity to connect with new researchers and explore new fields that could influence your work. It could even inspire your research to take new and exciting directions that you may not have previously considered.
2. How can they decide which conference they need to submit?
Please submit your paper to the conference that you would typically choose. In the event that we determine your paper may be a better fit for another conference, we will provide guidance to help you make the best decision for your research.
3. Can they submit to CI and present at the HCOMP?
Absolutely! Both conferences will feature the papers presented at each other, creating a unique opportunity for cross-pollination and collaboration between the two communities. All papers will be presented and exposed to both HCOMP and CI audiences, facilitating the exchange of ideas and the development of new insights.
4. Will the review process be different in both conferences or do they share the same.
The review process for each conference will follow its typical procedure. This ensures that all papers are evaluated according to the same rigorous standards and criteria that are standard for each conference.
We welcome everyone who is interested in crowdsourcing and human computation to: