AutoMin 2021 Call for Participation

AutoMin 2021: The 1st Shared Task on Automatic Minuting

When most of our interactions went virtual, the need for automatic support for smooth running of the online events such as project meetings became more intense. Summarizing meeting contents is one of them. Meeting minutes keep a record of what was discussed at a meeting. It is usually a written document with little or no structure (perhaps a hierarchical bulleted list) aimed at informing the participants and non-participants of what happened during the meeting. ‘Automatic minuting’ tools would be a useful addition to better comprehend the meeting contents quickly. People adopt different styles when ‘taking notes’ or ‘recording minutes’ of the meeting. The minutes also depend on the category of the meeting, the intended audience, and the goal or objective of the meeting. Text or speech summarization methods known from the past would rank close to this task. However, Automatic Minuting is challenging due to the absence of agreed-upon guidelines, variety of minuting practices, and lack of extensive background research.

We propose AutoMin, the first shared task on automatic minuting of meeting proceedings. Our objective is to drive community efforts towards understanding the challenges of the task and develop tools for this important use case especially in the current world which had to go on-line far more than expected. With this shared task, we would invite the speech and natural language processing community to investigate the challenges of automatic minuting with real meeting data in two different settings: technical project meetings (both in English and Czech) and parliamentary proceedings (English).

AutoMin Task

We propose one main task and two subsidiary tasks. The subsidiary tasks are optional.

    Main Task A: The main task consists of automatically creating minutes from multiparty meeting transcripts. The generated minute would be evaluated both via automatic and manual metrics.
  • Task B: Given a pair of meeting transcript and minute, the task is to identify whether the minute belongs to the transcript. During our data preparation from meetings on similar topics, we found that this task could be challenging given the similarity in various named-entities.
  • Task C: Given a pair of minutes, the task is to identify whether the two minutes belong to the same or different meetings. This sub-task is important as we want to uncover how minutes created by two different persons for the same meeting may differ in content and coverage.

Procedure Overview

  1. We have released our trial data, to illustrate the task and the formats used.
  2. We will release available training and dev-set data which will match in a format exactly with the trial data.
  3. The training and dev-set data will contain the inputs and the reference outputs. (E.g. for Task A, this will be the transcripts and one or sometimes more manually created minutes.)
  4. The shared task itself will run for a month: Mid June till Mid July, 2021. You will be given test inputs and no reference outputs. You will be expected to submit your outputs during the evaluation, before the System Output Submission deadline.
  5. You will be expected to write a paper describing your system and submit by the System Report Submission deadline.
  6. Due to time constraints, we expect that the official results of the task will be available only after the System Report Submission deadline; you will have a chance to include them in your report before the Camera-Ready deadline for the report.
  7. Based on the results and system reports, we will invite about 50% of teams to write an extended description of their system, to form a special issue of Information journal by MDPI.

Data

The data for the shared task would be available in the following Github repository. More task-specific details to use the data would be provided in due time on our website. https://github.com/ELITR/automin-2021

Aside from the data we release, we recommend the following datasets to use in your training although our domains do not match:

Task participants are allowed to use any further data. When submitting, you need to indicate which data was used to train your system:
  • Minimal - minimal submissions use only the in-domain training data
  • Constraint - constraint submissions use the in-domain training data and CNN-DM, AMI, and ICSI.
  • Non-constraint - non-constraint submissions may use any other data.

In any case, please clearly describe which data was used in what way in your system paper. A comprehensive list of summarization datasets could be found here:

Evaluation

Manual Evaluation of Task A
For the manual evaluation of Task A, we will use several quality criteria which are common for evaluation of text samples produced by automatic language generation systems: adequacy, readability, and grammaticality. Unlike other similar tasks, textual coherence will not be taken into account, because we believe meeting minutes are not always supposed to have a coherent textual form. The manual evaluation will be carried out blindly by our annotators. If you would like to contribute to the evaluation, it would be highly appreciated. Please get in touch.
Automatic Evaluation of Task A
ROUGE would be the primary metric for automatic evaluation (ROUGE-1, ROUGE-2, ROUGE-L)
Evaluation of Tasks B and C
For the subsidiary tasks (B and C), classwise F1 would be the evaluation metric.

Submission Platform

We would ask our participants to host their system-runs in their own GitHub repository and share the link with us with exactly the system requirements/environment to run their code. Also, they would submit their automatically generated outputs. In general, you will be expected to submit the outputs of your system in Task A and optionally in Task B and/or Task C, in some fairly simple format based on plain text. Please refer to the submission page for details.

Publication

All teams are required to submit a brief technical report describing their method. Please use the Interspeech template for your system description reports. All reports must be a minimum of 2 pages and a maximum of: 5 pages excluding references (for single task), 8 pages (for multiple tasks). Reports must be written in English. Authors would submit their papers to minute@ufal.mff.cuni.cz. The proceedings would be added to the ISCA archive.

Extended Versions to Journal

We would additionally invite selected authors to submit a full-paper to a special issue of the open access Information journal from MDPI which is indexed within Scopus, ESCI (Web of Science), Ei Compendex, DBLP, and many other databases. The journal submissions would undergo further review. Authors of invited papers should be aware that the final submitted manuscript must provide a minimum of 50% new content and not exceed 30% copy/paste from the proceedings paper.

Contact

For further information about this task and dataset, please contact: