2021 Shared Task Description: Automatic Grading of Evidence - 10 years later

Basic Task Description

This task is a re-visit of the ALTA 2011 Shared task. At that time, the best-performing system used cascaded Support Vector Machine [Molla & Sarker, 2011]. We want to know if more recent developments in text processing can do better!

The basic task is to build an automatic evidence grading system for evidence-based medicine. Evidence-based medicine is a medical practice which requires practitioners to search medical literature for evidence when making clinical decisions. The practitioners are also required to grade the quality of extracted evidence on some chosen scale. The goal of the grading system is to automatically determine the grade of an evidence given the article abstract(s) from which the evidence is extracted.

You will be provided with:

1. a set of training documents;
2. a set of development documents; and
3. closer to the submission deadline, a set of unlabelled test documents.

The grading scale used for this task is the Strength of Recommendation Taxonomy (SORT). This taxonomy has 3 grades - A (strong), B (moderate) and C (weak). The grade of an evidence depends on multiple factors and information about this grading scale can be found in the paper by Ebell et al. (2004)

The grades used for this task have been generated by medical experts. Your task is to implement a grading system based on the training and development datasets, to then run over the test documents to determine the grade of each evidence.

Data Files and Submission

We will use CodaLab for this year's competition (https://competitions.codalab.org/competitions/33739). On registration to the competition site you will be able to do the following:

• Download the training data and devtest set for developing your system.
• Participate in discussion forums.
• Submit the results of your system using the devtest data and join the leaderboard.
• Download the test data and submit the final results.

In order to access the CodaLab pages and data, you need to register.

Important Dates

Publications