|Language Technology Programming Competition 2021|
2021 Shared Task Description: Automatic Grading of Evidence - 10 years later
Basic Task Description
This task is a re-visit of the ALTA 2011 Shared task. At that time, the best-performing system used cascaded Support Vector Machine [Molla & Sarker, 2011]. We want to know if more recent developments in text processing can do better!
The basic task is to build an automatic evidence grading system for evidence-based medicine. Evidence-based medicine is a medical practice which requires practitioners to search medical literature for evidence when making clinical decisions. The practitioners are also required to grade the quality of extracted evidence on some chosen scale. The goal of the grading system is to automatically determine the grade of an evidence given the article abstract(s) from which the evidence is extracted.
You will be provided with:
The grading scale used for this task is the Strength of Recommendation Taxonomy (SORT). This taxonomy has 3 grades - A (strong), B (moderate) and C (weak). The grade of an evidence depends on multiple factors and information about this grading scale can be found in the paper by Ebell et al. (2004)
The grades used for this task have been generated by medical experts. Your task is to implement a grading system based on the training and development datasets, to then run over the test documents to determine the grade of each evidence.
Data Files and Submission
We will use CodaLab for this year's competition (https://competitions.codalab.org/competitions/33739). On registration to the competition site you will be able to do the following:
In order to access the CodaLab pages and data, you need to register.