2022 Shared Task Description

Basic Task Description

This task is a re-visit of the ALTA 2012 Shared Task. At that time, the best-performing system Lui, 2012, Amini et al., 2012] used feature stacking and logistic regression. We want to know whether more recent developments in text processing can do better than then.

The goal of this task is to build automatic sentence classifiers that can map the content of biomedical abstracts into a set of pre-defined categories, which are used for Evidence-Based Medicine (EBM). EBM practitioners rely on specific criteria when judging whether a scientific article is relevant to a given question. They generally follow the PICO criterion: Population (P) (i.e., participants in a study); Intervention (I); Comparison (C) (if appropriate); and Outcome (O) (of an Intervention). Variations and extensions of this classification have been proposed, and for this task we will extend PICO by adding the classes Background (B) and Study Design (S); and including sentences that have no relevant content: Other (O). Therefore, the goal will be to classify the provided sentences according to the PIBOSO schema. Such information could be leveraged in various ways: e.g., to improve search performance; to enable structured querying with specific categories; and to aid users in more quickly making judgements against specified PICOSO criteria.

This is a multi-label classification problem, since each sentence can have more than one label. The tagset is defined as follows:

• Background: Material that informs and may place the current study in perspective, e.g. work that preceded the current; information about disease prevalence; etc;
• Population: The group of individual persons, objects, or items comprising the study's sample, or from which the sample was taken for statistical measurement;
• Intervention: The act of interfering with a condition to modify it or with a process to change its course (includes prevention);
• Outcome: The sentence(s) that best summarise(s) the consequences of an intervention;
• Study Design: The type of study that is described in the abstract;
• Other: Any sentence not falling into one of the other categories and presumed to provide little help with clinical decision making, i.e. non-key or irrelevant sentences.

More information about this problem, the construction of the dataset, and a benchmark can be found in Kim et al. (2011). The original data is provided by former NICTA, now CSIRO DATA61, and curated for the earlier (2012) shared task by Iman Amini and David Martinez. The data for this (2022) shared task has been adapted by Diego Molla (diego.molla-aliod@mq.edu.au).

Data Files and Submission

Important Dates