ALTA logo Language Technology Programming Competition 2020
Task Description
Useful Information

2020 Shared Task Description: Assess Human Behaviour

Basic Task Description

Can we assess human behaviour automatically from textual data?

Human behaviour can be negatively or positively assessed based on a reference set of social norms. When judgement is explicitly stated in narratives, e.g., "They are hard-working and honest.", we encounter appraisal words such as "hard-working" and "honest" used between interlocutors for advancing an opinion.

Attitude positioning plays an important role in Martin and White's (2005) proposed Appraisal framework for analysing someone's use of evaluative language to negotiate solidarity. The APPRAISAL framework is concerned with the use of linguistic markers for identifying and track the ways attitudes are invoked in authored text. The framework defines three subsystems for evaluative meaning making: (1) ATTITUDE; (2) ENGAGEMENT; and (3) GRADUATION. The ATTITUDE framework is further divided into three subsystems: (1) AFFECT (registering of emotions); (2) APPRECIATION (evaluations of natural and semiotic phenomena); and (3) JUDGEMENT (evaluations of people and their behavior).

The judgement subsystem has two regions: social esteem and social sanction.

Social esteem tends to function as admiration or criticism and can be subdivided into three subcategories:

  • normality (how unusual one is): "He is old-fashioned".
  • capacity (how capable one is): "Self-driven 12 year old is a maths genius".
  • tenacity (how resolute one is): "They are hard-working and honest".

Social sanction functions as praise or condemnation and can be subdivided into two subcategories:

  • veracity (how honest/truthful one is): "They are hard-working and honest"
  • propriety (how ethical one is): "She is too arrogant to learn the error of her ways".

Participants are referred to The Appraisal website for further reading about the Appraisal framework.

The goal of this task is to develop a computational model that can identify and classify judgements expressed in textual segments. Participants are challenged to predict the judgement appraised by classifying each short-text messages into one or more label candidates (or none): normality, capacity, tenacity, veracity, propriety. The follow table shows examples of messages and their classification.

Read and try to comprehend what you have commented on. 01000
Fans of adoring Dictatorships and Totalitarians 00001
Keep going like you always have done. 00100
She showed her true colors. 00001
He is a nasty person. 10001
Corruption 101 00010


We will use Kaggle in Class to evaluate the systems.

Data Files and Submission

We will use Kaggle in Class for this year's competition (look for the ALTA 2020 Challenge). The data files and submission instructions will be provided in the competition website.

In order to access the Kaggle in Class pages, you need to register.

Dataset and descriptions

The task includes one dataset sourced from the SemEval 2018 AIT DISC annotated with Judgement categories.

Important Dates

Release of training data On registration
Deadline for submission of results over test data 30 Oct 2020
Notification of results 3 Nov 2020
Deadline for submission of system description 20 Nov 2020
Presentation of results at the ALTA workshop 13-15 Jan 2021


© ALTA 2020. Competition Organisers.