2020 Shared Task Description: Assess Human Behaviour

Basic Task Description

Can we assess human behaviour automatically from textual data?

Human behaviour can be negatively or positively assessed based on a reference set of social norms. When judgement is explicitly stated in narratives, e.g., "They are hard-working and honest.", we encounter appraisal words such as "hard-working" and "honest" used between interlocutors for advancing an opinion.

Attitude positioning plays an important role in Martin and White's (2005) proposed Appraisal framework for analysing someone's use of evaluative language to negotiate solidarity. The APPRAISAL framework is concerned with the use of linguistic markers for identifying and track the ways attitudes are invoked in authored text. The framework defines three subsystems for evaluative meaning making: (1) ATTITUDE; (2) ENGAGEMENT; and (3) GRADUATION. The ATTITUDE framework is further divided into three subsystems: (1) AFFECT (registering of emotions); (2) APPRECIATION (evaluations of natural and semiotic phenomena); and (3) JUDGEMENT (evaluations of people and their behavior).

The judgement subsystem has two regions: social esteem and social sanction.

Social esteem tends to function as admiration or criticism and can be subdivided into three subcategories:

• normality (how unusual one is): "He is old-fashioned".
• capacity (how capable one is): "Self-driven 12 year old is a maths genius".
• tenacity (how resolute one is): "They are hard-working and honest".

Social sanction functions as praise or condemnation and can be subdivided into two subcategories:

• veracity (how honest/truthful one is): "They are hard-working and honest"
• propriety (how ethical one is): "She is too arrogant to learn the error of her ways".

Participants are referred to The Appraisal website for further reading about the Appraisal framework.

The goal of this task is to develop a computational model that can identify and classify judgements expressed in textual segments. Participants are challenged to predict the judgement appraised by classifying each short-text messages into one or more label candidates (or none): normality, capacity, tenacity, veracity, propriety. The follow table shows examples of messages and their classification.

Evaluation

We will use Kaggle in Class to evaluate the systems.

Data Files and Submission

In order to access the Kaggle in Class pages, you need to register.

Dataset and descriptions

The task includes one dataset sourced from the SemEval 2018 AIT DISC annotated with Judgement categories.

Important Dates