We approach Aspect-based Sentiment Analysis as a cascade of two subtasks: Aspect Category Detection (ACD) and Aspect Category Polarity (ACP).
For example, let us consider the sentence describing an hotel:
I servizi igienici sono puliti e il personale cordiale e disponibile. (Toilets are clean but the staff is not friendly nor helpful.)
In the ACD task one or more “aspect categories” evoked in a sentence are identified, e.g. the pulizia (cleanliness) and staff categories.
In the Aspect Category Polarity (ACP) task, the polarity of each expressed category is recognized, e.g. a positive category polarity is expressed concerning the pulizia category while it is negative if considering the staff category.
In our evaluation framework, the set of aspect categories is known and given to the participants, so the ACD task can be seen as a multi-class, non-exclusive classification task where each input text has to be classified as evoking or not each aspect category. The systems will be asked to return a binary vector where each dimension corresponds to an aspect category and the values 0 (false) and 1 (true) indicate whether each aspect has been detected in the text. The following table shows examples of annotation for the ACD task with four aspects.
| Text | Cleanliness | Staff | Comfort | Location |
|---|---|---|---|---|
| I servizi igienici sono puliti e il personale cordiale e disponibile | 1 | 1 | 0 | 0 |
| La posizione è molto comoda per il treno e la metro. | 0 | 0 | 0 | 1 |
| Ottima la disponibilità del personale, e la struttura della stanza | 0 | 1 | 1 | 0 |
For the ACP task, the input is the review text paired with the set of aspects identified in the text by the ACD subtask, and the goal is to assign polarity labels to each of the aspect category.
Two binary polarity labels are expected for each aspect: POS an NEG, indicating a positive and negative sentiment expressed towards a specific aspect, respectively. Note that the two labels are not mutually exclusive: in addition to the annotation of positive aspects (POS:true, NEG:false) and negative aspects (POS:false, NEG:true), there can be aspects with no polarity, or neutral polarity (POS:false, NEG:false). This is also the default polarity annotation for the aspects that are not detected in a text. Finally, the polarity of an aspect can be mixed (POS:true, NEG:true), in cases where both sentiments are expressed towards a certain aspect in a text. The following table summarizes the possible annotations with examples.
| Text | Aspect | POS | NEG |
|---|---|---|---|
| Il bagno andrebbe ristrutturato | Cleanliness | 0 | 0 |
| Camera pulita e spaziosa. | Cleanliness | 1 | 0 |
| Pulizia della camera non eccelsa. | Cleanliness | 0 | 1 |
| Il bagno era pulito ma lasciava un po a desiderare | Cleanliness | 1 | 1 |
The participants may choose to submit only the results of the ACD subtask, or both tasks. In the latter case, the output of the ACD task is used as input for the ACP.
As a constraint on the results submitted for the ACP task, the polarity of an aspect for a given sentence can be different than (POS:false, NEG:false) only if the aspect is detected in the ACD step.
The participants are allowed to use external resources such as lexicons in their systems. They are also allowed to use external data, either annotated or non-annotated, for training their systems, e.g. word embeddings. At the end of the competition, only one rank will be proposed, without distinction between systems that use/don’t use external resources. However, we encourage the participants to make choices that keep the scientific contribution significant, e.g. avoid potential overlap between the training set and the test set, avoid data from booking.com. All external resources are expected to be described in detail in the systems’ reports.