We evaluate the ACD and ACP subtasks separately by comparing the classifications provided by the participant systems to the gold standard annotations of the test set.<\/p>\n
For the ACD task, we compute Precision, Recall and F-score defined as:
\n,
\nwhere Precision () and Recall (
) are defined as:
\n.
\nHere is the set of aspect category annotations that a system returned for all the test sentences, and
is the set of the gold (correct) aspect category annotations.
\nFor instance, if a review is labeled in the gold standard with the two aspects
\n,
\nand the system predicts the two aspects
\n,
\nwe have that ,
and
so that
,
and
.
\nFor the ACD task the baseline will be computed by considering a system which assigns the most frequent aspect category (estimated over the training set) to each sentence. <\/p>\n
For the ACP task we will evaluate the entire chain, thus considering both the aspect categories detected in the sentences together with their corresponding polarity, in the form of pairs.
\nWe again compute Precision, Recall and F-score now defined as
\n.
\nPrecision () and Recall (
) are defined as
\n,
\nwhere is the set of
pairs that a system returned for all the test sentences, and
is the set of the gold (correct) pairs annotations.<\/p>\n
For instance, if a review is labeled in the gold standard with the pairs
\n,
\nand the system predicts the three pairs
\n,
\nwe have that ,
and
so that
,
and
.<\/p>\n
For the ACP task, the baseline will be computed by considering a system which assigns the most frequent pair (estimated over the training set) to each sentence. <\/p>\n
We will produce separate rankings for the tasks, based on the scores. Participants who submit only the result of the ACD task will appear in the first ranking only.<\/p>\n","protected":false},"excerpt":{"rendered":"
We evaluate the ACD and ACP subtasks separately by comparing the classifications provided by the participant systems to the gold standard annotations of the test set. For the ACD task, we compute Precision, Recall and F-score defined as: , where Precision () and Recall () are defined as: . Here is the set of aspect … Continue reading Evaluation<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":[],"_links":{"self":[{"href":"http:\/\/sag.art.uniroma2.it\/absita\/wp-json\/wp\/v2\/pages\/114"}],"collection":[{"href":"http:\/\/sag.art.uniroma2.it\/absita\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"http:\/\/sag.art.uniroma2.it\/absita\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"http:\/\/sag.art.uniroma2.it\/absita\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"http:\/\/sag.art.uniroma2.it\/absita\/wp-json\/wp\/v2\/comments?post=114"}],"version-history":[{"count":19,"href":"http:\/\/sag.art.uniroma2.it\/absita\/wp-json\/wp\/v2\/pages\/114\/revisions"}],"predecessor-version":[{"id":161,"href":"http:\/\/sag.art.uniroma2.it\/absita\/wp-json\/wp\/v2\/pages\/114\/revisions\/161"}],"wp:attachment":[{"href":"http:\/\/sag.art.uniroma2.it\/absita\/wp-json\/wp\/v2\/media?parent=114"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}