{"id":37,"date":"2018-03-19T09:34:03","date_gmt":"2018-03-19T09:34:03","guid":{"rendered":"http:\/\/sag.art.uniroma2.it\/absita\/?page_id=37"},"modified":"2018-09-30T15:55:56","modified_gmt":"2018-09-30T15:55:56","slug":"data","status":"publish","type":"page","link":"http:\/\/sag.art.uniroma2.it\/absita\/data\/","title":{"rendered":"Data"},"content":{"rendered":"<p style=\"font-size: 16px; text-align: justify;\">The data source chosen for creating the datasets is the popular website <a href=\"https:\/\/www.booking.com\">booking.com<\/a>. The platform allows users to share their opinions about hotels through a positive\/negative textual review and a fine-grain rating system that assigns a score to\u00a0 different aspects: <strong>cleanliness, comfort, amenities, staff, value for money, free\/paid WiFi, location<\/strong>. The websitedetailed on the s provides a large number of reviews in many languages, including Italian.<\/p>\n<p><img loading=\"lazy\" class=\"alignnone wp-image-57\" src=\"http:\/\/sag.art.uniroma2.it\/absita\/wp-content\/uploads\/2018\/03\/booking-300x171.png\" alt=\"\" width=\"391\" height=\"223\" srcset=\"http:\/\/sag.art.uniroma2.it\/absita\/wp-content\/uploads\/2018\/03\/booking-300x171.png 300w, http:\/\/sag.art.uniroma2.it\/absita\/wp-content\/uploads\/2018\/03\/booking-768x437.png 768w, http:\/\/sag.art.uniroma2.it\/absita\/wp-content\/uploads\/2018\/03\/booking-1024x583.png 1024w, http:\/\/sag.art.uniroma2.it\/absita\/wp-content\/uploads\/2018\/03\/booking.png 1694w\" sizes=\"(max-width: 391px) 100vw, 391px\" \/><\/p>\n<p style=\"font-size: 16px; text-align: justify;\">More than <strong>10,000 sentences<\/strong> from hotel reviews in Italian\u00a0have been extracted using the Python implementation of <a href=\"https:\/\/scrapy.org\/\">Scrapy<\/a>, an open source framework to extract data from websites.\u00a0The hotel&#8217;s page situated in Naples, Bologna, Milan have been analyzed and the reviews which contain textual contents have been extracted. The longer reviews have been split into single self-reliant sentences using <a href=\"http:\/\/tint.fbk.eu\/\">Tint (The Italian NLP tool) <\/a>library.<\/p>\n<p style=\"font-size: 16px; text-align: justify;\">In order to guarantee a balanced distribution of positive and negative sentences in the portion of the dataset reserved for each annotator, we randomly selected the sentences alternating positive and negative annotated phrases, leveraging the review-level information already provided by <span style=\"color: #0000ff;\"><i><a href=\"http:\/\/booking.com\/\" target=\"_blank\" rel=\"noopener\">booking.com<\/a> <\/i><\/span><\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n<h2><span style=\"font-size: 26px; font-weight: bold;\">Data Annotation Process<\/span><\/h2>\n<p style=\"font-size: 16px; text-align: justify;\">In order to obtain a complete dataset for ABSA, we annotated the sentences from the hotel reviews according to seven aspects: <em>pulizia<\/em> (cleanliness), <em>comfort<\/em>, <em>servizi<\/em> (amenities), <em>staff<\/em>, <em>qualita-prezzo<\/em> (value), <em>wifi<\/em> (wireless Internet connection) and <em>posizione<\/em> (location). For each aspect, the polarity (positive, negative) of its mention has been annotated. The positive and negative polarities are annotated independently, thus for each aspect four sentiment classes are possible: <em>positive<\/em> (positive=yes, negative=no), <em>negative<\/em> (positive=no, negative=yes),\u00a0<em>neutral<\/em> (positive=no, negative=no),\u00a0<em>mixed<\/em> (positive=yes, negative=yes).<\/p>\n<p style=\"font-size: 16px; text-align: justify;\">The annotation task involved the following classes:<\/p>\n<ul style=\"font-size: 16px; text-align: justify;\">\n<li><i>cleanliness_positive<\/i><\/li>\n<li><i>cleanliness_negative<\/i><\/li>\n<li><i>comfort_positive<\/i><\/li>\n<li><i>comfort_negative<\/i><\/li>\n<li><i>amenities_positive<\/i><\/li>\n<li><i>amenities_negative<\/i><\/li>\n<li><i>staff_positive<\/i><\/li>\n<li><i>staff_negative<\/i><\/li>\n<li><i>value_positive<\/i><\/li>\n<li><i>value_negative<\/i><\/li>\n<li><i>wifi_positive<\/i><\/li>\n<li><i>wifi_negative<\/i><\/li>\n<li><i>location_positive<\/i><\/li>\n<li><i>location_negative<\/i><\/li>\n<li><b><i>other_positive<\/i><br \/>\n<\/b><\/li>\n<li><i><b>other_negative<\/b><\/i><\/li>\n<\/ul>\n<p style=\"font-size: 16px; text-align: justify;\">Please note that the special topic &#8220;other&#8221; has been added for completeness, to annotate sentences with opinions on aspects not among the seven\u00a0 considered by the task. <b>The aspect &#8220;other&#8221; is provided additionally and it will be not part of the evaluation of results provided for the task.<\/b><\/p>\n<p style=\"font-size: 16px; text-align: justify;\"><b>Four different annotators<\/b> have been involved in the task in parallel, i.e., dividing the dataset into four equal subsets\u00a0of about<b>~ 2500 sentences<\/b>\u00a0each. Furthermore, a portion of the dataset made of 250 sentences was annotated by all four annotators. We computed the calculated the inter-annotator agreement per-class\u00a0 over this subset and reported a\u00a0variation between 85% and 100% (percentage of sentences for which all annotators agreed).<\/p>\n<p style=\"font-size: 16px; text-align: justify;\"><i>Incomplete, irrelevant, and incomprehensible sentences have been discarded from the dataset during the annotation.<\/i><\/p>\n<p style=\"font-size: 16px; text-align: justify;\"><i>Finally, the <em>*_presence<\/em> field for the ACD task is computed as the logic inclusive OR of the respective <em>*_positive<\/em> and <em>*_negative<\/em> fields.<\/i><\/p>\n<h2><span style=\"font-size: 26px; font-weight: bold;\">Data Format<\/span><\/h2>\n<p style=\"font-size: 16px; text-align: justify;\">The data format used is <b>CSV with UTF-8 encoding and semicolon as separator<\/b>. Using this annotation, it is possible to keep the information about which are sentences observed in the same review structure. It is important to note that in booking.com the order of positive and negative sentences is strictly defined, and this could influence the learning. To overcome this issue, we shuffled the sentences in each review. As a consequence, the final id showed in the data file will not reflect the real order of the sentences in the review.<br \/>\n<b>The text of the sentence will be provided at the end of the row and delimited by <em>&#8220;<\/em><\/b>. Moreover, three binary values are provided for each aspect indicating, respectively: the presence of the aspect in the sentence (aspect\\presence:0\/1), the positive polarity for that aspect (aspect\\pos:0\/1) and finally the negative polarity (aspect\\neg:0\/1). An example of the annotated dataset in the proposed format:<\/p>\n<pre style=\"font-size: 13px;\"><strong>id_sentence; aspect1_presence; aspect1_pos; aspect1_neg; ... sentence;<\/strong>\r\n20160624;0;0;0;0;0;0;0;0;0;0;0;0;1;1;0;0;0;0;1;1;0;\"Considerato il prezzo e per una sola notte, va benissimo come punto di appoggio.\"\r\n20160625;1;0;1;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;\"Almeno i servizi igienici andrebbero rivisti e migliorati nella pulizia.\"\r\n20160626;0;0;0;1;0;1;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;\"La struttura purtroppo \u00e8 vecchia e ci vorrebbero dei lavori di ammodernamento.\"<\/pre>\n<h2>Download<\/h2>\n<ul>\n<li style=\"font-size: 16px;\">30\/5\/2018:\u00a0<b>trial<\/b>\u00a0data are now available for download: <a href=\"http:\/\/sag.art.uniroma2.it\/absita\/wp-content\/uploads\/2018\/05\/trial_30.csv\">trial_30.csv<\/a><\/li>\n<li>4\/6\/2018: <strong>training<\/strong> data are now available for download:\u00a0<a href=\"http:\/\/sag.art.uniroma2.it\/absita\/wp-content\/uploads\/2018\/06\/absita_2018_training.csv\">absita_2018_training.csv<\/a><\/li>\n<li>10\/9\/2018: <strong>test<\/strong> data are now available for download:\u00a0<a href=\"http:\/\/sag.art.uniroma2.it\/absita\/wp-content\/uploads\/2018\/09\/absita_2018_test.zip\">absita_2018_test.zip<\/a><\/li>\n<li>30\/9\/2018: <strong>gold standard test set<\/strong> and <strong>evaluation script<\/strong> are now available for download:\u00a0<a href=\"http:\/\/sag.art.uniroma2.it\/absita\/wp-content\/uploads\/2018\/09\/absita_2018_evaluation.zip\">absita_2018_evaluation.zip<\/a><\/li>\n<\/ul>\n<h2>Results<\/h2>\n<p>&nbsp;<\/p>\n<p>Task ACD: Aspect Category Detection<\/p>\n<p>rankMicro-PrecisionMicro-RecallMicro-F1-score<\/p>\n<table>\n<thead><\/thead>\n<tbody>\n<tr>\n<td>1<\/td>\n<td>0.8397<\/td>\n<td>0.7837<\/td>\n<td>0.8108<\/td>\n<\/tr>\n<tr>\n<td>2<\/td>\n<td>0.8713<\/td>\n<td>0.7504<\/td>\n<td>0.8063<\/td>\n<\/tr>\n<tr>\n<td>3<\/td>\n<td>0.8697<\/td>\n<td>0.7481<\/td>\n<td>0.8043<\/td>\n<\/tr>\n<tr>\n<td>4<\/td>\n<td>\u00a00.8626<\/td>\n<td>0.7519<\/td>\n<td>0.8035<\/td>\n<\/tr>\n<tr>\n<td>5<\/td>\n<td>0.8819<\/td>\n<td>0.7378<\/td>\n<td>0.8035<\/td>\n<\/tr>\n<tr>\n<td>6<\/td>\n<td>0.898<\/td>\n<td>0.6937<\/td>\n<td>0.7827<\/td>\n<\/tr>\n<tr>\n<td>7<\/td>\n<td>0.8658<\/td>\n<td>0.697<\/td>\n<td>0.7723<\/td>\n<\/tr>\n<tr>\n<td>8<\/td>\n<td>0.7902<\/td>\n<td>0.7181<\/td>\n<td>0.7524<\/td>\n<\/tr>\n<tr>\n<td>9<\/td>\n<td>0.6232<\/td>\n<td>0.6093<\/td>\n<td>0.6162<\/td>\n<\/tr>\n<tr>\n<td>10<\/td>\n<td>0.6164<\/td>\n<td>0.6134<\/td>\n<td>0.6149<\/td>\n<\/tr>\n<tr>\n<td>11<\/td>\n<td>0.5443<\/td>\n<td>0.5418<\/td>\n<td>0.5431<\/td>\n<\/tr>\n<tr>\n<td>12<\/td>\n<td>0.6213<\/td>\n<td>0.433<\/td>\n<td>0.5104<\/td>\n<\/tr>\n<tr>\n<td>baseline<\/td>\n<td>0.4111<\/td>\n<td>0.2866<\/td>\n<td>0.3377<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>Task ACP: Aspect Category Polarity<\/p>\n<p>rankMicro-PrecisionMicro-RecallMicro-F1-score<\/p>\n<table>\n<thead><\/thead>\n<tbody>\n<tr>\n<td>1<\/td>\n<td>0.8264<\/td>\n<td>0.7161<\/td>\n<td>0.7673<\/td>\n<\/tr>\n<tr>\n<td>2<\/td>\n<td>0.8612<\/td>\n<td>0.6562<\/td>\n<td>0.7449<\/td>\n<\/tr>\n<tr>\n<td>3<\/td>\n<td>0.7472<\/td>\n<td>0.7186<\/td>\n<td>0.7326<\/td>\n<\/tr>\n<tr>\n<td>4<\/td>\n<td>0.7387<\/td>\n<td>0.7206<\/td>\n<td>0.7295<\/td>\n<\/tr>\n<tr>\n<td>5<\/td>\n<td>0.8735<\/td>\n<td>0.5649<\/td>\n<td>0.6861<\/td>\n<\/tr>\n<tr>\n<td>6<\/td>\n<td>0.6869<\/td>\n<td>0.5409<\/td>\n<td>0.6052<\/td>\n<\/tr>\n<tr>\n<td>7<\/td>\n<td>0.4123<\/td>\n<td>0.3125<\/td>\n<td>0.3555<\/td>\n<\/tr>\n<tr>\n<td>8<\/td>\n<td>0.5452<\/td>\n<td>0.2511<\/td>\n<td>0.3439<\/td>\n<\/tr>\n<tr>\n<td>baseline<\/td>\n<td>0.2451<\/td>\n<td>0.1681<\/td>\n<td>0.1994<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n","protected":false},"excerpt":{"rendered":"<p>The data source chosen for creating the datasets is the popular website booking.com. The platform allows users to share their opinions about hotels through a positive\/negative textual review and a fine-grain rating system that assigns a score to\u00a0 different aspects: cleanliness, comfort, amenities, staff, value for money, free\/paid WiFi, location. The websitedetailed on the s &hellip; <a href=\"http:\/\/sag.art.uniroma2.it\/absita\/data\/\" class=\"more-link\">Continue reading <span class=\"screen-reader-text\">Data<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":[],"_links":{"self":[{"href":"http:\/\/sag.art.uniroma2.it\/absita\/wp-json\/wp\/v2\/pages\/37"}],"collection":[{"href":"http:\/\/sag.art.uniroma2.it\/absita\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"http:\/\/sag.art.uniroma2.it\/absita\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"http:\/\/sag.art.uniroma2.it\/absita\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/sag.art.uniroma2.it\/absita\/wp-json\/wp\/v2\/comments?post=37"}],"version-history":[{"count":36,"href":"http:\/\/sag.art.uniroma2.it\/absita\/wp-json\/wp\/v2\/pages\/37\/revisions"}],"predecessor-version":[{"id":170,"href":"http:\/\/sag.art.uniroma2.it\/absita\/wp-json\/wp\/v2\/pages\/37\/revisions\/170"}],"wp:attachment":[{"href":"http:\/\/sag.art.uniroma2.it\/absita\/wp-json\/wp\/v2\/media?parent=37"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}