End-to-end communication between humans and robots in natural language is challenging for the deep interaction of different cognitive abilities. For a robot to react to a user command like “take the book on the table”, a number of implicit assumptions should be met. First, at least two entities, a book and a table, must exist in the environment and the speaker must be aware of such entities. Accordingly, the robot must have access to an inner representation of the objects, e.g., an explicit map of the environment. Second, mappings from lexical references to real world entities must be developed or made available. Grounding here links symbols (e.g., words) to the corresponding perceptual information. Spoken Language Understanding (SLU) for interactive dialogue systems acquires a specific nature, when applied in Interactive Robotics.

In (Bastianelli et al, 2016), a spoken language understanding (SLU) process that integrates perceptual and linguistic information has been proposed. This process allows to produce command interpretations that coherently express constraints about the world (including the selection of all the involved entities), the Robotic Platform (with all its inner representations and capabilities) and the pure linguistic level. A discriminative approach based on the Markovian formulation of Support Vector Machines, known as SVM-HMM, is adopted, where grounded information is directly injected within the learning algorithm, showing that the integration of linguistic and perceptual knowledge improves the quality and robustness of the overall interpretation process.

Back to Top

---

 

The outlined SLU process represents the engine of LU4R. It has the goal of producing an interpretation of an user utterance in terms of Frame Semantics, that guarantees a linguistic and cognitive basis to the interpretation. Specifically, we consider the formalization adopted in the FrameNet database. According to Framenet, actions expressed in user utterances can be modeled as semantic frames. These are micro-theories about real world situations, e.g. the action of taking. Each frame specifies also the set of participating entities, called frame elements, e.g., the THEME representing the object that is taken during the Taking action. For example, for the sentence

whose corresponding parsed version can be

In this section we introduce all the packages required to run LU4R and carry out its deployment on a robotic platform.

LU4R is released along with the following packages:

• LU4R
• LU4R is released as a single executable jar executable. This package contains the LU4R system, pre-trained models in English for the SLU process and configuration files. It provides REST services to interact with. The dataset is composed of 527 sentences and has been used in the empirical investigation in (Bastianelli et al, 2016).
• Download the latest version (Version 0.2.1). Requires Java 1.8+.
• LU4R ROS interface
• This package contains the ROS node. It is an orchestrator between the robot, the Android application and LU4R. It communicates with the Android application, acting as a Server. Once the list of transcriptions is received from the Android app, it is forwarded to LU4R.
• Checkout the code from https://github.com/andreavanzo/lu4r_ros_interface (Version 0.1.0)
• LU4R Android app
• It is a simple Android application that allows to: (i) send to the ROS node the list of transcriptions for a given spoken command (through the Google Speech APIs) and (ii) move the robotic platform through a virtual joypad (the ROS node publishes on /cmd_vel ).
• Download the latest version (Version 0.1.1)

If you are in trouble with LU4R and you are interested in receiving support, write an email to croce@info.uniroma2.it or vanzo@dis.uniroma1.it.

In this section, a short tutorial for the complete usage of LU4R is presented.

Functional-wise, LU4R is a service that can be invoked through HTTP communication. Its implementation is realized through a server that keeps listening to natural language sentences and outputs an interpretation of them. The usage of LU4R passes through two phases: the startup phase, where the process is run and initialized, and a service phase, where LU4R is ready to receive requests. In order to start playing with LU4R, you need to initialize it. Launching the service is quite simple. In fact, it can be performed through the following command line:

java -jar -Xmx1G lu4r-x.x.x.jar [type] [output] [lang] [port]

where:

• [type] is the LU4R operating mode, basic or simple. The basic setting does not contemplate perceptual knowledge during the interpretation process. Conversely, the simple configuration relies on perceptual information, enabling a context-sensitive interpretation of the command at the predicate level.
• [output] is the preferred output format of the interpretation. The available formats are:
• xml: this is the default output format. The interpretation is given in the XDG format (eXtended Dependency Graph) and XML compliant container (see Basili and Zanzotto, 2002). Notice that the LU4R ROS interface features methods for easy extraction of the interpreted predicates directly from an XDG.
• amr: the interpretation is given in the Abstract Meaning Representation.
• conll: it is a CoNLL-like tabular representation of the predicates found.
• [lang]: the operating language of LU4R. At the moment, only the en (English) version is supported.
• [port]: the listening port of LU4R.

The startup process is completed once the message

Server launched: listening on port [port]

is prompted.

For example, the command

java -Xmx1G lu4r-0.1.0.jar simple amr en 9090

launches the perceptual chain ( simple ) in English ( en ) on the port 9090 and returns the interpretation in AMR format ( amr ).

Once the service has been initialized, it is possible to start asking for interpreting user utterances. The server thus waits for messages carrying the utterance transcriptions to be parsed. Each sentence here corresponds to a speech recognition hypothesis. Hence, it can be paired with the corresponding transcription confidence score, useful in the re-ranking phase. The body of the message must then contain the list of hypotheses encoded as a JSON array called hypotheses , where each entry is a transcription paired with a confidence according to the following syntax:

{"hypotheses":[
		{"transcription":"take the book on the table",
		 "confidence":"0.9",
		 "rank":"1"},
		...,
		{"transcription":"did the book on the table",
		 "confidence":"0.2",
		 "rank":"5"}
		 ]
}

where:

• transcription is the text corresponding to the spoken utterance;
• confidence is the confidence value provided by the ASR;
• rank is the ranking position among the hypotheses list.

Additionally, when the simple configuration is selected, it requires the list of entities populating the environment the robot is operating into, in order to enable in order to enable the interpretation process that depends on the environment. Notice that if the entity list is empty, the simple SLU process operates as the basic one. This additional information must be passed as the entities parameter in the following JSON format:

{"entities":[
		{"atom":"book1",
		 "type":"book",
		 "preferredLexicalReference":"book",
		 "alternativeLexicalReferences":["volume","manual",...],
		 "coordinate":{
		  "x":"13.0",
		  "y":"6.5",
		  "z":"3.5",
		  "angle":"3.5"}},
		...
		{"atom":"table1",
		 "type":"table",
		 "preferredLexicalReference":"table",
		 "alternativeLexicalReferences":["bar","counter","desk","board"],
		 "coordinate":{
		  "x":"12.0",
		  "y":"8.5",
		  "z":"0.0",
		  "angle":"1.6"}}
		 ]
}

where:

• atom is a unique identifier of the entity;
• type represents the class of the entity (e.g. book, table,...);
• preferredLexicalReference is the preferred word used to refer to that particular entity;
• alternativeLexicalReference is a list of alternative words used to refer to the entity;
• coordinate represents the position the entity, in terms of planar coordinates (`x`,`y`), elevation (`z`) and `angle` is the orientation.

The above parameters are described in details in the README file provided with LU4R.

The service can be invoked with a HTTP POST request that puts together the hypo and entities JSONs as follows:

http://IP_ADDRESS:PORT/service/nlu
POST parameters: hypo={"hypotheses":[...]}
		 entities={"entities":[...]}

Additional examples, that show how to invoke the chain from a BASH command-line, are reported and described in the README file provided with the chain.

The LU4R ROS interface is in charge of receiving the list of transcriptions from an external ASR (e.g., the LU4R Android app, here) and forward it to LU4R. In order to launch the node, you need ROS in your Linux machine. Once the ROS installation is completed, place the provided ROS package lu4r_ros_interface in your catkin_ws/src folder. Then, run

roscore

to start the ROS master. The ROS node can be finally launched through the following command:

rosrun lu4r_ros_interface android_interface _port:=[port] _lu4r_ip:=[lu4r_ip_address] _lu4r_port:=[lu4r_port] _semantic_map:=[semantic_map]

where:

• _port: the listening port of the LU4R ROS interface. This is required by the Android app, enabling a TCP connection between them.
• _lu4r_ip_address: the ip address of LU4R. If the LU4R and the LU4R ROS interface are on the same machine, ignore this argument.
• _lu4r_port: the listening port of LU4R.
• _semantic_map: the semantic map to be employed, among the ones available into semantic_maps. The semantic maps are in JSON format, and represent the configuration of the environment (e.g., objects, locations,...) in which the robot is operating. Whenever a simple configuration of LU4R is chosen, the interpretation process is sensitive to different semantic maps.

The Android app enable the ASR through the Google Speech APIs. It is provided as an Android application package (APK) file, to be installed on an Android smartphone or tablet (requires Android 5+).

In order to enable a TCP connection between LU4R Android app and LU4R ROS node, the smartphone hosting the app and the laptop running the ROS node must be in the same subnet (e.g. connected to the same WiFi network). Moreover, LU4R Android app requires an internet connection, in order to retrieve the transcription through the Google ASR. The app needs to be configured, by modifying the settings accessible from the three-dots in the top-right corner:

• Connection: everything concerning the connection parameters. The IP address must be the same assigned to the machine running LU4R ROS node. The port is the one set up for the LU4R ROS node server.
• Joypad: everything concerning the virtual joypad configuration.
• Speech-To-Text: configuration of the ASR engine.
• Text-To-Speech: configuration of the TTS engine.

In the home screen, a right-swipe open the menu, showing the different operating modalities. Speech Interface is the panel in charge of sending the hypothesized transcription to LU4R ROS node.