Following natural language route instructions


Following natural language route instructions

Show full record

Title: Following natural language route instructions
Author: MacMahon, Matthew Tierney
Abstract: Following natural language instructions requires transforming language into situated conditional procedures; robustly following instructions, despite the director's natural mistakes and omissions, requires the pragmatic combination of language, action, and domain knowledge. This dissertation demonstrates a software agent that parses, models and executes human-written natural language instructions to accomplish complex navigation tasks. We compare the performance against people following the same instructions. By selectively removing various syntactic, semantic, and pragmatic abilities, this work empirically measures how often these abilities are necessary to correctly navigate along extended routes through unknown, large-scale environments to novel destinations. To study how route instructions are written and followed, this work presents a new corpus of 1520 free-form instructions from 30 directors for 252 routes in three virtual environments. 101 other people followed these instructions and rated them for quality, successfully reaching and identifying the destination on only approximately two-thirds of the trials. Our software agent, MARCO, followed the same instructions in the same environments with a success rate approaching human levels. Overall, instructions subjectively rated 4 or better of 6 comprise just over half of the corpus; MARCO performs at 88% of human performance on these instructions. MARCO's performance was a strong predictor of human performance and ratings of individual instructions. Ablation experiments demonstrate that implicit procedures are crucial for following verbal instructions using an approach integrating language, knowledge and action. Other experiments measure the performance impact of linguistic, execution, and spatial abilities in successfully following natural language route instructions.
Department: Electrical and Computer Engineering
Subject: Natural language processing (Computer science)--Computer programs Computational linguistics--Computer programs
Date: 2007-08

Files in this work

Description: Supplement (TGZ File)
Size: 7.976Mb
Format: application/octet-stream
Download File: macmahonm82386.pdf
Size: 3.909Mb
Format: application/pdf

This work appears in the following Collection(s)

Show full record

Advanced Search


My Account