Understanding & predicting the skills needed to answer a visual question

Simple item page
dc.contributor.advisorGurari, Danna
dc.contributor.committeeMemberKaradkar, Unmil P
dc.creatorZeng, Xiaoyu, M.S. in Information Studies
dc.date.accessioned2019-09-16T20:23:35Z
dc.date.available2019-09-16T20:23:35Z
dc.date.created2019-05
dc.date.issued2019-06-20
dc.date.submittedMay 2019
dc.date.updated2019-09-16T20:23:35Z
dc.description.abstractWe proposed a method to automatically identify the relevant cognitive skills to perform a visual question answering (VQA) task. Focusing on a subset of VizWiz 1.0 and VQA 2.0 data, we collected labels for five skill categories, extracted multimodal features from images and their corresponding question, and trained a recurrent neural network with LSTM encoders to perform binary multi-label classification for the two main cognitive skills to answer a visual question: text recognition and color recognition. Our results demonstrated the potential of using a skill predictor to improve current visual question answering frameworks. This project made two main contributions. First, we provided an in-depth analysis of our data and highlighted the fact that, as a popular traditional benchmark dataset, VQA 2.0 cannot sufficiently model the information needs of visually impaired users in the context of visual question answering for accessibility. Secondly, we developed a skill prediction algorithm that can potentially help to label and route tasks for automated or human-in-the-loop systems of vision-assistive technologies
dc.description.departmentInformation
dc.format.mimetypeapplication/pdf
dc.identifier.urihttps://hdl.handle.net/2152/75855
dc.identifier.urihttp://dx.doi.org/10.26153/tsw/2957
dc.language.isoen
dc.subjectVisual question answering
dc.subjectMultimodal machine learning
dc.subjectAccessibility
dc.titleUnderstanding & predicting the skills needed to answer a visual question
dc.title.alternativeUnderstanding and predicting the skills needed to answer a visual question
dc.typeThesis
dc.type.materialtext
thesis.degree.departmentInformation
thesis.degree.disciplineInformation Studies
thesis.degree.grantorThe University of Texas at Austin
thesis.degree.levelMasters
thesis.degree.nameMaster of Science in Information Studies

Access full-text files

Original bundle

Now showing 1 - 1 of 1
Thumbnail Image
Name:
ZENG-MASTERSREPORT-2019.pdf
Size:
5.51 MB
Format:
Adobe Portable Document Format
Download

License bundle

Now showing 1 - 2 of 2
No Thumbnail Available
Name:
PROQUEST_LICENSE.txt
Size:
4.45 KB
Format:
Plain Text
Description:
Download
No Thumbnail Available
Name:
LICENSE.txt
Size:
1.84 KB
Format:
Plain Text
Description:
Download

Collections

UT Electronic Theses and Dissertations