Multi-timescale representation learning in LSTM Language Models

Simple item page
dc.contributor.advisorHuth, Alexander G.
dc.creatorMahto, Shivangi
dc.date.accessioned2022-11-21T22:57:51Z
dc.date.available2022-11-21T22:57:51Z
dc.date.created2020-05
dc.date.issued2022-10-06
dc.date.submittedMay 2020
dc.date.updated2022-11-21T22:57:52Z
dc.description.abstractRepresentations within Language Models (LMs) are difficult to interpret. For example, how different layers of an LSTM LM retain information over different periods of time is unclear. In this paper, we present methods to interpret and control the timescale of information routing through an LSTM unit. We found out that a standard LSTM LM favors representations of small timescale information (up to 20 tokens). We then introduce a prior based on statistical properties of natural language, which is applied on the distribution of timescale across LSTM units to achieve an effective multi-timescale LM. The proposed model learns representations of both short as well long timescale. It also achieves better prediction performance than a standard LSTM LM on Penn Treebank and WikiText-2 datasets, especially on rare words.
dc.description.departmentComputer Science
dc.format.mimetypeapplication/pdf
dc.identifier.urihttps://hdl.handle.net/2152/116770
dc.identifier.urihttp://dx.doi.org/10.26153/tsw/43665
dc.language.isoen
dc.subjectLanguage Models
dc.subjectTimescale learning
dc.subjectLSTMs
dc.subjectNatural language processing
dc.subjectMachine Learning
dc.titleMulti-timescale representation learning in LSTM Language Models
dc.typeThesis
dc.type.materialtext
thesis.degree.departmentComputer Sciences
thesis.degree.disciplineComputer Science
thesis.degree.grantorThe University of Texas at Austin
thesis.degree.levelMasters
thesis.degree.nameMaster of Science in Computer Sciences

Access full-text files

Original bundle

Now showing 1 - 1 of 1
Thumbnail Image
Name:
MAHTO-THESIS-2020.pdf
Size:
765.63 KB
Format:
Adobe Portable Document Format
Download

License bundle

Now showing 1 - 2 of 2
No Thumbnail Available
Name:
PROQUEST_LICENSE.txt
Size:
4.45 KB
Format:
Plain Text
Description:
Download
No Thumbnail Available
Name:
LICENSE.txt
Size:
1.84 KB
Format:
Plain Text
Description:
Download

Collections

UT Electronic Theses and Dissertations