Show simple item record

dc.contributor.advisorGhosh, Joydeepen
dc.creatorDaruru, Srivatsavaen
dc.date.accessioned2010-12-20T20:40:40Zen
dc.date.accessioned2010-12-20T20:40:46Zen
dc.date.available2010-12-20T20:40:40Zen
dc.date.available2010-12-20T20:40:46Zen
dc.date.issued2010-08en
dc.date.submittedAugust 2010en
dc.identifier.urihttp://hdl.handle.net/2152/ETD-UT-2010-08-1838en
dc.descriptiontexten
dc.description.abstractThe unprecedented and exponential growth of data along with the advent of multi-core processors has triggered a massive paradigm shift from traditional single threaded programming to parallel programming. A number of parallel programming paradigms have thus been proposed and have become pervasive and inseparable from any large production environment. Also with the massive amounts of data available and with the ever increasing business need to process and analyze this data quickly at the minimum cost, there is much more demand for implementing fast data mining algorithms on cheap hardware. This thesis explores a parallel programming model called dataflow, the essence of which is computation organized by the flow of data through a graph of operators. This paradigm exhibits pipeline, horizontal and vertical parallelism and requires only the data of the active operators in memory at any given time allowing it to scale easily to very large datasets. The thesis describes the dataflow implementation of two data mining applications on huge datasets. We first develop an efficient dataflow implementation of a Collaborative Filtering (CF) algorithm based on weighted co-clustering and test its effectiveness on a large and sparse Netflix data. This implementation of the recommender system was able to rapidly train and predict over 100 million ratings within 17 minutes on a commodity multi-core machine. We then describe a dataflow implementation of a non-parametric density based clustering algorithm called Auto-HDS to automatically detect small and dense clusters on a massive astronomy dataset. This implementation was able to discover dense clusters at varying density thresholds and generate a compact cluster hierarchy on 100k points in less than 1.3 hours. We also show its ability to scale to millions of points as we increase the number of available resources. Our experimental results illustrate the ability of this model to “scale” well to massive datasets and its ability to rapidly discover useful patterns in two different applications.en
dc.format.mimetypeapplication/pdfen
dc.language.isoengen
dc.subjectDataflow processingen
dc.subjectData miningen
dc.subjectDistributed computingen
dc.subjectLarge scale data miningen
dc.subjectParallel processingen
dc.titleDataflow parallelism for large scale data miningen
dc.date.updated2010-12-20T20:40:46Zen
dc.contributor.committeeMemberMarin, Nenaen
dc.description.departmentComputer Sciencesen
dc.type.genrethesisen
thesis.degree.departmentComputer Sciencesen
thesis.degree.disciplineComputer Sciencesen
thesis.degree.grantorUniversity of Texas at Austinen
thesis.degree.levelMastersen
thesis.degree.nameMaster of Science in Computer Sciencesen


Files in this item

Icon

This item appears in the following Collection(s)

Show simple item record