Browsing by Subject "Reproducibility"

Now showing 1 - 3 of 3

Kubernetes provenance
(2020-09-14) Lin, William, M.S. in Computer Sciences; Chidambaram, Vijay; Rossbach, Christopher J.
The field of machine learning (ML) has experienced a period of renaissance since the 2000s. First, exponential increase in computational power and improvements in hardware has finally allowed machine learning algorithms to process the same amount of data in minutes and hours rather than hundreds of years. Second, the model of cloud computing made large scale clusters inexpensive and available to anyone at the click of a button, allowing them to scale their algorithms without having to personally maintain hundreds or even thousands of machines. However, despite the huge rise in popularity of machine learning in both research and industry, the ML community is facing a crisis of being able to reproduce results. Although the existing machine learning frameworks all have the ability to re-execute the same piece of code saved by a researcher, the typical workflow could involve different frameworks and accesses to data on remote machines. These cross-framework workflows can not be replicated by a single frameworks provenance system, and often contain customized scripts and processes that can further obscure the ability for future replication and repeatability. I make the argument in this thesis that because of machine learning’s need for scale and frequent training on large clusters, Kubernetes serves as a good common layer for the systems community to interpose a layer of provenance collection to aid the ML community in reproducing results that make use of multiple machines, frameworks, and hardware platforms. In addition, I also propose two new mechanisms for collecting fine-grained provenance information from Kubernetes without modifying the application or host operating system.
Nondeterminism as a reproducibility challenge for deep reinforcement learning
(2018-08) Nagarajan, Prabhat Mahadevan; Stone, Peter, 1971-
In recent years, deep neural networks have powered many successes in deep reinforcement learning (DRL) and artificial intelligence by serving as effective function approximators in high-dimensional domains. However, there are several difficulties in reproducing such successes. These difficulties have risen due to several factors, including researchers' limited access to compute power and a general lack of knowledge of implementation details that are critical for reproducing results successfully. However, nondeterminism is a reproducibility challenge that is perhaps less emphasized despite being particularly relevant in DRL. DRL algorithms tend to have high variance, in no small part due to the fact that agents must learn from a nonstationary training distribution in the presence of additional sources of randomness that are absent from other machine learning paradigms. The high variance of DRL algorithms, combined with the low sample sizes used in research, makes it difficult to match reported results. As such, the ability to control for sources of nondeterminism is especially important for achieving reproducibility in DRL. If we are to maximize progress in DRL, we need research to be reproducible and verifiable, ensuring the validity of our claims. Reproducibility is a necessary prerequisite for improving upon or comparing algorithms, both of which are done frequently in DRL research. In this thesis, we take steps towards studying the impact of nondeterminism on two important pillars of DRL research: the reproducibility of results and the statistical comparison of algorithms. We do so by (1) enabling deterministic training in DRL by identifying and controlling for all sources of nondeterminism present during training, (2) performing a sensitivity analysis that shows how these sources of nondeterminism can impact a DRL agent's performance and policy, and (3) showing how nondeterminism negatively impacts algorithm comparison in DRL and describing how deterministic training can mitigate this negative impact. We find that individual sources of nondeterminism such as the random network initialization can affect an agent's performance substantially. We also find that the current sample sizes used in DRL may not satisfactorily capture differences in performance between two algorithms. Lastly, we make available our deterministic implementation of deep Q-learning.
Superior longitudinal fasciculus microstructure and its functional triple-network mechanisms in depressive rumination
(2018-12) Pisner, Derek Alexander; Schnyer, David M.; Beevers, Christopher
Depressive rumination, which involves a repetitive focus on one's distress, is associated with function connectivity disturbances of Default-Mode, Salience, and Executive-Control networks, comprising the so-called "triple-network" of attention. Missing, however, is a multimodal account of rumination that neuroanatomically explains the perseveration of these dysfunctional networks as a stable human trait. Using diffusion and functional Magnetic Resonance Imaging, we explored multimodal relationships between rumination severity, white-matter microstructure, and resting-state functional connectivity in N=39 depressed adults, and then directly replicated our findings in a demographically-matched, independent sample (N=39). Among the fully-replicated results, three core findings emerged. First, rumination severity is associated with both disintegrated and desegregated functional connectivity of the triple-network. Second, global microstructural inefficiency of the right Superior Longitudinal Fasciculus (SLF) provides a neuroanatomical connectivity basis for rumination and accounts for anywhere between 25-37% of the variance in rumination (Discovery: p corr<0.01; Replication: p corr<0.01; MSE=0.05). Finally, microstructure of the right SLF and auxiliary white-matter is strongly associated with functional connectivity biomarkers of rumination, both within and between components of the triple-network (Discovery: R²=0.36, p corr<0.05; Replication: R²=0.25, p corr<0.05; MSE=0.04-0.06). By cross-validating discovery with replication, our findings advance a reproducible microstructural-functional brain connectivity model of depressive rumination that unifies neurodevelopmental and neurocognitive perspectives.