Principal Component Analysis (PCA) is a fundamental pillar of modern data pipelines, but its traditional implementation is woefully inadequate for modern data problems. In this work we present our contributions to the field of streaming principal component analysis---research that adds critical flexibility to one of the most common tools in optimization. This includes a practical new algorithm--AdaOja--which we outline in the context of both streaming principal component analysis and streaming kernel principal component analysis. We also present new mathematical theory inspired by our study of theoretical convergence for this algorithm. Streaming principal component analysis and streaming kernel principal component analysis can be seamlessly integrated into many applications with significant improvements in scalability. We specifically demonstrate the considerable improvements that can be achieved by applying streaming KPCA to kernel analog forecasting (KAF) for dynamical systems.