Exploring the relationship between Deep Neural Networks and Neural Tangent Kernel via covariance manipulation

In this study, we examine the relationship between Deep Neural Networks (DNNs) and the Neural Tangent Kernel (NTK), with a particular emphasis on covariance manipulation. Using the foundational framework provided by Roberts and Yaida in their book, "Deep Learning Theory," we methodically explore the impact of introducing a Diagonal matrix [upsih] on covariance. Our work results in the derivation of equations that detail the variance of the stochastic second layer NTK of the preactivations and the variance of the fluctuations in the stochastic second layer metric of preactivations. Through numerical analysis, we compare these derived variances. Our findings, while preliminary, suggest a noteworthy observation regarding the possible relationship between DNNs and NTK. This study serves as a stepping stone to further research in the realm of deep learning theory.