Evaluation of the LBJ School’s graduate-level math readiness program

Graduate-level policy students come to the discipline with a diverse spectrum of experience – some with a strong math background and others without. The main work of this study is to provide insights on the LBJ School of Public Affair’s math readiness program for incoming masters students at the University of Texas at Austin. In 2017, the LBJ School designed and implemented a new, three-phase math readiness program for incoming masters students. The three phases include: (1) summer self-study through online learning; (2) math validation exams and on-campus review sessions during orientation; and (3) ongoing math support during the fall semester. The central question of the report asks: is there evidence that the LBJ School’s math readiness program drives student success outcomes? To support this program evaluation, I performed a brief literature review and an environmental scan of peer institutions that was informed by interviews with faculty and staff at public policy schools across the country. Many schools are experimenting with different combinations of online, in-person, targeted, or blanket approaches to math readiness over the summer months. The leadership at eight of the nine schools profiled in this study believe that math readiness is an important topic at their school and something worth dedicating resources towards. This program evaluation takes a mixed methods approach to considering one cohort of incoming graduate students at the LBJ School in 2017. Data were collected from various sources, including admissions data, a post-orientation survey, and course grades submitted by faculty, among others. Qualitative results include survey feedback and participation records. Quantitative results include univariate analysis and OLS regression models with two dependent variables to represent student success in the quantitative core courses. This study finds mixed evidence as to whether or not the LBJ School’s math readiness program in 2017 impacted student outcomes in the short term. The univariate and descriptive analysis showed strong empirical evidence of associations between the Phase 1 and Phase 2 components of the math readiness program and student quantitative performance, most of which goes away in the final regression models. However, the study may be under-powered to detect significant associations between quantitative performance and certain components of the math readiness program, such as validation exam scores and the delta variables for students’ self-assessed improvements over the summer. Based on regression analysis, the strongest predictors of success in the quantitative core courses were factors determined well before students began the math readiness program: undergraduate GPA and quantitative GRE scores. Qualitatively, student participation and feedback strongly support the school continuing to provide math resources for incoming students and offer some ideas for program modifications moving forward.