Browsing by Subject "Treatment effect heterogeneity"

Now showing 1 - 2 of 2

Enhancing generalizability and feasibility in sample selection : a methodological study of cluster analysis for stratifying populations
(2023-08) Furman, Gleb; Pustejovsky, James E.; Whittaker, Tiffany A.; Beretvas, Tasha; Tipton, Elizabeth; Roberts, Gregory J
This dissertation focuses on the critical need for research findings that are applicable and generalizable to diverse populations in the context of policy-making and funding allocation. Biases favoring majority groups often emerge due to overlooked variations within subgroups and inadequate sampling strategies. The objective of this study is to help address this issue by providing accessible and effective methods for selecting representative samples, with the ultimate goal of promoting the inclusion of diverse populations and ensuring unbiased estimation of main effects. In the realm of educational intervention research, randomized control trials (RCTs) have played a pivotal role in demonstrating efficacy. However, reliance on convenience sampling restricts the generalizability of findings beyond the study sample. Recent research has highlighted the lack of representative sampling in federally funded efficacy studies, necessitating the development of design-based approaches to enhance generalizability. The present study focuses specifically on stratified sampling using cluster analysis as a promising method for achieving representative samples. In this context, cluster analysis serves as a dimension reduction technique, enabling the population to be stratified based on covariates associated with treatment effect heterogeneity. The selected stratified samples facilitate population-level inference, addressing the limitations of convenience sampling. The primary aim of this study is to investigate the influence of various decisions in the cluster analysis process on the generalizability and feasibility of stratified sampling. By utilizing Monte Carlo simulation and real-world data, the findings shed light on the optimal number of high-quality strata that enhance generalizability without imposing significant recruitment challenges. These findings offer valuable guidance to researchers in effectively allocating resources and devising sampling strategies that maximize the impact of their study designs. Additionally, this study introduces a novel simulation design framework that can be extended for future methodological research. The framework offers flexibility in designing and testing recruitment strategies and accommodates various algorithms for modeling participation bias. By developing rigorous research designs that promote the inclusion of diverse populations, this study informs effective policy-making and funding allocation, ensuring that research findings are applicable to a broad range of demographic groups.
Self-regulated learning and treatment effect heterogeneity in educational interventions : a formal model and simulation study
(2023-04-18) Schuetze, Brendan Alexander; Yan, Veronica X.; Muenks, Katherine; von Hippel, Paul; Whittaker, Tiffany
Research has shown that many interventions or field experiments in education result in effects that are either non-significant or conventionally considered small. Perhaps more concerningly, the interventions with the largest sample sizes and strongest grant-funding support tend to show the smallest effects, with an average of only one-twentieth of a standard deviation of benefit across the entire sample. It appears that few, if any, educational interventions have large or uniform effects across student populations. Rather, it is commonly observed that only small sub-samples of students benefit from any one educational intervention. As a result, educational researchers have called for a “heterogeneity revolution” that will illuminate these sources of variation in treatment effects across sub-populations. Given the recency of these calls, there are many open questions about how to best model sources of variation. At present, most analyses of heterogeneity in education use relatively crude analyses, focused on demographic factors, such as socio-economic status or cultural background, as potential moderators of effects. Though demographic factors are not moderators of interventions in-and-of-themselves (they are proxy variables), there is little existing educational theory to explain this heterogeneity in terms of fundamental self-regulated learning processes known to underlie educational success. I propose a formal model of intervention effectiveness grounded in self-regulated learning called the Learning Production Function. This model allows for evaluating the joint effects of motivational, cognitive, and metacognitive interventions on populations of learners differing in terms of motivation, learning speed, metacognitive accuracy, and prior knowledge. Given this model, I conduct a simulation study. Using this simulation to model different intervention-student combinations, I answer questions such as: “which groups of learners would benefit from motivational interventions?” and “why do large effects of study strategies in laboratory studies fail to translate to self-regulated learning contexts?” This model advances the field’s understanding of different classes of educational interventions, why they work, who they benefit, and how to best combine interventions to help both at-risk and high-achieving student populations. Furthermore, this present dissertation improves upon educational theories by moving beyond “box and arrow” linear mediation models to more formalized understanding of the effects of educational interventions.