Assessing workers’ decision quality with scarce ground truth data

Accurately assessing workers’ decision quality is fundamental for management, and the efficiency of expert and crowd-sourcing markets. This paper establishes novel ML and AI methods to accurately evaluate workers’ decision accuracy and bias with scarce ground truth (GT or gold standard GS) data, and to further improve accuracy assessment through costly effectively acquiring GT if given an acquisition budget. Without the proposed methods, assessing workers’ decision quality typically requires GT data to compare with workers’ noisy decisions. However, GT is often prohibitively costly to acquire for even a small fraction of each worker’s decisions. For example, physicians may determine a diagnosis and initiate a treatment, yet the correct decision, such as the one that can be established by a panel of physicians. Consequently, in practice, there is often poor transparency regarding physicians’ decision quality. In my dissertation, I collaborating with my coauthors developed the groundwork for achieving scalable and inexpensive assessments of workers’ decision accuracy and bias. The empirical results show that the decision accuracy assessment with very limited GT improves the best available approach by 60% to 93%; my bias assessment produces either comparable to or outperforms the commonly used existing approach; my cost-effective GT acquisition strategy applied in Amazon Mechanical Workers’ accuracy assessment achieves the same performance only using 1/3 of the GT or improve the assessment by 24%. All proposed methods have significant implications in many impactful domains including health care, fraud detection, fact checking, and online labor markets. The methods proposed in this dissertation address the problem of estimating workers’ decision accuracy and bias from historical data with scarcely available ground truth, and achieve the state of the art performance. This dissertation lays the groundwork towards increasing transparency in workers’ (sources’) decision quality.