Aligning data with organization's and workers' goals: designing data labeling systems for social service case notes

In the era of data-driven approaches in non-profit and social service government organizations, prevailing data collection methods for performance and funding reports are ineffective and unsatisfactory for both workers and organizational leaders. Within social service provision, evaluating outcomes necessitates intricate subjective assessments, resulting in social workers equipped with profound insights into services and outcomes shouldering the burden of manual record-keeping. Simultaneously, organizational leaders grapple with insufficient data for reporting. While existing research explores data collection challenges, there is a dearth of studies that delve into solutions for enhancing these systems. This study examines data labeling systems that encapsulate client interaction outcomes, focusing on caseworkers aiding those experiencing homelessness. Despite advances in domains such as crowd-sourced data labeling, their approaches often fail to consider the unique values and contexts of social workers who intertwine data labeling with their caregiving work. By employing interviews, ideation, and a speed-dating approach, we scrutinize preferences, potential solutions, and challenges in crafting efficient data labeling systems. We evaluate 15 diverse design ideas across four dimensions: alignment with case management objectives, comprehensive portrayal of caseworker contributions, clarity in data labels, and enhancements in labeling process usability. Our findings highlight the collective aspiration for data labeling systems that cater to varied stakeholder information goals while effectively capturing nuanced casework details, streamlining data labeling into a seamless, efficient task. Leveraging our insights, we offer design implications for enhancing data labeling systems, aligning them with the objectives of both organizations and workers.