Analyzing group behavior from language use with natural language processing and experimental methods : three applications in political science and sociology

This dissertation presents three independent research projects with the common goal of analyzing and understanding group behavior from naturally occurring text, applying Natural Language Processing (NLP) and experimental methods to the domains of political science, sociology, and cognitive science.

The first project develops a case study examining a grassroots initiative to bring an Ohio anti-labor bill to state-wide referendum. Social media platforms like Twitter present new opportunities for researchers to listen in on natural conversations, but this data is unstructured and too large for qualitative or manual analysis. I demonstrate the use of NLP and Machine Learning tools to identify opinions and extract trends from this text data, while addressing pitfalls and biases of these methods.

The second project describes issues with measuring impact and influence using traditional citation analysis, and demonstrates how incorporating full text data improves citation network models. Citation analysis arose to address the need to quantify, filter, and rank scientific publications as they outgrew any single researcher’s ability to comprehensively survey all literature relevant to their research. The problem is that most citation metrics are based solely on network metadata: they operate under the assumption that every citation connotes the same amount of influence as any other, completely ignoring text content. I investigate textual features and comparison metrics indicative of citation relationships, and use my citation prediction system to demonstrate that even simple methods can improve citation models beyond the typical binary cited-or-not network.

Finally, the third project examines how individuals’ beliefs change upon receiving new information. Multiple factors affect this behavior, like reliability of the source, and believability or coherence of information, but there is no one-size-fits-all model describing how people are influenced by new information. I present a novel experimental design to measure belief and confidence change, and show that increased reliability of new information boosts confidence, and that higher confidence decreases likelihood of changing one’s beliefs. The results also suggest some counter-intuitive behaviors: reliability has no discernible effect on willingness to change one’s belief, disagreement is more influential than agreement, and prior confidence has a non-linear effect on how new information changes confidence.