Designing a multi-perspective search system using large language models and retrieval augmented generation

In the context of information retrieval, multi-perspective search is a desired solution when the search query focuses on contentious topics that might not have clear factual grounds for an answer - "Should humans colonize space" being an example of such a search query. Although the explicit intent of this query might require a definitive answer ("Yes"/"No"), an ideal search result of the query should add the necessary context, or perspective along with the definitive answer. Added to this is the facet that there can be multiple such perspectives that can be used to answer the question, and hence the need for multi-perspective search systems. However, seeking diverse perspectives in information-seeking contexts is a challenging problem to solve - traditional search engines, while effective in aggregating data, often fall short in providing a cohesive context, particularly when addressing complex, contentious topics. Motivated by these shortcomings, this thesis introduces a multi-perspective search system that leverages the capabilities of Large Language Models (LLMs) and Retrieval Augmented Generation (RAG). Given a search topic of interest, the proposed system employs LLMs to generate and embody diverse personas that represent different perspectives on the topic of interest. Each persona then presents their perspective as part of a simulated debate format, resembling a hypothetical discussion between the different stakeholders. Retrieval Augmented Generation is employed to provide substantiating evidence as part of each argument presented in the debate. This innovative approach allows users to explore a topic through a dialogue that synthesizes multiple perspectives, offering a richer and more nuanced understanding of the topic of interest. The system is designed with a user interface that supports this complex interaction, making it accessible and engaging for users. The development of this system not only advances the field of multi-perspective search but also opens new avenues for potential applications in conversational interfaces, decision-making support systems, and online discussions on digital platforms. This thesis discusses the motivation for multi-perspective search systems, conceptualization of the proposed approach, the design of the system’s interface and architecture, implementation challenges, and potential use-cases of the proposed system, setting a robust foundation for future enhancements and wider application.