Publications

Adaptive QA Within a Society of LLMs

Published in arXiv Preprint, 2024

In question answering (QA), different questions can be effectively addressed with different answering strategies. Some require a simple lookup, while others need complex, multi-step reasoning to be answered adequately. This observation motivates the development of a dynamic method that adaptively selects the most suitable QA strategy for each question, enabling more efficient and effective systems capable of addressing a broader range of question types. To this aim, we build on recent advances in the orchestration of multiple large language models (LLMs) and formulate adaptive QA as a dynamic orchestration challenge. We define this as a contextual multi-armed bandit problem, where the context is defined by the characteristics of the incoming question and the action space consists of potential communication graph configurations among the LLM agents. We then train a linear upper confidence bound model to learn an optimal mapping between different question types and their corresponding optimal multi-LLM communication graph representation. Our experiments show that the proposed solution is viable for adaptive orchestration of a QA system with multiple modules, as it combines the superior performance of more complex strategies while avoiding their costs when simpler strategies suffice.

Mohanna Hoveyda, Arjen P. de Vries, Harrie Oosterhuis, Faegheh Hasibi, Maarten de Rijke

Real-World Conversational Entity Linking Requires More Than Zero-Shots

Published in ACL 2024 (Findings), 2024

Entity linking (EL) in conversations faces notable challenges in practical applications, primarily due to the scarcity of entity-annotated conversational datasets and sparse knowledge bases (KB) containing domain-specific, long-tail entities. Our findings reveal that previous evaluation approaches fall short of capturing real-world complexities for zero-shot EL, highlighting the necessity for new approaches to design and assess conversational EL models to adapt to limited resources. The evaluation setup and the dataset proposed in this research are made publicly available.

Mohanna Hoveyda, Arjen Vries, Faegheh Hasibi, and Maarten Rijke. Published In Findings of the Association for Computational Linguistics (ACL), 2024.
Download Paper | Download Slides

Network Science-Based Structural Analysis of Semantic Representations

Published in , 2023

This study aims to classify semantic representations based on the structural characteristics of their graph structures. To this end, we studied the structural differences between various semantic representations such as contextualized, sensory-grounded, knowledge-enriched, and human-based semantic representations. Since classifying the semantic representations based on their graph structures may be confounded by the graph sizes, we introduced a novel statistical approach that enables clustering of semantic representations while considering the effect of graph size in the comparisons.

Mohanna Hoveyda, Paulino Villas Boas, Mahmood Bijankhan, Mostafa Salehi.
Download Paper