|
QMOS: Enhancing LLMs for telecommunications with question-masked loss and option shuffling
|
Authors: Blessed Guda, Gabrial Zencha Ashungafac, Lawrence Francis, Carlee Joe-Wong
Status: Final
Date of publication: 18 December 2025
Published in: ITU Journal on Future and Evolving Technologies, Volume 6 (2025), Issue 4, Pages 391-399
Article DOI : https://doi.org/10.52953/NMOM3434
|
Abstract:
Large Language models (LLMs) have brought about substantial advancements in the field of Question Answering (QA) systems. These models do remarkably well in addressing intricate inquiries in a variety of disciplines. However, because of domain-specific vocabulary, complex technological concepts, and the requirement for exact responses, applying LLMs to specialized sectors like telecommunications presents additional obstacles. GPT-3.5 has been used in recent work, to obtain noteworthy accuracy for telecommunication-related questions in a Retrieval Augmented Generation (RAG) framework. Notwithstanding these developments, the practical use of models such as GPT-3.5 is restricted by their proprietary nature and high computing demands. This paper introduces Question-Masked Option Shuffle (QMOS), an innovative approach which uses a question-masked loss and option shuffling trick to enhance the performance of LLMs in answering multiple-choice questions in the telecommunication domain. We focus on using open-source, smaller language models (Phi-2 and Falcon-7B) within an enhanced RAG framework. Our multi-faceted approach involves several enhancements to the whole LLM-RAG pipeline of fine-tuning, retrieval, prompt engineering and inference. Our approaches significantly outperform existing results, achieving accuracy improvements from baselines of 24.70% to 49.30% with Falcon-7B and from 42.07% to 84.65% with Phi-2. All code developed for this work is publicly available as open-source. |
Keywords: Large language models, option batch-shuffle trick, question-masked loss, RAG, telecommunication
Rights: © International Telecommunication Union, available under the CC BY-NC-ND 3.0 IGO license.
|
|
|
|
| |