Project CS 2024: Industry Collaboration with Ericsson š¶
Presenting an LLM-based Project at Ericsson HQ, Stockholm
On December 6, 2024, I had the pleasure of supervising a talented group of students who successfully completed their Project CS course (30 credits) with a final presentation at Ericsson HQ in Stockholm.
Overview of the Project
They presented a research project, Optimizing Large Language Models for Domain-Specific Tasks, conducted at Uppsala University, focusing on optimizing Large Language Models (LLMs) for the telecommunications domain.
The project uses the TSpec-LLM dataset, a large collection of 3GPP telecommunications standards documents, and addresses the challenge of adapting general-purpose LLMs to highly specialized technical domains. The work involves:
- Data preprocessing: Designing a modular pipeline to clean and structure domain data, making it suitable for machine learning tasks.
- Model training: Fine-tuning pre-trained T5 (flan-t5-base) models on this domain-specific dataset for question-answering tasks.
- Alternative approach comparison: Implementing and evaluating Retrieval-Augmented Generation (RAG) systems as a contrasting strategy.
- Evaluation: Comparing fine-tuned models and RAG using BLEU scores and semantic cosine similarity (via Sentence-BERT) to measure both syntactic and semantic accuracy.
- Implementation: Developing both LangGraph-based and RAG-based chatbot systems, integrating data preprocessing, retrieval pipelines, and response generation modules.
Key Findings
- Fine-tuned models achieved strong semantic accuracy, especially for tasks with well-defined, closed datasets.
- RAG models performed better for knowledge-intensive or context-rich tasks but incurred higher computational costs and latency.
- Fine-tuned small models (like Flan-T5 Small) showed promise for deployment in resource-constrained environments (e.g., offline devices).
- The study emphasizes cosine similarity as a more reliable evaluation metric than BLEU for semantic tasks.
Student Team
Iām proud to have supervised this exceptional group of students:
Each student brought unique perspectives and skills to the project, contributing to its overall success.