Chatbot for ALICE Run3 simulation and analysis tasks

In the scope of this project we developed AskALICE, an open-source, local chatbot to assist users with ALICE Run3 simulation and analysis tasks within the O² framework. We implemented a RAG pipeline using LangChain, ChromaDB and free models downloaded from HuggingFace. We served the LLMs with llama.cpp on limited AMD hardware. Our knowledge base consists of 400 documents, presentations and transcribed talks. Our evaluation dataset contains 35 question-answer pairs provided by experts. To evaluate the answer grading ability of LLMs, we correlated their inputs to those of Gemini-2.5-Flash API model. We selected four answer correctness metrics (LLM-as-judge score, embedding semantic similarity, ROUGE-L score and BLEU score) and evaluated 7 different models (Qwen2.5, Gemini, Gemma, Mistral, DeepSeek, Gpt, Qwen3) before and after RAG. Our pipeline significantly improved the results and Qwen3-30B-A3B-Instruct performed the best. We also optimized database retrieval parameters and explored the trade-off between performance and response time. We made our chatbot available in Mattermost as user @askalicebeta and defined configurable components. We set up a LangFuse server for call tracing and collecting user feedback. In the last Chapter 12 we described our findings, limitations and provided guidelines for future work. Source code and documentation are available at: https://github.com/ta5946/alice-rag.