A team of researchers from Novosibirsk State University took first place in the international scientific competition SemEval-2026 Task 8 "MTRAGEval: Evaluating Multi-Turn RAG Conversations". The team was led by Ivan Bondarenko, an industrial Associate Professor at NSU and Researcher at NSU’s Department of Mechanics and Mathematics Applied Digital Technologies Laboratory. The results of the competition will be presented at the world's largest conference on computational linguistics, ACL, that will be held in the summer of 2026.
The competition was hosted by IBM and included three tracks. The NSU team participated in Task B, generating answers to user questions based on provided reference documents and the history of a multi-turn dialogue. 26 teams participated and the NSU team came in first achieving a quality metric of 0.7827 (conditioned harmonic mean). This significantly exceeded the organizers best baseline result (0.6390) by 14.4 percentage points.
Bondarenko described their work,
Our team proposed three key approaches to be victorious in the competition. The first involved iteratively improving the system prompt using an LLM agent. We developed a multi-agent system in which a large Gemini neural network analyzes the model's output and suggests improvements to the system prompt. This process is repeated iteratively until a plateau is reached. The second approach involved the use of in-context learning, in which the model learns to perform a task based on several examples of correct solutions to the problem provided in the input context. For each problem category, the researchers selected the most typical examples using the medoid method in the metric space of embeddings. These examples were added to the prompt to demonstrate the correct behavior of the model. This approach has consistently shown the best results.
