Cross-Lingual RAG Evaluation

Overview

Most RAG systems are built and evaluated in English, while many users ask in German or Spanish over a knowledge base that is mostly English. The study measures that gap under controlled conditions on XQuAD, a professionally translated benchmark in which every German and Spanish question maps to an exact English paragraph, so retrieval can be scored without guesswork: 1,190 questions per language over 240 paragraphs.

Approach

Five retrieval conditions run behind one interface: BM25, an English-only embedding model, two multilingual embedding models, translating the question before retrieval, and a multilingual cross-encoder reranking the best first stage. Each is scored by recall@k and MRR with bootstrap confidence intervals, and differences are tested with a paired bootstrap on the same questions.

Answers were then generated by a local LLM from the top three paragraphs, for the naive and the best setup, and scored by an LLM judge. Before its scores were used, the judge was checked, per language, against labels on 150 answers from a second model of a different family. That is a weaker check than human labels, and is reported as one.

Outcome

A naive English retriever ranks the right paragraph first for 88.5% of English questions but only 36.9% of German and 34.0% of Spanish ones. Each fix recovers most of that gap, and they stack: multilingual embeddings with reranking reach 95.8% for German and 97.6% for Spanish, within a few points of English. A model roughly twice the size performed no better.

At answer level, correctness followed retrieval: when the right paragraph was missing, the model said it did not know rather than inventing an answer, in 43 of 44 cases. The judge agreed with the second model’s labels on correctness (Cohen’s κ 0.98), but its faithfulness scores could not be validated, and the write-up reports them as such.

What breaks, and why

Half of the German and Spanish questions share no meaningful word with the English paragraph that answers them. On those, keyword search finds the right paragraph 1–2% of the time and English-only embeddings 17–19%. The multilingual retrievers barely notice: with reranking they still reach 93–96%.

Keyword search also did much worse on Spanish than on German. The obvious explanation, that German keeps more English words, turned out to be wrong: both languages share English words at almost the same rate. The real cause is Spanish articles. Words like el, la and los are rare in English text, appearing only in names like Los Angeles, so keyword search treats them as highly informative and sends Spanish questions to the wrong paragraphs. Dropping short words from the queries nearly doubled Spanish accuracy, from 19% to 36%, left German unchanged, and closed the gap.