How it is measured
Any system of this kind can look like it works on three well-chosen questions. This one was measured over 142 questions written and labelled by hand, and the awkward results are published with the rest.
Each rung adds exactly one thing
Six versions of the same system, each adding exactly one capability to the one before it. The gap between two rows is therefore attributable to that capability rather than to a bundle of changes. Two measurements per row: what it retrieves, and what it retrieves that no longer applies.
The article that settles the question is retrievedRepealed articles are served to the model (lower is better)
Lexical search, added at the "hybrid" rung, made performance worse by 7.5 points. It is published as measured. A table where every row improves on the one above it is a table nobody really measured: only the flattering runs were kept. Reranking then recovered that loss, and more.
The date filter is the change that matters most. Without it, 63.3 % of questions retrieved at least one repealed article, and the model answered correctly from a text that no longer applies. No accuracy metric sees that error: the answer is well written and properly sourced, simply wrong. With the filter: 0.0 %.
| rung | what it adds | article found | answers right | citations | repealed | € / question |
|---|---|---|---|---|---|---|
| baseline | fixed-size chunks | 84.2 % | 45.0 % – 76.7 % | 58.7 % | 66.7 % | 0.0016 |
| article | article-sized chunks | 85.8 % (+1.7) | 46.7 % – 80.8 % | 57.9 % | 67.5 % | 0.0020 |
| hybrid | + lexical search | 78.3 % (−7.5) | 45.8 % – 74.2 % | 58.2 % | 61.7 % | 0.0077 |
| rerank | + reranking | 87.5 % (+9.2) | 49.2 % – 81.7 % | 59.9 % | 62.5 % | 0.0040 |
| rewrite | + query rewriting | 88.3 % (+0.8) | 50.0 % – 81.7 % | 58.2 % | 63.3 % | 0.0042 |
| filtered | + date filter and precedence | 90.8 % (+2.5) | 44.2 % – 80.0 % | 69.7 % | 0.0 % | 0.0045 |
Why accuracy is published as a range
Every other number on this page can be checked without knowing labour law. Deciding whether an answer says the same thing as its reference cannot — and nobody qualified has done it.
Shipped configuration · 142 questions
Expected citations are correct by construction: each question was written from the article that answers it, so the retrieval measurement rests on nobody’s memory. Answer accuracy is the one exception. A first calibration attempt scored Cohen’s kappa 0.489 against a non-expert reader — too weak an agreement to validate anything. So the automatic judge is published as unvalidated rather than certified on a bad sample.
Where the questions come from
The 142 questions are built from the corpus: start from an article, then write a question it answers. The expected citation is therefore correct by construction, and the retrieval measurement depends on nobody’s memory. About 15 % of the questions deliberately have no answer in the corpus, to measure refusals.
The answers were then reviewed by a model from a different vendor, which flagged 12.7 % of rows. Flagged rows were adjudicated by hand, with a control sample drawn from the rows it passed — to check the reviewer as much as the answers.
Justified refusals and false refusals are always published together: a system that refused everything would score 100 % on the first and 100 % on the second. Here, 100.0 % justified refusals against 8.3 % refused in error.
The demo runs the exact code that produced these numbers, and CI fails if the two drift apart. Every eval run is committed to the repository: no number can be revised quietly.