An LLM-judged relevance test for search engines
Version 0.1.0 — November 2026
relevance-tester automates the base document method described in How to measure a relevance of the Enterprise Search Engine. In the original method people wrote the queries and judged the results; here an LLM does both.
A test is one base document and one query written so that this document is one of the best answers to it. The engine runs the query, and only its top 10 results count. Every other result of the top 10 is compared with the base document by the LLM:
The score of the test is 1 plus its mistakes, at most 10. A base document not in the 10 results counts as position 11, with all 10 results above it. When the engine returns fewer than 10 results and the base is not among them, it is not found and scores 10. Section 8 shows examples.
The way from the zip file to the first result, in six steps:
| Step | What you do | Section |
|---|---|---|
| 1. Install | Download and unzip the file, then install the program | 4 |
| 2. Connect | Write a connector for your engine and check it; with SANDI, skip this step | 5 |
| 3. Configure | Write relevance-tester.ini: the LLM, your engine, the topics and the setups to compare | 6 |
| 4. Generate | relevance-tester generate: the test queries, once | 7 |
| 5. Evaluate | relevance-tester evaluate --version 1.0: the score of each setup | 7 |
| 6. Read and compare | Open the report; after a change, evaluate again and compare | 8 |
Download relevance-tester-0.1.0.zip from
https://www.softcorporation.com/products/relevance-tester/relevance-tester-0.1.0.zip
and unzip it into a folder of your choice:
unzip relevance-tester-0.1.0.zip
cd relevance-tester-0.1.0
Without the unzip command, extract it with your file manager, or with
python3 -m zipfile -e relevance-tester-0.1.0.zip .
Then choose one of three ways to run it.
pipx installs the relevance-tester command in its own environment:
pipx install .
relevance-tester --help
With pipx install -e . instead, the command runs the code in this folder,
so an updated copy of the files needs no new install.
python3 -m venv .venv
. .venv/bin/activate # Windows: .venv\Scripts\activate
pip install .
relevance-tester --help
Install the only dependency, requests, and run the package from the unzipped folder:
pip install requests
python3 -m relevance_tester --help
In this case write python3 -m relevance_tester wherever this page
writes relevance-tester; it runs the same program with the same options.
On Windows the interpreter may be called python or py.
The program's own tests need no search engine and no LLM, only requests in the Python that runs them:
python3 -m unittest discover -s tests
relevance-tester talks to a search engine through a connector: one Python class with three methods. The program never knows which engine it talks to. SANDI search has a built-in connector (Section 5.4); for any other engine, write one as shown here.
| Method | Returns |
|---|---|
from_config(section) | The connector, set up from its section of the config file (key → text) |
search(query, n) | The top n results in the engine's order, as Result(rank, id, title, url) |
get_document(doc_id) | The whole document for the LLM to read, as Document(id, title, text, url), or None |
describe() | Optional: the settings saved with every run, to tell setups apart |
| Call | When | What it means for the connector |
|---|---|---|
search(topic, 100) | generate: once per topic | It returns up to max_rank results (100), not only 10 |
get_document(id) | generate: each base document candidate | The LLM writes the queries from the title and the first 4,000 characters of the text |
search(query, 10) | evaluate: every query, 4 at a time | It is called from several threads at once and must be safe to |
get_document(id) | evaluate: the base document and each of the top 10, once per document per run, 4 at a time | The LLM judges from the title and the first 6,000 characters of the text: put the important text first |
describe() | Saved with every run | --resume warns when the settings of the saved run differ from the config |
--resume continues.get_document returns None for a result, the result counts as worse than the base. When the base document itself is missing, the test is marked as an error.search: the results in the engine's own order, ranks from 1, ids as strings.get_document: the same ids as search, the title on its own, and as text everything a user would read to decide whether the document answers the query.describe: the settings that make two setups different, so that every run records what it measured.requests.Session, a timeout on every request and a short retry loop.from_config. The config file is then as private as the API key.
This connector expects GET /<index>/search?q=...&size=n to return
{"hits": [{"id": ..., "title": ..., "url": ...}]} and
GET /<index>/documents/<id> to return
{"id": ..., "title": ..., "body": ...}. Adapt the requests and the field
names to your API. Save it as myengine.py in the folder you will run
from:
import time
import requests
from relevance_tester.engine import Document, Engine, Result, option, required
class MyEngine(Engine):
name = "myengine"
@classmethod
def from_config(cls, section):
return cls(url=required(section, "url"),
index=option(section, "index", "documents"),
token=option(section, "token", ""),
timeout=float(option(section, "timeout", "60")))
def __init__(self, url, index, token, timeout):
self.url, self.index, self.timeout = url.rstrip("/"), index, timeout
self.session = requests.Session()
if token:
self.session.headers["Authorization"] = "Bearer " + token
def _get(self, path, **params):
"""The JSON answer of GET url/index/path; None for 404; three tries."""
for attempt in range(3):
try:
response = self.session.get("%s/%s/%s" % (self.url, self.index, path),
params=params, timeout=self.timeout)
if response.status_code == 404:
return None
response.raise_for_status()
return response.json()
except requests.RequestException:
if attempt == 2:
raise
time.sleep(2 * (attempt + 1))
def search(self, query, n):
hits = (self._get("search", q=query, size=n) or {}).get("hits", [])[:n]
return [Result(rank=i + 1, id=str(hit["id"]), title=hit.get("title", ""),
url=hit.get("url", "")) for i, hit in enumerate(hits)]
def get_document(self, doc_id):
doc = self._get("documents/%s" % doc_id)
if doc is None:
return None
return Document(id=doc_id, title=doc.get("title", ""), text=doc.get("body", ""),
url=doc.get("url", ""))
def describe(self):
return {"engine": self.name, "url": self.url, "index": self.index}
describe leaves the token out: it is saved in the data files, which should
hold no secret.
The config names the class as module:Class, with the connector's own
keys next to it (Section 6 has the whole file):
[my-site]
engine = myengine:MyEngine
url = http://localhost:9000
index = documents
token = ; if the engine needs one
python3 -m relevance_tester finds myengine.py in the folder
you run from; the installed relevance-tester command finds it on
PYTHONPATH, so run it as PYTHONPATH=. relevance-tester ...,
or register the connector in a package (Section 5.5).
This script loads the connector from the config exactly as the program does and calls
it once; no LLM is involved. Save it as check_engine.py next to the config:
import sys
from relevance_tester.config import Config
from relevance_tester.engine import load_engine_class
name, query = sys.argv[1], sys.argv[2]
section = Config("relevance-tester.ini").engine_section(name)
engine = load_engine_class(section["engine"]).from_config(section)
print(engine.describe())
results = engine.search(query, 10)
for result in results:
print(result.rank, result.id, result.title)
if results:
doc = engine.get_document(results[0].id)
print(doc.title if doc else "get_document returned None")
print(len(doc.text) if doc else 0, "characters:", doc.text[:300] if doc else "")
python3 check_engine.py my-site pricing
Check that the ranks start at 1, that the ids of search work in
get_document, and that the text is the document's content, not empty and
not only its metadata.
The built-in connector, engine = sandi, talks to the SANDI search API of
one client. Its keys:
| Key | Default | Meaning |
|---|---|---|
client | required | The SANDI client whose collection is searched |
url | http://localhost:8081/sandi/search | The search endpoint |
type | vector | The search type: vector, legacy or hybrid |
precision | low | low or high |
rerank | yes | Whether SANDI reranks the results |
title_field | title | The field that holds the title |
text_fields | every stored field that is not internal or an id | Comma separated fields that make up the text the LLM reads |
timeout | 120 | Seconds to wait for one request |
search sends a POST to url with the
client, the query, page 1 of size n, the search type,
precision and rerank, and asks only for the id, title
and URL fields. Synonyms, query expansion, RAG answers, spell checking and deep
search are always off, so that the test measures the ranking alone. The id of a
result is its _id, else its id.
get_document sends the same request with the query
*, filtered to _id:"<id>", with all fields and no
reranking. The text is every stored field except the internal ones (prefixed
_), the id, the title and the URL, joined; text_fields
limits it to the fields named.
SUCCESS counts as an error.
describe records the client, URL, type, precision and
rerank, so every run saves its setup.
In an installed package, register the class and use its short name,
engine = myengine:
[project.entry-points."relevance_tester.engines"]
myengine = "mypackage.myengine:MyEngine"
Another LLM provider is added the same way: a subclass of
relevance_tester.llm.LLM with a complete() method, named by
provider = module:Class in [llm].
Every setting is in one file, relevance-tester.ini in the folder you run
from (or the file given with --config, or
~/.config/relevance-tester/config.ini). The zip has
relevance-tester.example.ini to start from. A whole file for your own
engine:
[llm]
base_url = https://api.openai.com/v1
api_key = ; your key, or leave empty and set OPENAI_API_KEY
model = gpt-4o-mini ; writes the queries
judge_models = gpt-4o-mini ; judges; comma separated, several judges vote
[judge]
cache = relevance-tester-cache.sqlite
[my-site] ; a collection: the name used on the command line
engine = myengine:MyEngine
url = http://localhost:9000
index = documents
topics = pricing,install,license,backup,upgrade,security,report,export,user,error
[my-site.current] ; setup "current"
index = documents
[my-site.new-ranking] ; setup "new-ranking"
index = documents-v2
| Section | Holds |
|---|---|
[llm] | The LLM: base_url, api_key, model that writes the queries, judge_models that judge |
[generate] | How the queries are made: topics, max_rank (100), docs_per_topic (3), queries_per_document (3), min_chars (200) |
[run], [judge] | workers (4): parallel requests; cache in [judge]: the file that keeps judgments |
[report] | worst (10): the number of worst tests listed in the report |
[<collection>] | One collection of your engine: the connector, its keys and the topics |
[<collection>.<setup>] | One setup of that collection: only the keys that differ |
-o key=value), then the setup, then the collection, then the
sections [llm] to [report]. So topics can be
in [generate] for every collection or in one collection.
data/<collection>/ in the folder you run from, or to
data_dir of the collection section.
cache key, judgments are saved, and a
pair already judged costs nothing again: a new build is judged only on the results
that changed. Without it, every pair goes to the LLM.
chmod 600) and out
of version control. The document texts are sent to the LLM; for a private
collection, use a local model, e.g. base_url = http://localhost:11434/v1
for Ollama, with no key.
There are three commands. Run them from the folder that holds your config file.
| Command | What it does |
|---|---|
generate | Searches the topics and has the LLM write the test queries; once per collection |
evaluate | Runs the queries on the engine, judges the top 10 and writes a report for each setup |
compare | Puts two evaluated results side by side; calls no engine and no LLM |
relevance-tester generate my-site # one collection
relevance-tester generate # every collection that has no queries yet
It prints each topic with the ranks of its base documents, then each base document as its queries are written:
'pricing': 100 results, ranks [1, 50, 100]
3 queries Pricing and plans [pricing, rank 1]
The queries are written once and kept in queries.jsonl: every later
evaluation uses the same tests. Read some of them; they should look like what your
users type. For a new query set, give another file name with
--queries q2.jsonl.
--version is required. It is any name for what you test: a build number,
a date or a change, such as 1.0 or new-boosts. Each result is
saved under a label, <setup>-<version>, or the
version alone for a collection without setups:
relevance-tester evaluate my-site --version 1.0 # every setup: current-1.0, new-ranking-1.0
relevance-tester evaluate my-site.new-ranking --version 1.0 # one setup: new-ranking-1.0
relevance-tester evaluate --version 1.0 # every setup of every collection
It shows the progress, then one line per setup with the path of its report:
25/90 queries
...
25/90 tests judged
...
new-ranking-1.0: average score 3.24 (95% interval 2.69-3.79), 30 documents, 90 queries
report: data/my-site/report-new-ranking-1.0.html
--resume: what is done is kept and the rest continues.
relevance-tester compare my-site current-1.0 new-ranking-1.0 # two setups, one version
relevance-tester compare my-site current-1.0 current-1.1 # one setup, two versions
The first label is the reference. Both must be in the same collection and run on the same queries.
generate makes one LLM call per base document, more when queries are
rejected: about 30 to 90 calls for 10 topics.
evaluate makes up to 10 calls per test and judge model: up to about 900
calls for 90 tests. A small model such as gpt-4o-mini is enough and cheap; with the
cache, an unchanged pair is never judged twice, and a test whose base is not among
fewer than 10 results needs no calls.
workers).| File in data/<collection>/ | Content |
|---|---|
queries.jsonl | The test queries, one per line, with the base document and topic |
run-<label>.jsonl | The top 10 results of every query, as the engine returned them |
judged-<label>.jsonl | The judgments of every pair, with the LLM's reasons |
report-<label>.html, .csv | The report, and the score of every test |
compare-<a>-<b>.html, .csv | The comparison of two labels |
The average score is the mean of the scores of all tests. Lower is better: 1 means the engine made no ordering mistakes involving the base documents, 10 that it did not find them. An average of 3.24 means that a typical query has about 2 ordering mistakes in its top 10 that involve the document it was written for.
| Test | Score |
|---|---|
| The base at position 1, nothing below it is better | 1 |
| The base at position 7, the 6 results above it all better | 1 |
| The base at position 1, 3 better results below it | 4 |
| The base at position 5; above it 2 better and 1 the same; below it 2 better: 5 − 2.5 + 2 | 4.5 |
| The base not in the top 10; of the 10, 6 better and 2 the same: 11 − 7 | 4 |
| The base not in the top 10, all 10 better | 1 |
| The base not among 3 results returned | 10 |
The score is not the average position: a base document at position 7 under six better results scores 1, because that order is right.
The score has no absolute scale: it depends on the collection and the topics. Evaluate what you have now first, such as the current build or a plain keyword search, and compare every change with it. A change is better when its average score is lower and the comparison shows more better tests than worse ones.
The interval shows how sure the average is: a different set of base documents of the same size would likely give an average in this range. When the intervals of two labels overlap widely, the difference may be chance. More topics give narrower intervals.
| Section | What it shows |
|---|---|
| Summary | The average score and its interval, and the share of tests with score 1, with score 1 to 5, and with the base not in the top 10 |
| nDCG@10 and MRR@10 | The same ranks in the form of nDCG@10 and MRR@10 (see below); higher is better |
| Found and not found | How many bases were in the top 10, how many at position 1, and the average score of each group |
| By topic | The average score of each topic: where the engine is weak |
| Judges | How often the judges answered "the same" or gave no valid answer |
| Worst tests | The tests with the highest scores, with the results judged better or the same, to read through |
The "Found and not found" section tells where to work. When most of the score comes from bases not in the top 10, the engine misses documents (recall); when it comes from bases in the top 10, it orders them badly (ranking).
These are not the nDCG@10 and MRR@10 of a test with relevance judgments (qrels). In such a test, the judgments mark every relevant document of a query, and nDCG@10 counts all of them. Here each test has one base document, and no other document is marked relevant. The two numbers are only another way of writing the ranks this test produced:
They give the top positions more weight than the average score does, but they hold nothing the scores do not. Compare them only between labels of this program, never with published nDCG@10 or MRR@10 figures.
compare counts the tests where the second label is better, the same and
worse than the reference, with a sign test: a small p-value means the difference is
unlikely to be chance. The tests with the biggest differences are listed with their
results, so a change can be checked on real queries.
| Message | What to do |
|---|---|
no config file | Run from the folder with relevance-tester.ini, or give --config |
no engine section [x] | The name on the command line is not a section of the config; the message lists the sections |
no API key | Set api_key in [llm], or the OPENAI_API_KEY environment variable |
no topics | Set topics in the collection or in [generate] |
No module named 'myengine' | The connector file is not found: run from its folder with python3 -m relevance_tester, or with PYTHONPATH=. |
| A connection error | The engine is not reachable at its url; test it with check_engine.py |
queries.jsonl not found: run 'generate' first | Run generate for the collection |
queries.jsonl exists | The queries are kept on purpose; for a new set, use --queries with another name |
evaluate needs --version | Give --version with a name for what you test |
label ... exists | Use another version, or --resume to continue a run that stopped |
different query sets | compare needs two labels run on the same queries |
relevance-tester is released under the Apache License 2.0, copyright 2026
SoftCorporation LLC. See the LICENSE file in the zip.