relevance-tester

An LLM-judged relevance test for search engines

Version 0.1.0 — November 2026

1Overview

relevance-tester measures how well a search engine ranks its results, on your own collection, without relevance judgments made in advance. An LLM writes test queries for documents of the collection and then judges the engine's top 10 results. The result is one number per engine setup, the average score: 1 is a perfect order, 10 the worst. The queries and the judgments are kept, so that builds and setups are compared on the same tests. It is open source (Apache License 2.0).

What you need

Contents

  1. Overview
  2. How the test works
  3. Quick start
  4. Installing from the zip file
  5. Connecting your search engine
  6. Configuration
  7. Running the commands
  8. Interpreting the results
  9. Troubleshooting
  10. License

2How the test works

relevance-tester automates the base document method described in How to measure a relevance of the Enterprise Search Engine. In the original method people wrote the queries and judged the results; here an LLM does both.

One test

A test is one base document and one query written so that this document is one of the best answers to it. The engine runs the query, and only its top 10 results count. Every other result of the top 10 is compared with the base document by the LLM:

The score of the test is 1 plus its mistakes, at most 10. A base document not in the 10 results counts as position 11, with all 10 results above it. When the engine returns fewer than 10 results and the base is not among them, it is not found and scores 10. Section 8 shows examples.

The steps

Topics (single words)
→
Base documents from the engine
→
LLM writes queries
→
Engine runs the queries
→
LLM judges pairs
→
Report

3Quick start

The way from the zip file to the first result, in six steps:

StepWhat you doSection
1. InstallDownload and unzip the file, then install the program4
2. ConnectWrite a connector for your engine and check it; with SANDI, skip this step5
3. ConfigureWrite relevance-tester.ini: the LLM, your engine, the topics and the setups to compare6
4. Generaterelevance-tester generate: the test queries, once7
5. Evaluaterelevance-tester evaluate --version 1.0: the score of each setup7
6. Read and compareOpen the report; after a change, evaluate again and compare8

4Installing from the zip file

Download relevance-tester-0.1.0.zip from https://www.softcorporation.com/products/relevance-tester/relevance-tester-0.1.0.zip and unzip it into a folder of your choice:

unzip relevance-tester-0.1.0.zip
cd relevance-tester-0.1.0

Without the unzip command, extract it with your file manager, or with python3 -m zipfile -e relevance-tester-0.1.0.zip .

Then choose one of three ways to run it.

A. With pipx (recommended)

pipx installs the relevance-tester command in its own environment:

pipx install .
relevance-tester --help

With pipx install -e . instead, the command runs the code in this folder, so an updated copy of the files needs no new install.

B. With pip in a virtual environment

python3 -m venv .venv
. .venv/bin/activate              # Windows: .venv\Scripts\activate
pip install .
relevance-tester --help

C. Without installing

Install the only dependency, requests, and run the package from the unzipped folder:

pip install requests
python3 -m relevance_tester --help

In this case write python3 -m relevance_tester wherever this page writes relevance-tester; it runs the same program with the same options. On Windows the interpreter may be called python or py.

Checking the installation

The program's own tests need no search engine and no LLM, only requests in the Python that runs them:

python3 -m unittest discover -s tests

5Connecting your search engine

relevance-tester talks to a search engine through a connector: one Python class with three methods. The program never knows which engine it talks to. SANDI search has a built-in connector (Section 5.4); for any other engine, write one as shown here.

MethodReturns
from_config(section)The connector, set up from its section of the config file (key → text)
search(query, n)The top n results in the engine's order, as Result(rank, id, title, url)
get_document(doc_id)The whole document for the LLM to read, as Document(id, title, text, url), or None
describe()Optional: the settings saved with every run, to tell setups apart

5.1 How the program uses a connector

CallWhenWhat it means for the connector
search(topic, 100)generate: once per topicIt returns up to max_rank results (100), not only 10
get_document(id)generate: each base document candidateThe LLM writes the queries from the title and the first 4,000 characters of the text
search(query, 10)evaluate: every query, 4 at a timeIt is called from several threads at once and must be safe to
get_document(id)evaluate: the base document and each of the top 10, once per document per run, 4 at a timeThe LLM judges from the title and the first 6,000 characters of the text: put the important text first
describe()Saved with every run--resume warns when the settings of the saved run differ from the config

5.2 Writing a connector, step by step

  1. Choose the config keys: how to reach the engine (URL, index, token) and the settings a setup should be able to change.
  2. search: the results in the engine's own order, ranks from 1, ids as strings.
  3. get_document: the same ids as search, the title on its own, and as text everything a user would read to decide whether the document answers the query.
  4. describe: the settings that make two setups different, so that every run records what it measured.
  5. Errors: one requests.Session, a timeout on every request and a short retry loop.
  6. Authentication: a token as a key of the section, read in from_config. The config file is then as private as the API key.
  7. Check it before any LLM call (Section 5.3).

This connector expects GET /<index>/search?q=...&size=n to return {"hits": [{"id": ..., "title": ..., "url": ...}]} and GET /<index>/documents/<id> to return {"id": ..., "title": ..., "body": ...}. Adapt the requests and the field names to your API. Save it as myengine.py in the folder you will run from:

import time

import requests

from relevance_tester.engine import Document, Engine, Result, option, required


class MyEngine(Engine):
    name = "myengine"

    @classmethod
    def from_config(cls, section):
        return cls(url=required(section, "url"),
                   index=option(section, "index", "documents"),
                   token=option(section, "token", ""),
                   timeout=float(option(section, "timeout", "60")))

    def __init__(self, url, index, token, timeout):
        self.url, self.index, self.timeout = url.rstrip("/"), index, timeout
        self.session = requests.Session()
        if token:
            self.session.headers["Authorization"] = "Bearer " + token

    def _get(self, path, **params):
        """The JSON answer of GET url/index/path; None for 404; three tries."""
        for attempt in range(3):
            try:
                response = self.session.get("%s/%s/%s" % (self.url, self.index, path),
                                            params=params, timeout=self.timeout)
                if response.status_code == 404:
                    return None
                response.raise_for_status()
                return response.json()
            except requests.RequestException:
                if attempt == 2:
                    raise
                time.sleep(2 * (attempt + 1))

    def search(self, query, n):
        hits = (self._get("search", q=query, size=n) or {}).get("hits", [])[:n]
        return [Result(rank=i + 1, id=str(hit["id"]), title=hit.get("title", ""),
                       url=hit.get("url", "")) for i, hit in enumerate(hits)]

    def get_document(self, doc_id):
        doc = self._get("documents/%s" % doc_id)
        if doc is None:
            return None
        return Document(id=doc_id, title=doc.get("title", ""), text=doc.get("body", ""),
                        url=doc.get("url", ""))

    def describe(self):
        return {"engine": self.name, "url": self.url, "index": self.index}

describe leaves the token out: it is saved in the data files, which should hold no secret.

The config names the class as module:Class, with the connector's own keys next to it (Section 6 has the whole file):

[my-site]
engine = myengine:MyEngine
url = http://localhost:9000
index = documents
token =                         ; if the engine needs one

python3 -m relevance_tester finds myengine.py in the folder you run from; the installed relevance-tester command finds it on PYTHONPATH, so run it as PYTHONPATH=. relevance-tester ..., or register the connector in a package (Section 5.5).

5.3 Checking the connector

This script loads the connector from the config exactly as the program does and calls it once; no LLM is involved. Save it as check_engine.py next to the config:

import sys

from relevance_tester.config import Config
from relevance_tester.engine import load_engine_class

name, query = sys.argv[1], sys.argv[2]
section = Config("relevance-tester.ini").engine_section(name)
engine = load_engine_class(section["engine"]).from_config(section)
print(engine.describe())
results = engine.search(query, 10)
for result in results:
    print(result.rank, result.id, result.title)
if results:
    doc = engine.get_document(results[0].id)
    print(doc.title if doc else "get_document returned None")
    print(len(doc.text) if doc else 0, "characters:", doc.text[:300] if doc else "")
python3 check_engine.py my-site pricing

Check that the ranks start at 1, that the ids of search work in get_document, and that the text is the document's content, not empty and not only its metadata.

5.4 The SANDI connector

The built-in connector, engine = sandi, talks to the SANDI search API of one client. Its keys:

KeyDefaultMeaning
clientrequiredThe SANDI client whose collection is searched
urlhttp://localhost:8081/sandi/searchThe search endpoint
typevectorThe search type: vector, legacy or hybrid
precisionlowlow or high
rerankyesWhether SANDI reranks the results
title_fieldtitleThe field that holds the title
text_fieldsevery stored field that is not internal or an idComma separated fields that make up the text the LLM reads
timeout120Seconds to wait for one request

5.5 Registering a connector in a package

In an installed package, register the class and use its short name, engine = myengine:

[project.entry-points."relevance_tester.engines"]
myengine = "mypackage.myengine:MyEngine"

Another LLM provider is added the same way: a subclass of relevance_tester.llm.LLM with a complete() method, named by provider = module:Class in [llm].

6Configuration

Every setting is in one file, relevance-tester.ini in the folder you run from (or the file given with --config, or ~/.config/relevance-tester/config.ini). The zip has relevance-tester.example.ini to start from. A whole file for your own engine:

[llm]
base_url = https://api.openai.com/v1
api_key =                       ; your key, or leave empty and set OPENAI_API_KEY
model = gpt-4o-mini             ; writes the queries
judge_models = gpt-4o-mini      ; judges; comma separated, several judges vote

[judge]
cache = relevance-tester-cache.sqlite

[my-site]                       ; a collection: the name used on the command line
engine = myengine:MyEngine
url = http://localhost:9000
index = documents
topics = pricing,install,license,backup,upgrade,security,report,export,user,error

[my-site.current]               ; setup "current"
index = documents

[my-site.new-ranking]           ; setup "new-ranking"
index = documents-v2

The sections

SectionHolds
[llm]The LLM: base_url, api_key, model that writes the queries, judge_models that judge
[generate]How the queries are made: topics, max_rank (100), docs_per_topic (3), queries_per_document (3), min_chars (200)
[run], [judge]workers (4): parallel requests; cache in [judge]: the file that keeps judgments
[report]worst (10): the number of worst tests listed in the report
[<collection>]One collection of your engine: the connector, its keys and the topics
[<collection>.<setup>]One setup of that collection: only the keys that differ

Choosing the topics

The config file holds your API key: keep it private (chmod 600) and out of version control. The document texts are sent to the LLM; for a private collection, use a local model, e.g. base_url = http://localhost:11434/v1 for Ollama, with no key.

7Running the commands

There are three commands. Run them from the folder that holds your config file.

CommandWhat it does
generateSearches the topics and has the LLM write the test queries; once per collection
evaluateRuns the queries on the engine, judges the top 10 and writes a report for each setup
comparePuts two evaluated results side by side; calls no engine and no LLM

generate

relevance-tester generate my-site            # one collection
relevance-tester generate                    # every collection that has no queries yet

It prints each topic with the ranks of its base documents, then each base document as its queries are written:

'pricing': 100 results, ranks [1, 50, 100]
3 queries  Pricing and plans  [pricing, rank 1]

The queries are written once and kept in queries.jsonl: every later evaluation uses the same tests. Read some of them; they should look like what your users type. For a new query set, give another file name with --queries q2.jsonl.

evaluate

--version is required. It is any name for what you test: a build number, a date or a change, such as 1.0 or new-boosts. Each result is saved under a label, <setup>-<version>, or the version alone for a collection without setups:

relevance-tester evaluate my-site --version 1.0               # every setup: current-1.0, new-ranking-1.0
relevance-tester evaluate my-site.new-ranking --version 1.0   # one setup: new-ranking-1.0
relevance-tester evaluate --version 1.0                       # every setup of every collection

It shows the progress, then one line per setup with the path of its report:

25/90 queries
...
25/90 tests judged
...
new-ranking-1.0: average score 3.24 (95% interval 2.69-3.79), 30 documents, 90 queries
report: data/my-site/report-new-ranking-1.0.html

compare

relevance-tester compare my-site current-1.0 new-ranking-1.0    # two setups, one version
relevance-tester compare my-site current-1.0 current-1.1        # one setup, two versions

The first label is the reference. Both must be in the same collection and run on the same queries.

Cost and time

Files

File in data/<collection>/Content
queries.jsonlThe test queries, one per line, with the base document and topic
run-<label>.jsonlThe top 10 results of every query, as the engine returned them
judged-<label>.jsonlThe judgments of every pair, with the LLM's reasons
report-<label>.html, .csvThe report, and the score of every test
compare-<a>-<b>.html, .csvThe comparison of two labels

8Interpreting the results

The average score

The average score is the mean of the scores of all tests. Lower is better: 1 means the engine made no ordering mistakes involving the base documents, 10 that it did not find them. An average of 3.24 means that a typical query has about 2 ordering mistakes in its top 10 that involve the document it was written for.

TestScore
The base at position 1, nothing below it is better1
The base at position 7, the 6 results above it all better1
The base at position 1, 3 better results below it4
The base at position 5; above it 2 better and 1 the same; below it 2 better: 5 − 2.5 + 24.5
The base not in the top 10; of the 10, 6 better and 2 the same: 11 − 74
The base not in the top 10, all 10 better1
The base not among 3 results returned10

The score is not the average position: a base document at position 7 under six better results scores 1, because that order is right.

Start with a baseline

The score has no absolute scale: it depends on the collection and the topics. Evaluate what you have now first, such as the current build or a plain keyword search, and compare every change with it. A change is better when its average score is lower and the comparison shows more better tests than worse ones.

The 95% interval

The interval shows how sure the average is: a different set of base documents of the same size would likely give an average in this range. When the intervals of two labels overlap widely, the difference may be chance. More topics give narrower intervals.

The report

SectionWhat it shows
SummaryThe average score and its interval, and the share of tests with score 1, with score 1 to 5, and with the base not in the top 10
nDCG@10 and MRR@10The same ranks in the form of nDCG@10 and MRR@10 (see below); higher is better
Found and not foundHow many bases were in the top 10, how many at position 1, and the average score of each group
By topicThe average score of each topic: where the engine is weak
JudgesHow often the judges answered "the same" or gave no valid answer
Worst testsThe tests with the highest scores, with the results judged better or the same, to read through

The "Found and not found" section tells where to work. When most of the score comes from bases not in the top 10, the engine misses documents (recall); when it comes from bases in the top 10, it orders them badly (ranking).

nDCG@10 and MRR@10

These are not the nDCG@10 and MRR@10 of a test with relevance judgments (qrels). In such a test, the judgments mark every relevant document of a query, and nDCG@10 counts all of them. Here each test has one base document, and no other document is marked relevant. The two numbers are only another way of writing the ranks this test produced:

They give the top positions more weight than the average score does, but they hold nothing the scores do not. Compare them only between labels of this program, never with published nDCG@10 or MRR@10 figures.

The comparison

compare counts the tests where the second label is better, the same and worse than the reference, with a sign test: a small p-value means the difference is unlikely to be chance. The tests with the biggest differences are listed with their results, so a change can be checked on real queries.

What the score does not tell

9Troubleshooting

MessageWhat to do
no config fileRun from the folder with relevance-tester.ini, or give --config
no engine section [x]The name on the command line is not a section of the config; the message lists the sections
no API keySet api_key in [llm], or the OPENAI_API_KEY environment variable
no topicsSet topics in the collection or in [generate]
No module named 'myengine'The connector file is not found: run from its folder with python3 -m relevance_tester, or with PYTHONPATH=.
A connection errorThe engine is not reachable at its url; test it with check_engine.py
queries.jsonl not found: run 'generate' firstRun generate for the collection
queries.jsonl existsThe queries are kept on purpose; for a new set, use --queries with another name
evaluate needs --versionGive --version with a name for what you test
label ... existsUse another version, or --resume to continue a run that stopped
different query setscompare needs two labels run on the same queries

10License

relevance-tester is released under the Apache License 2.0, copyright 2026 SoftCorporation LLC. See the LICENSE file in the zip.

Python 3.9+ Apache License 2.0 LLM-judged No qrels needed Any search engine OpenAI-compatible LLMs