BEIR Benchmark Results: TREC-COVID and NFCorpus
Updated system — October 2026
Both datasets are part of the BEIR benchmark [1], which tests zero-shot retrieval: the models are not trained on the test collections. Both are biomedical, but they differ in almost every property that matters for search evaluation.
TREC-COVID [2] is built from CORD-19 (COVID-19 Open Research Dataset), a collection of scientific articles about COVID-19 and related coronaviruses; BEIR uses the title and abstract of each article. Its 50 topics were written during the pandemic and are phrased as full questions, for example "what is the origin of COVID-19". Relevance was judged by assessors with biomedical expertise on a graded scale, over pools of documents returned by the systems that took part in the original TREC challenge.
NFCorpus (NutritionFacts Corpus) [3] is built from NutritionFacts.org, a health and nutrition website, and the medical articles it cites. Queries are short and written in everyday language, while the documents are scientific abstracts in medical terminology. Relevance is graded and is derived from the links between the website pages and the cited articles.
| Property | TREC-COVID | NFCorpus |
|---|---|---|
| Source | CORD-19 article titles and abstracts | NutritionFacts.org and cited medical articles |
| Domain | Biomedical — epidemiology, treatment, transmission | Biomedical — nutrition, diet, disease prevention |
| Corpus size | 171,332 documents | 3,633 documents |
| Test queries | 50 | 323 |
| Query style | Full questions, about 11 words on average | Short everyday phrases, about 3 words on average |
| Relevant documents per query | About 494 on average | About 38 on average, varying widely by query |
| Relevance labels | Graded, judged by experts over pooled results | Graded, derived from links to cited articles |
| Main difficulty | Ranking the best documents first in a large corpus | Vocabulary gap between everyday queries and medical text |
Both benchmarks were run on the updated system with the same models and settings. Each benchmark was run twice, without and with the reranker. Both benchmarks were then repeated with Qwen3-Embedding-0.6B in place of Qwen3-Embedding-4B, not quantized (BF16) and at its native 1024 dimensions, with all other settings unchanged.
| Component | Setting |
|---|---|
| Index | Apache Solr 9.8.1 with SolrCloud (ZooKeeper) |
| Embedding model | Qwen3-Embedding-4B [4], GGUF Q4 quantization, embeddings truncated to 1024 dimensions |
| Second embedding model | Qwen3-Embedding-0.6B [4], not quantized (BF16), native 1024 dimensions |
| Vector search | top_k = 100 |
| NLP query analysis | Extraction of entities and important words from the query |
| Synonym expansion | None |
| Precision setting | Low |
| Reranker | Qwen3-Reranker-4B [4], GGUF Q4 quantization, top 30 candidates (when used) |
NLP is used at query time to extract the entities and the important words of the query, which focus the text-matching part of the search. No synonyms are added, so the system works on the original wording of the query. In the main runs both models are 4-bit quantized, and the embeddings are cut from the native size of the model to 1024 dimensions, which keeps the index and the vector search small.
| Metric | No reranker | With reranker — top 30 | Change |
|---|---|---|---|
| NDCG@5 | 0.9163 | 0.9219 | +0.0056 |
| NDCG@10 | 0.8805 | 0.9073 | +0.0268 |
| Precision@10 | 0.9100 | 0.9380 | +0.0280 |
| MRR@10 | 1.0000 | 1.0000 | 0.0000 |
| Recall@10 | 0.0236 | 0.0242 | +0.0006 |
| Recall@100 | 0.1539 | 0.1539 | 0.0000 |
| MAP@10 | 0.0229 | 0.0235 | +0.0006 |
| Metric | No reranker | With reranker — top 30 | Change |
|---|---|---|---|
| NDCG@5 | 0.4100 | 0.4479 | +0.0379 |
| NDCG@10 | 0.3909 | 0.4104 | +0.0195 |
| Precision@10 | 0.2957 | 0.3068 | +0.0111 |
| MRR@10 | 0.5930 | 0.6167 | +0.0237 |
| Recall@10 | 0.1942 | 0.2015 | +0.0073 |
| Recall@100 | 0.3406 | 0.3406 | 0.0000 |
| MAP@10 | 0.1463 | 0.1582 | +0.0119 |
| Metric | No reranker | With reranker — top 30 | Change |
|---|---|---|---|
| NDCG@5 | 0.8881 | 0.9305 | +0.0424 |
| NDCG@10 | 0.8638 | 0.9057 | +0.0419 |
| Precision@10 | 0.9040 | 0.9420 | +0.0380 |
| MRR@10 | 0.9467 | 0.9800 | +0.0333 |
| Recall@10 | 0.0232 | 0.0243 | +0.0011 |
| Recall@100 | 0.1729 | 0.1729 | 0.0000 |
| MAP@10 | 0.0222 | 0.0236 | +0.0014 |
| Metric | No reranker | With reranker — top 30 | Change |
|---|---|---|---|
| NDCG@5 | 0.4079 | 0.4338 | +0.0259 |
| NDCG@10 | 0.3659 | 0.3902 | +0.0243 |
| Precision@10 | 0.2656 | 0.2854 | +0.0198 |
| MRR@10 | 0.5835 | 0.6018 | +0.0183 |
| Recall@10 | 0.1749 | 0.1900 | +0.0151 |
| Recall@100 | 0.3107 | 0.3107 | 0.0000 |
| MAP@10 | 0.1393 | 0.1501 | +0.0108 |
Without any reranker the system reaches NDCG@10 = 0.8805 and Precision@10 = 0.9100: on average 9.1 of the first 10 results are relevant, in a corpus of 171,332 documents. MRR@10 = 1.0000 means that for every one of the 50 queries the first result is relevant.
Reranking the top 30 candidates with Qwen3-Reranker-4B raises NDCG@10 by 2.7 points to 0.9073 and Precision@10 by 2.8 points to 0.9380. NDCG@5 moves much less (+0.0056), so the first five positions were already well ordered by the first stage; most of the gain comes from the second half of the first page, where the reranker replaces weaker results with relevant ones.
MRR@10 stays at 1.0000: after reranking the first result is still relevant for every one of the 50 queries.
Recall@100 is 0.1539 in both runs. The reranker only reorders the first 30 results, so the set of the first 100 does not change.
Recall@10 = 0.0242 and MAP@10 = 0.0235 look small, but with about 500 relevant documents per query ten results cannot cover more than about 2% of them. They do not indicate a weakness of the system; NDCG@10, Precision@10 and MRR@10 are the metrics that describe quality on this dataset.
On NFCorpus the same system reaches NDCG@10 = 0.3909 without a reranker and NDCG@10 = 0.4104 with it. With the reranker, MRR@10 = 0.6167 means that the first relevant result is typically in the first or the second position, and Precision@10 = 0.3068 is about 3 relevant results in the first 10, on a dataset where many queries have few relevant documents to find.
Recall is informative here: with the reranker, 20.2% of all relevant documents are in the first 10 results and 34.1% in the first 100. Unlike TREC-COVID, MAP@10 = 0.1582 reflects real ranking quality, because the relevant set is small enough for ten results to cover a meaningful part of it.
On NFCorpus the reranker raises every ranking metric: NDCG@10 by 2.0 points, MRR@10 by 2.4 points (0.5930 to 0.6167), MAP@10 by 1.2 points and Recall@10 from 0.1942 to 0.2015. The largest gain is at the top of the list: NDCG@5 rises by 3.8 points, from 0.4100 to 0.4479. Recall@100 is 0.3406 in both runs, for the same reason as on TREC-COVID.
On TREC-COVID the first positions are already filled with relevant documents by the first stage, and the reranker mostly improves positions 6 to 10 (NDCG@5 +0.6 points, NDCG@10 +2.7 points). On NFCorpus the first stage leaves more room at the top, and the reranker mostly improves the first five positions (NDCG@5 +3.8 points, NDCG@10 +2.0 points). In both cases the gain comes from reordering the same 30 candidates.
NDCG@10 of 0.9073 on TREC-COVID and 0.4104 on NFCorpus come from the same models and the same settings. The difference is the test, not the system: TREC-COVID rewards putting some of many relevant documents first, with queries that share the vocabulary of the documents; NFCorpus asks to find a few relevant documents from short everyday queries that often share few words with them. Each score should be read against other systems on the same dataset (Section 5).
Both benchmarks were run with no synonyms. The vocabulary bridging on NFCorpus and the results on TREC-COVID come from the embedding model, the NLP extraction of entities and important words, and the reranker.
| NDCG@10 | 4B Q4 | 0.6B BF16 | Difference |
|---|---|---|---|
| TREC-COVID, no reranker | 0.8805 | 0.8638 | −0.0167 |
| TREC-COVID, with reranker | 0.9073 | 0.9057 | −0.0016 |
| NFCorpus, no reranker | 0.3909 | 0.3659 | −0.0250 |
| NFCorpus, with reranker | 0.4104 | 0.3902 | −0.0202 |
In all four runs the 4-bit quantized model Qwen3-Embedding-4B scores higher in NDCG@10 than Qwen3-Embedding-0.6B in full precision. The larger model keeps more retrieval quality after quantization than the smaller model has without it. The 4B Q4 file is 2.5 GB, about twice the 1.2 GB of the 0.6B BF16 weights. On TREC-COVID with the reranker the difference is 0.2 points, within the noise of 50 queries (Section 1).
The reranker narrows the difference. With the 0.6B model it raises NDCG@10 by 4.2 points on TREC-COVID (0.8638 to 0.9057) and by 2.4 points on NFCorpus (0.3659 to 0.3902), against 2.7 and 2.0 points with the 4B model. On TREC-COVID the difference falls from 1.7 to 0.2 points, and Precision@10 is 0.9420 with the 0.6B model against 0.9380 with the 4B model.
Recall@100 shows what the reranker cannot change: it reorders the first 30 results and does not add documents the first stage missed. On TREC-COVID Recall@100 is 0.1729 with the 0.6B model and 0.1539 with the 4B model: the 0.6B model finds about 12% more of the relevant documents in the first 100 but orders them less well, and the reranker largely corrects the order. On NFCorpus it is 0.3107 against 0.3406: the 4B model finds about 10% more of the relevant documents in the first 100, and the difference remains after reranking.
Qwen3-Embedding-4B and the reranker are both 4-bit GGUF quantizations, and the Qwen3-Embedding-4B embeddings are truncated to 1024 dimensions. The Qwen3-Embedding-4B results are obtained with these reduced models, not with the full-precision ones (Section 6).
The tables place the new results next to NDCG@10 figures published for other systems on the same datasets.
| System | NDCG@10 | Setup and source |
|---|---|---|
| BGE-M3 (dense) | 0.5472 | Retrieval only, MTEB results [15] |
| BM25 | 0.6559 | Retrieval only, BEIR paper [1] |
| gte-large | 0.7022 | Retrieval only, MTEB results [15] |
| SPLADE-v2 | 0.7057 | Retrieval only [5] |
| BGE-large-en-v1.5 | 0.7470 | Retrieval only [7], MTEB results [15] |
| jina-embeddings-v5-text-nano | 0.7760 | Retrieval only, MTEB results [15] |
| jina-embeddings-v3 | 0.7781 | Retrieval only [13] |
| jina-embeddings-v5-text-small | 0.7849 | Retrieval only, MTEB results [15] |
| ColBERT v2 | 0.7854 | Retrieval only [6] |
| KaLM-Embedding-V2 (mini-instruct-v2) | 0.7927 | Retrieval only, MTEB results [15] |
| monoT5-3B | 0.8071 | Reranks BM25 top 100 [16] |
| Cohere Rerank-v2 | 0.8181 | Reranks BM25 top 100 [16] |
| Hybrid RRF of SPLADE and BGE-base | 0.8282 | Rank fusion, no reranker [10] |
| KaLM-Embedding-V2.5 (mini-instruct-v2.5) | 0.8298 | Retrieval only, MTEB results [15] |
| ListT5-3B | 0.8470 | Reranks BM25 top 100 [16] |
| ResRank (sliding window) | 0.8500 | Reranks BM25 top 100 [11] |
| mxbai-rerank-large | 0.8533 | Reranks BM25 top 100 [16] |
| RankQwen3-8B | 0.8537 | Reranks BM25 top 100 [12] |
| gte-Qwen2-1.5B-instruct | 0.8538 | Retrieval only, MTEB results [15] |
| SANDI Solr — no reranker | 0.8638 | Retrieval only, Qwen3-Embedding-0.6B BF16, 1024 dimensions |
| jina-reranker-v3 | 0.8659 | Reranks jina-embeddings-v3 top 100 [13] |
| Qwen3-Reranker-4B | 0.8708 | Reranks jina-embeddings-v3 top 100 [13] |
| SANDI Solr — no reranker | 0.8805 | Retrieval only, Qwen3-Embedding-4B Q4, 1024 dimensions |
| Qwen3-Embedding-0.6B | 0.8864 | Retrieval only [14] |
| Qwen3-Reranker-4B | 0.8875 | Reranks Qwen3-Embedding-0.6B top 100 [14] |
| DeBERTa-v3 cross-encoder | 0.8920 | Reranks SPLADE-v3 top 200, full documents [9] |
| Qwen3-Reranker-8B | 0.8966 | Reranks Qwen3-Embedding-0.6B top 100 [14] |
| Querit-Reranker-4B | 0.8982 | Reranks Qwen3-Embedding-0.6B top 100 [14] |
| SANDI Solr — with Qwen3-Reranker-4B Q4 | 0.9057 | Reranks Qwen3-Embedding-0.6B BF16, 1024 dimensions, top 30 candidates |
| SANDI Solr — with Qwen3-Reranker-4B Q4 | 0.9073 | Reranks Qwen3-Embedding-4B Q4, 1024 dimensions, top 30 candidates |
| System | NDCG@10 | Setup and source |
|---|---|---|
| BGE-M3 (dense) | 0.3141 | Retrieval only, MTEB results [15] |
| BM25 | 0.3255 | Retrieval only, BEIR paper [1] |
| SPLADE-v2 | 0.3382 | Retrieval only [5] |
| ColBERT v2 | 0.3480 | Retrieval only [6] |
| KaLM-Embedding-V2 (mini-instruct-v2) | 0.3517 | Retrieval only, MTEB results [15] |
| Qwen3-Embedding-0.6B | 0.3634 | Retrieval only [14] |
| Cohere Rerank-v2 | 0.3636 | Reranks BM25 top 100 [16] |
| SANDI Solr — no reranker | 0.3659 | Retrieval only, Qwen3-Embedding-0.6B BF16, 1024 dimensions |
| jina-embeddings-v3 | 0.3665 | Retrieval only [13] |
| mxbai-rerank-large | 0.3708 | Reranks BM25 top 100 [16] |
| KaLM-Embedding-V2.5 (mini-instruct-v2.5) | 0.3711 | Retrieval only, MTEB results [15] |
| ListT5-3B | 0.3770 | Reranks BM25 top 100 [16] |
| BGE-large-en-v1.5 | 0.3806 | Retrieval only [7], MTEB results [15] |
| gte-large | 0.3817 | Retrieval only, MTEB results [15] |
| jina-embeddings-v5-text-nano | 0.3869 | Retrieval only, MTEB results [15] |
| jina-reranker-v3 | 0.3892 | Reranks jina-embeddings-v3 top 100 [13] |
| monoT5-3B | 0.3897 | Reranks BM25 top 100 [16] |
| SANDI Solr — with Qwen3-Reranker-4B Q4 | 0.3902 | Reranks Qwen3-Embedding-0.6B BF16, 1024 dimensions, top 30 candidates |
| SANDI Solr — no reranker | 0.3909 | Retrieval only, Qwen3-Embedding-4B Q4, 1024 dimensions |
| gte-Qwen2-1.5B-instruct | 0.3934 | Retrieval only, MTEB results [15] |
| jina-embeddings-v5-text-small | 0.3981 | Retrieval only, MTEB results [15] |
| ResRank (sliding window) | 0.3994 | Reranks BM25 top 100 [11] |
| RankQwen3-8B | 0.4005 | Reranks BM25 top 100 [12] |
| SANDI Solr — with Qwen3-Reranker-4B Q4 | 0.4104 | Reranks Qwen3-Embedding-4B Q4, 1024 dimensions, top 30 candidates |
| Qwen3-Reranker-4B | 0.4135 | Reranks Qwen3-Embedding-0.6B top 100 [14] |
| Qwen3-Reranker-4B | 0.4156 | Reranks jina-embeddings-v3 top 100 [13] |
| Qwen3-Reranker-8B | 0.4279 | Reranks Qwen3-Embedding-0.6B top 100 [14] |
| Querit-Reranker-4B | 0.4335 | Reranks Qwen3-Embedding-0.6B top 100 [14] |
Most of the published results use uncompressed models at full precision, at a higher cost in GPU memory and compute and with slower indexing and search.
SANDI Solr is a compromise that trades some of this precision for search speed. Embeddings are stored as BYTE vectors (1 KB per 1024-dimension vector, about 1 GB per million chunks), which reduces index size and speeds up vector scoring. GGUF compressed models allow the whole system, including the Qwen3.8-27B LLM (used for query analysis and answers, not in these benchmark runs), to run on a single NVIDIA RTX 3090.
Reranking 100 candidates, as in most of the published reranker results, is costly: a cross-encoder runs the model once per query-candidate pair, and the cost grows linearly with the number of candidates. On an NVIDIA A100 with 100 TREC-COVID candidates, Qwen3-Reranker-4B takes 1.39 s per query, Qwen3-Reranker-8B 2.32 s and the listwise RankQwen3-8B 16.93 s [12]. GPT-4 reranking of 50 documents can take up to a minute [9] at about $0.60 per query [8].
SANDI Solr reranks the top 30 candidates truncated to 1000 characters, a fraction of the cost of reranking 100 full documents.
Each corpus was indexed with document titles and texts, and every test query was sent to the SANDI Solr search API. Results were scored against the standard BEIR test relevance judgments with the ranx library, using graded relevance for NDCG.
Install libraries: pip install ranx requests
Run Python code for TREC-COVID:
https://softcorporation.com/products/sandi/evaluate-beir-trec-covid.txt
Run Python code for NFCorpus:
https://softcorporation.com/products/sandi/evaluate-beir-nfcorpus.txt
The updated SANDI Solr system was tested on two BEIR benchmarks with one set of settings: top_k = 100, NLP extraction of entities and important words from the query, no synonyms, Low precision. With Qwen3-Embedding-4B (GGUF Q4, 1024 dimensions), on TREC-COVID it reaches NDCG@10 = 0.8805 and MRR@10 = 1.0000 with no reranker, and NDCG@10 = 0.9073 and Precision@10 = 0.9380 with Qwen3-Reranker-4B quantized Q4 over the top 30 candidates. On NFCorpus it reaches NDCG@10 = 0.3909 with no reranker, and NDCG@10 = 0.4104, MRR@10 = 0.6167 and Recall@100 = 0.3406 with the reranker.
With Qwen3-Embedding-0.6B not quantized (BF16), the system reaches NDCG@10 = 0.8638 with no reranker and 0.9057 with the reranker on TREC-COVID, and 0.3659 and 0.3902 on NFCorpus. The quantized 4B model scores higher in NDCG@10 in all four runs.
The two benchmarks test different abilities. TREC-COVID, with a large corpus, 50 long queries and hundreds of relevant documents per query, measures how well the best documents are ranked first; its recall and MAP figures are low by construction. NFCorpus, with a small corpus, 323 short queries and few relevant documents per query, measures how well the system bridges the gap between everyday wording and medical text; its recall and MAP figures are meaningful, and its ceiling for top-10 metrics is lower. Reading each result within its own dataset, the SANDI Solr is strong on both.