BEIR TREC-COVID Benchmark Results
Updated version — July 2026
TREC-COVID is a biomedical retrieval benchmark built from the CORD-19 corpus. It is part of the BEIR benchmark [3], which tests zero-shot generalization of retrieval systems across diverse domains.
| Property | Value |
|---|---|
| Corpus size | 171,332 scientific articles |
| Number of queries | 50 COVID-19 research topics |
| Relevant docs per query | 100–500+ (judged by medical experts) |
| Domain | Biomedical — epidemiology, treatment, transmission |
The high number of relevant documents per query is an important characteristic of this dataset. It makes recall-based metrics (Recall@10, MAP@10) inherently low for any system returning only 10–100 results, regardless of result quality. This is discussed further in Section 5.
Four configurations were tested, each building on the previous:
| Component | Config A | Config B | Config C | Config D |
|---|---|---|---|---|
| Index | Apache Solr 9.8.1 with SolrCloud (ZooKeeper) | |||
| Embedding model | GTE-Large [1] | GTE-Large [1] | Qwen3-Embedding-0.6B [2] | Qwen3-Embedding-4B [2] truncated to 1024 dims |
| Search strategy | Hybrid: KNN vector search + BM25 text search | |||
| NLP entity extraction | Yes | Yes | Yes | Yes |
| Reranker | — | Qwen3-Reranker-0.6B [2] | Qwen3-Reranker-0.6B [2] | Qwen3-Reranker-0.6B [2] |
| Reranking candidates | — | Top 30 | Top 30 | Top 30 |
| WordNet synonym expansion | — | — | WordNet (English) [8] | — |
NLP entity extraction is applied in all four configurations: at query time, SpaCy analyses the query to identify and extract named entities and keywords, which are used to focus and refine the search terms passed downstream.
Configuration C additionally replaces GTE-Large with Qwen3-Embedding-0.6B, which shares the same model family as the reranker and is fine-tuned for retrieval with instruction prefixes. The key differentiator of Config C over A and B is synonym expansion based on WordNet — a comprehensive English lexical database developed at Princeton University [8]. After NLP entity extraction, each extracted term is further expanded with its WordNet synonyms before being passed to both the BM25 and KNN stages, broadening lexical coverage without modifying the index.
Configuration D scales the embedding model up to Qwen3-Embedding-4B, with output embeddings truncated to 1024 dimensions so that index size and KNN search cost remain identical to Config C. Document chunk embeddings are aggregated by summation. Retrieval fetches the top 100 candidates (top_k = 100) and fuses the KNN and BM25 result lists with reciprocal rank fusion at medium precision; the top 30 fused candidates are passed to Qwen3-Reranker-0.6B. Unlike Config C, no synonym expansion is applied — the original query, refined by NLP entity extraction only, is submitted directly.
| Metric | Config A GTE-Large, no reranker |
Config B GTE-Large + Qwen3 reranker |
Config C Qwen3-Emb-0.6B + reranker + synonyms |
Config D Qwen3-Emb-4B + reranker, no synonyms |
D vs C |
|---|---|---|---|---|---|
| NDCG@5 | 0.8544 | 0.8621 | 0.9070 | 0.9138 | +0.0068 |
| NDCG@10 | 0.8087 | 0.8411 | 0.8828 | 0.8866 | +0.0038 |
| Precision@10 | 0.8420 | 0.8800 | 0.9220 | 0.9240 | +0.0020 |
| MRR@10 | 0.9900 | 0.9800 | 0.9800 | 0.9800 | 0.0000 |
| Recall@10 | 0.0219 | 0.0226 | 0.0236 | 0.0238 | +0.0002 |
| Recall@100 | 0.1420 | 0.1420 | 0.1649 | 0.1632 | −0.0017 |
| MAP@10 | 0.0206 | 0.0208 | 0.0227 | 0.0230 | +0.0003 |
| System | NDCG@10 | Notes |
|---|---|---|
| DPR | 0.3326 | Dense Passage Retrieval |
| TAS-B | 0.4817 | Topic-Aware Sampling BERT |
| ANCE | 0.6543 | Approx. Nearest Neighbor Negative CE |
| BM25 | 0.6559 | BEIR paper baseline [3] |
| SPLADE-v2 | 0.7057 | Sparse learned representations [5] |
| SPLADE-v3 (retrieval only) | 0.7470 | Sparse retriever, no reranker [10] |
| BGE-large (retrieval only) | ~0.770 | FlagEmbedding, no reranker [7] |
| ColBERT v2 | 0.7854 | Late interaction model [4] |
| GPT-3.5 Turbo reranker (top-50 from SPLADE-v3) | 0.7860 | Listwise LLM reranking [10] |
| MonoT5 reranker (top-100) | ~0.807 | Sequence-to-sequence cross-encoder |
| SANDI Solr — Config A (GTE-Large, no reranker) | 0.8087 | This work — hybrid KNN + BM25 only |
| SANDI Solr — Config B (GTE-Large + Qwen3-Rer-0.6B) | 0.8411 | This work — 30 rerank candidates |
| RankZephyr (top-100 from SPLADE++ ED) | 0.8535 | Open-weight listwise reranker, as reported in [10] |
| RankGPT (GPT-4 reranker) | 0.8551 | LLM-based listwise reranking [6] |
| Qwen3-Reranker-0.6B (top-100 from jina-embeddings-v3) | 0.8628 | jina-reranker-v3 paper, Sep 2025 [9] |
| jina-reranker-v3 | 0.8659 | Listwise reranker, top-100 candidates [9] |
| Qwen3-Reranker-4B (top-100 from jina-embeddings-v3) | 0.8708 | jina-reranker-v3 paper, Sep 2025 [9] |
| GPT-4 reranker (top-100 from SPLADE-v3, sliding window) | 0.8820 | Zero-shot listwise LLM reranking [10] |
| SANDI Solr — Config C (Qwen3-Emb-0.6B + Qwen3-Rer-0.6B + WordNet) | 0.8828 | This work — 30 rerank candidates with query expansion |
| SANDI Solr — Config D (Qwen3-Emb-4B 1024d + Qwen3-Rer-0.6B) | 0.8866 | This work — best result; 30 rerank candidates, no query expansion |
| DeBERTa-v3 cross-encoder (top-200 from SPLADE-v3) | 0.8920 | Reranks 200 candidates at full document length [10] |
Configs A, B, and D operate in a zero-shot setting and can be directly compared to published baselines. Config B (NDCG@10 = 0.8411) using only a 0.6B reranker places above ColBERT v2, MonoT5, BGE-large, and within 1.4 points of RankGPT (GPT-4, 0.8551), at a fraction of the compute cost. Config C adds WordNet synonym expansion at query time. Config D (NDCG@10 = 0.8866) is the strongest configuration in this study: it surpasses RankGPT by 3.2 points using a 4B embedding model truncated to 1024 dimensions and a 0.6B reranker. It also places above the most recent published TREC-COVID results from the jina-reranker-v3 paper (Sep 2025) [9], where the strongest system reaches 0.8708 over top-100 candidates from jina-embeddings-v3.
The only entry in the table above Config D comes from the SPLADE-v3 reranking study of Déjean, Clinchant and Formal (2024) [10]: a 304M-parameter DeBERTa-v3 cross-encoder reranking the top 200 SPLADE-v3 candidates, at 0.8920. The same paper reports SPLADE-v3 retrieval alone at 0.7470, so the reranking stage contributes a gain of 14.5 points over its own first stage. Its zero-shot status is the standard BEIR one — neither SPLADE-v3 nor the cross-encoder was trained on TREC-COVID, both being trained on MS MARCO — although the cross-encoder was trained specifically to rerank SPLADE output, so the two stages are co-tuned. The paper's GPT-4 numbers (0.8820 with a sliding window over 100 candidates, 0.8690 at depth 25) come from zero-shot prompting and place below the cross-encoder on this dataset, which supports the study's overall conclusion: traditional cross-encoders remain highly competitive with LLM-based rerankers.
Normalized Discounted Cumulative Gain accounts for graded relevance and rank position. Config C improves +4.2 points over Config B and +7.4 points over Config A. The gain from B to C is larger than the gain from A to B (+3.2 points), showing that synonym expansion and Qwen3 embeddings together contribute more than reranking alone. Config D pushes NDCG@10 further to 0.8866 — the best overall result — without any query expansion, showing that the stronger Qwen3-Embedding-4B model alone recovers and exceeds the gain that Config C obtains from synonyms, while remaining directly comparable to published baselines.
The top-5 improvement (+0.0449 from B to C) is consistent with the NDCG@10 gain, confirming that Qwen3 embeddings combined with synonym expansion improve ranking quality across the top result positions, not just in the tail. Config D reaches NDCG@5 = 0.9138, the strongest top-5 ranking of all configurations.
Config C reaches Precision@10 = 0.9220, meaning on average more than 9 out of 10 returned results are relevant. This is an exceptional result for a corpus of 171,332 documents. The +4.2 point gain over Config B is a strong signal of the benefit of synonym expansion: BM25 now matches relevant documents that would previously have been missed due to surface-form vocabulary differences. Config D edges this further to 0.9240 — with no synonym expansion at all.
Mean Reciprocal Rank measures the position of the first relevant result. Configs C and D maintain MRR@10 = 0.9800, exactly matching Config B. The reranker consistently places a highly relevant document at rank 1 in the vast majority of queries — a near-perfect first-result experience preserved across all reranked configurations.
Recall@100 increases by +0.0229 from B to C, a substantial improvement. Configs A and B both plateau at 0.1420 because the first-stage retrieval using GTE-Large embeddings reaches a coverage ceiling — the 30-candidate pool drawn from KNN + BM25 already captures as many relevant documents as that embedding can identify. Config C's improvement comes from two sources: Qwen3-Embedding-0.6B retrieves a different set of relevant documents than GTE-Large, and synonym expansion widens BM25 coverage to documents that share no exact query terms with the original query. Together they raise the effective candidate ceiling before reranking. Config D scores 0.1632 — slightly below Config C (−0.0017). Without synonym expansion, BM25 misses a small number of lexically divergent documents in the deep tail, so Config C remains the best choice when maximum recall matters.
Both metrics remain low, as structurally expected for TREC-COVID. The best Recall@100 (Config C, 0.1649) implies approximately 606 relevant documents per query on average (100 / 0.1649 ≈ 606). Retrieving 10 from ~600 relevant documents gives a theoretical recall ceiling near 1.6%:
Recall@10 ≈ 10 / avg_relevant_per_query ≈ 10 / 606 ≈ 1.6%
The observed 2.36–2.38% exceeds this floor, confirming that the retrieved top-10 is disproportionately relevant. Low recall and MAP are structural properties of the dataset, not of the retrieval system.
Config C presents the addition of English synonym expansion using WordNet [8], a large lexical database developed at Princeton University. At query time, query terms are expanded with WordNet synonym sets before being submitted to the retrieval pipeline. This is particularly effective on TREC-COVID, where queries use general language (e.g., "heart disease") while documents use clinical terminology (e.g., "cardiac disorder", "myocardial condition"). Synonym expansion bridges this vocabulary gap directly in the BM25 term-matching stage, recovering relevant documents that semantic embeddings alone do not always surface. The improvement in Precision@10 (+4.2 points) and Recall@100 (+2.3 points) is largely attributable to this effect.
Config C replaces GTE-Large with Qwen3-Embedding-0.6B [2], a retrieval-focused embedding model from the same family as the Qwen3-Reranker-0.6B used in the second stage. While GTE-Large shows strong English embedding quality, using matched embedding and reranker models from the same training lineage tends to improve pipeline coherence — the reranker is better calibrated to the score distribution produced by the embedding model. The improvement in Recall@100 (from 0.1420 to 0.1649) is partly attributable to Qwen3-Embedding-0.6B retrieving a different and more complementary set of candidates compared to GTE-Large.
Config D scales the embedding model from 0.6B to 4B parameters while truncating the output embeddings to the same 1024 dimensions and summing document chunk embeddings, so stored vectors and KNN latency are unchanged — the extra cost is confined to embedding inference. Retrieval fuses KNN and BM25 result lists with reciprocal rank fusion over the top 100 candidates at medium precision, and Qwen3-Reranker-0.6B refines the top 30. The result is the strongest of the four configurations: NDCG@10 = 0.8866 and Precision@10 = 0.9240 with no query-time expansion. That Config D surpasses Config C on every ranking metric while forgoing synonyms indicates that a sufficiently strong embedding model internalizes much of the vocabulary bridging that WordNet expansion provides — though Config C's higher Recall@100 (0.1649 vs 0.1632) shows synonym expansion still adds unique coverage in the deep tail.
The Config D results make clear that the quality of the embedding model is the decisive factor in overall retrieval quality: upgrading the first-stage embeddings to Qwen3-Embedding-4B produced the best scores on every ranking metric. Scaling the reranker, by contrast, does not pay off. Qwen3-Reranker-4B was evaluated in place of the 0.6B model, in the same role it plays in the jina-reranker-v3 experiments [9], and the outcome did not justify keeping it: the ranking gain was marginal, while inference cost per query rose sharply, since every candidate now has to be scored by a 4B-parameter model rather than a 0.6B one. It was therefore dropped from the final runs. Model capacity is better spent on the embedding stage, where it lifts the entire candidate pool, than on a reranker that only reorders the handful of top positions the compact 0.6B model already handles well at a fraction of the latency.
All SANDI configurations rerank only the top 30 candidates. The one published TREC-COVID result that scores higher than Config D — DeBERTa-v3 at 0.8920 [10] — reranks 200. Reranking 200 results per query is a very expensive approach and impractical in production: it means running the cross-encoder roughly seven times more often per query, which multiplies latency and GPU cost accordingly, and the cost is paid on every single query rather than at index time. For the 0.5 NDCG@10 points that separate it from Config D's 30-candidate pipeline, that trade is not worth making in an interactive search system.
The same study also shows how fragile the depth-200 number is. Truncating the documents fed to the reranker from 512 to 64 tokens drops DeBERTa-v3 at depth 200 from 0.8920 to 0.7910 — below Config A, which uses no reranker at all, and below the same cross-encoder at depth 50 on truncated documents (0.8260). Increasing depth therefore only helps when the reranker also receives full-length documents, which compounds the cost: more candidates, each longer. Config D reaches 0.8866 while scoring 30 candidates, so the two systems sit within half a point of each other at very different operating points.
Config D reaches Precision@10 = 0.9240, with Config C close behind at 0.9220 — on average more than 9 of the top-10 results are relevant. For search interfaces and RAG pipelines where users see only the top few results, this is the most practically relevant metric on this dataset. For use cases requiring high recall — systematic reviews, literature surveys, legal discovery — Config C's Recall@100 = 0.1649 is the appropriate target, and increasing the first-stage candidate pool beyond 30 would further improve it at the cost of higher reranking latency.
Configs C and D both maintain MRR@10 = 0.9800, identical to Config B. Despite synonym expansion (Config C) or a changed embedding model and RRF fusion (Config D) altering the candidate set, the Qwen3-Reranker consistently places a highly relevant document at rank 1. This confirms that the reranker is well calibrated to the Qwen3-Embedding score distribution and absorbs these candidate-set changes without any degradation to first-result quality.
GTE-Large, Qwen3-Embedding-0.6B, and Qwen3-Embedding-4B all handle semantic similarity across biomedical synonyms and paraphrases. BM25 handles exact keyword matches for drug names, gene identifiers, and technical terms. Synonym expansion further amplifies the BM25 component's ability to match on semantically equivalent surface forms. The three mechanisms are complementary and together cover the full vocabulary range of TREC-COVID queries.
Install libraries: pip install ranx requests
Run Python code:
https://softcorporation.com/products/sandi/evaluate-beir-trec-covid.txt
SANDI Solr was tested in four configurations on BEIR TREC-COVID. All four configurations include NLP entity extraction: at query time, SpaCy analyses the query to identify named entities and keywords used to focus the search. Config B (GTE-Large + Qwen3-Reranker-0.6B) scores NDCG@10 = 0.8411 and MRR@10 = 0.9800, placing above ColBERT v2, MonoT5, and BGE-large and within 1.4 points of GPT-4-based reranking using only a 0.6B cross-encoder — an impressive result in its own right. Config C adds WordNet synonym expansion, where each extracted entity is expanded with synonym sets before retrieval, lifting NDCG@10 to 0.8828 and achieving the highest Recall@100 (0.1649) — a powerful illustration of the enduring value of NLP. The best result of the four comes from Config D: Qwen3-Embedding-4B truncated to 1024 dimensions with summed chunk embeddings, reciprocal rank fusion over the top 100 candidates at medium precision, and Qwen3-Reranker-0.6B over the top 30. With no query expansion at all, Config D reaches NDCG@10 = 0.8866 and Precision@10 = 0.9240 — directly comparable to published baselines, surpassing GPT-4 listwise reranking (0.8551 for RankGPT, 0.8820 at best over SPLADE-v3 candidates [10]), a considerably more expensive approach. The single published result that scores higher, a DeBERTa-v3 cross-encoder at 0.8920 [10], gets there by reranking 200 candidates instead of 30, at full document length, and falls to 0.7910 when those documents are truncated (Section 6). Together the results show that a strong embedding model is the decisive component: Qwen3-Embedding-4B paired with the compact Qwen3-Reranker-0.6B delivers ranking quality on par with the strongest published systems at a fraction of their reranking depth, whereas the larger Qwen3-Reranker-4B, also evaluated here, traded a sharp increase in inference cost for a marginal ranking gain and was dropped. Classical lexical resources remain the tool of choice when deep recall matters.