What you get
A defensible answer to “which literature API should we build on”, measured on your own domain rather than on a vendor’s marketing page — and a regression harness you keep afterwards.Who it’s for
The engineer choosing a retrieval provider, and whoever has to justify that choice to a technical buyer or a research team who will notice if it is wrong.How Phylo did it
Before putting anything into production, Phylo ran a head-to-head benchmark. The team curated biomedical queries spanning several disease areas, each with a ground-truth list of papers already known to be highly relevant, then measured at top 20 results how many of the truly relevant papers each system actually surfaced.“We benchmarked across many different providers. We ended up choosing Consensus.” — Kexin Huang, Co-founder & CEO, Phylo
1
Curate queries across your real domain
Questions should span the disease areas, research contexts, or subfields your product actually serves — including the awkward ones. A benchmark built only from easy queries measures nothing.
2
Build ground truth per query
For each query, list the papers a domain expert would insist appear. This is the expensive part and the part that makes the result trustworthy; do not shortcut it with model-generated relevance labels.
3
Measure recall at a fixed k
How many ground-truth papers appear in the top k. Phylo used top 20. Fix k before you run anything, because choosing it afterwards is how benchmarks get massaged.
4
Hold the query text constant across providers
Different query phrasings favour different engines. Same strings, same k, and note where a provider needs different phrasing to perform — that is a real integration cost.
Running it against Consensus
include_semantic_score=true returns a relevance score for the top 50 results, which lets you check score calibration alongside raw recall. It is API-only — the MCP search tool does not expose it.
Benchmark unfiltered first. Filters change recall substantially, and a comparison where one provider is filtered and another is not measures your configuration rather than the engines. Add filters in a second pass to see how much precision they buy.
Keep it as a regression harness
The benchmark is more valuable after the decision than during it. Re-run it when you change prompts, filters, or query construction — it is the only way to tell whether a retrieval change actually helped or just felt better. Split the score in two: retrieval quality (did the right papers come back) and grounding faithfulness (did the answer stay true to them). A pipeline can fail at either, and the fixes are unrelated.What to check before you trust it
- Beware ground truth built from one provider’s output. If your labels came from a search on system A, system A wins by construction. Build them from expert knowledge or published reviews.
- Report recall per query, not just the mean. One catastrophic query hidden inside a good average is exactly the failure your users will hit.
- Re-check page size. Comparing top-20 against a provider capped below 20 is not a fair test — confirm the
page_sizeechoed back matches what you asked for. - Say when the difference is noise. With 30 queries, small gaps are not signal.
Related
Swap a hand-maintained paper corpus for full-text search
What to do once the benchmark decides it.
Screen thousands of target-disease pairs for evidence
The workload that makes retrieval quality worth measuring.
Best practices
Rate limits, grounding rules, and the five primitives.
All use cases
Browse the gallery by persona.