> ## Documentation Index
> Fetch the complete documentation index at: https://docs.consensus.app/llms.txt
> Use this file to discover all available pages before exploring further.

# Benchmark literature providers against your own ground truth

> Build a curated query set with known-relevant papers and measure which retrieval provider actually surfaces them.

## What you get

A defensible answer to "which literature API should we build on", measured on your own domain rather than on a vendor's marketing page — and a regression harness you keep afterwards.

## Who it's for

The engineer choosing a retrieval provider, and whoever has to justify that choice to a technical buyer or a research team who will notice if it is wrong.

## How Phylo did it

Before putting anything into production, [Phylo](https://consensus.app/home/community-voices/phylo/) ran a head-to-head benchmark. The team curated biomedical queries spanning several disease areas, each with a ground-truth list of papers already known to be highly relevant, then measured **at top 20 results** how many of the truly relevant papers each system actually surfaced.

> "We benchmarked across many different providers. We ended up choosing Consensus."
>
> — **Kexin Huang**, Co-founder & CEO, Phylo

<Steps>
  <Step title="Curate queries across your real domain">
    Questions should span the disease areas, research contexts, or subfields your product actually serves — including the awkward ones. A benchmark built only from easy queries measures nothing.
  </Step>

  <Step title="Build ground truth per query">
    For each query, list the papers a domain expert would insist appear. This is the expensive part and the part that makes the result trustworthy; do not shortcut it with model-generated relevance labels.
  </Step>

  <Step title="Measure recall at a fixed k">
    How many ground-truth papers appear in the top k. Phylo used top 20. Fix k before you run anything, because choosing it afterwards is how benchmarks get massaged.
  </Step>

  <Step title="Hold the query text constant across providers">
    Different query phrasings favour different engines. Same strings, same k, and note where a provider needs different phrasing to perform — that is a real integration cost.
  </Step>
</Steps>

## Running it against Consensus

```bash theme={null}
curl -G "https://api.consensus.app/v1/search" \
  -H "x-api-key: $CONSENSUS_API_KEY" \
  --data-urlencode "query=<benchmark query>" \
  --data-urlencode "page_size=20" \
  --data-urlencode "include_semantic_score=true"
```

`include_semantic_score=true` returns a relevance score for the top 50 results, which lets you check score calibration alongside raw recall. It is API-only — the MCP `search` tool does not expose it.

<Note>
  Benchmark unfiltered first. Filters change recall substantially, and a comparison where one provider is filtered and another is not measures your configuration rather than the engines. Add filters in a second pass to see how much precision they buy.
</Note>

## Keep it as a regression harness

The benchmark is more valuable after the decision than during it. Re-run it when you change prompts, filters, or query construction — it is the only way to tell whether a retrieval change actually helped or just felt better.

Split the score in two: **retrieval quality** (did the right papers come back) and **grounding faithfulness** (did the answer stay true to them). A pipeline can fail at either, and the fixes are unrelated.

## What to check before you trust it

* **Beware ground truth built from one provider's output.** If your labels came from a search on system A, system A wins by construction. Build them from expert knowledge or published reviews.
* **Report recall per query, not just the mean.** One catastrophic query hidden inside a good average is exactly the failure your users will hit.
* **Re-check page size.** Comparing top-20 against a provider capped below 20 is not a fair test — confirm the `page_size` echoed back matches what you asked for.
* **Say when the difference is noise.** With 30 queries, small gaps are not signal.

## Related

<CardGroup cols={2}>
  <Card title="Swap a hand-maintained paper corpus for full-text search" icon="arrows-rotate" href="/use-cases/swap-a-paper-corpus">
    What to do once the benchmark decides it.
  </Card>

  <Card title="Screen thousands of target-disease pairs for evidence" icon="table-cells" href="/use-cases/high-throughput-screening">
    The workload that makes retrieval quality worth measuring.
  </Card>

  <Card title="Best practices" icon="shapes" href="/use-cases/best-practices">
    Rate limits, grounding rules, and the five primitives.
  </Card>

  <Card title="All use cases" icon="grid-2" href="/use-cases">
    Browse the gallery by persona.
  </Card>
</CardGroup>