> ## Documentation Index
> Fetch the complete documentation index at: https://docs.consensus.app/llms.txt
> Use this file to discover all available pages before exploring further.

# Retrieval for RAG and AI agents

> What a RAG pipeline or research agent needs from a scholarly search API, how Consensus compares with PubMed, Europe PMC, OpenAlex and Semantic Scholar for grounding, and when web search APIs like Exa and Tavily fit.

A RAG pipeline or research agent is only as good as what its retrieval step puts in the context window. For scientific questions, that means four things from a search API:

1. **The right papers in the top few results** for a question written in plain language, because agents ask in sentences.
2. **Full-text snippets, not just titles**, so the model reads the passage that supports a claim.
3. **Metadata to weigh the evidence**, so the model can prefer a meta-analysis to a case report.
4. **Stable identifiers**, so every claim the model makes stays citable.

## How the APIs compare for RAG

| | Plain-language queries | Text for the context window |
| - | - | - |
| **Consensus** | <ul><li>Hybrid semantic and keyword search</li><li>Ranked for scientific relevance</li></ul> | <ul><li>Up to 3 query-relevant passages per paper, labeled by section (paid plans)</li><li>Abstract and a one-sentence takeaway</li></ul> |
| **PubMed** | <ul><li>Term matching with MeSH mapping<sup>[1](#source-1)</sup></li></ul> | <ul><li>Abstracts. Full text lives in PubMed Central<sup>[2](#source-2)</sup></li></ul> |
| **Europe PMC** | <ul><li>Keyword search over abstracts and full text<sup>[3](#source-3)</sup></li></ul> | <ul><li>Whole open-access articles as XML, to chunk yourself<sup>[4](#source-4)</sup></li></ul> |
| **OpenAlex** | <ul><li>Semantic search, up to 50 results per query<sup>[5](#source-5)</sup></li></ul> | <ul><li>Whole open-access PDFs and TEI XML, to chunk yourself<sup>[6](#source-6)</sup></li></ul> |
| **Semantic Scholar** | <ul><li>Relevance search and snippet search<sup>[7](#source-7)</sup></li></ul> | <ul><li>Passages of about 500 words through snippet search<sup>[7](#source-7)</sup></li></ul> |

## Full-text snippets, ready for a prompt

Add `include_full_text_chunks=true` to a Consensus search on a paid plan. Each eligible result gains up to 3 passages from the body of the paper, chosen for your query, each starting with the section it came from. You get the doses, methods and measured values that abstracts leave out, without downloading and chunking PDFs.

```python Python theme={null}
import os
import requests

resp = requests.get(
    "https://api.consensus.app/v1/search",
    headers={"x-api-key": os.environ["CONSENSUS_API_KEY"]},
    params={"query": "what dose of creatine improves working memory", "include_full_text_chunks": True},
    timeout=30,
)
resp.raise_for_status()

context = []
for paper in resp.json()["results"]:
    evidence = paper["full_text_chunks"] or [paper["abstract"]]
    for passage in evidence:
        context.append(f"[{paper['doi']}] ({paper['study_type']}, n={paper['sample_size']}) {passage}")
```

Snippets come from open-access papers. Partner full text improves which papers rank, but isn't returned. When a paper has no snippet, fall back to its `abstract` and `takeaway`. See [Full text in the API](https://docs.consensus.app/api-full-text.md).

## Or let Consensus run the agent

If you'd rather not build the retrieval loop, the [Threads API](https://docs.consensus.app/api-quickstart-threads.md) runs the Consensus research agent for you. It does more than search: it plans a multi-step strategy, crawls citations backward and forward from key papers, looks up authors, finds specific papers and reads them in full. It returns a written answer with numbered citations and the quotes it relied on.

* **`pro`** reads about 20 papers. Use it for most questions.
* **`deep`** reads about 50 papers and writes a structured report. It needs a paid plan.

For assistants and agents that speak MCP, the [Consensus MCP server](https://docs.consensus.app/consensus-mcp.md) offers the same search and research agent as tools.

## Web search APIs vs a scholarly search API

Exa and Tavily are web search APIs built for AI agents. Exa's search endpoint "lets you search the web", and offers a `publication` category for research papers and journal articles<sup>[8](#source-8)</sup>. Tavily calls itself "the web layer for AI agents"<sup>[9](#source-9)</sup>. They're the right tools for news, company pages and the open web.

The biggest difference is full text. Consensus searches the full text of papers from 13 publisher partners, including 7 of the 10 largest academic publishers, as well as every open-access paper, and hands your agent the passage from the paper that supports the point. That text isn't on the open web, so a web index can't search it. Search results carry snippets from open-access papers, and the research agent's citations carry the quotes it relied on.

A scholarly API also gives your agent:

* **A defined corpus of papers**, each with a DOI, journal and year, instead of a mix of papers, press releases and blog posts.
* **Evidence metadata**, so the agent can tell a randomized trial from a case report and a peer-reviewed paper from a preprint.
* **Retraction handling.** Consensus marks retracted papers and never uses them in its AI answers.

Many teams give their agent both: web search for current events and general facts, Consensus for anything that needs a citation to the literature.

## In production

* **Owkin** replaced a hand-maintained set of PubMed abstracts with the Consensus MCP server inside its AI scientist, and got 25x the paper coverage. See [Swap a hand-maintained paper corpus](https://docs.consensus.app/use-cases/swap-a-paper-corpus.md).
* **Phylo**'s agents fan each biology question out into parallel Consensus API searches to screen about 20,000 drug targets. See [Parallel search angles](https://docs.consensus.app/use-cases/parallel-search-angles.md).

## Related

> [card] [Add an evidence layer to an agent](https://docs.consensus.app/use-cases/add-an-evidence-layer.md)
>
> Give a working agent peer-reviewed literature as a tool.

> [card] [Compare scholarly APIs](https://docs.consensus.app/api-comparison)
>
> Why teams choose Consensus, and every API at a glance.

## Sources

Facts about other providers were checked against their official documentation on 5 November 2026. Limits and prices change, so check the source before you build on a number.

1. [PubMed User Guide](https://pubmed.ncbi.nlm.nih.gov/help/): Boolean operators, field tags, MeSH and Best Match sorting.
2. [About PMC](https://pmc.ncbi.nlm.nih.gov/about/intro/): "more than 10 million full-text article records".
3. [About Europe PMC](https://europepmc.org/About): 49.2 million abstracts and 12.5 million full-text articles; searches abstracts and full text; searches can be limited to clinical trials, preprints or article sections such as Methods; free to anyone.
4. [Europe PMC developer resources](https://europepmc.org/developers): REST, SOAP, OAI-PMH and FTP access to open-access content and metadata; automated bulk download of other content isn't allowed. The [Articles RESTful API](https://europepmc.org/RestfulWebService) includes `/{id}/fullTextXML`.
5. [OpenAlex semantic search](https://help.openalex.org/api/semantic-search/): embeds title, abstract and venue name; up to 50 results per query; 1 request a second; 17 supported filters, and others such as `cited_by_count` return a 400 error.
6. [OpenAlex full text](https://help.openalex.org/access/fulltext/): cached PDF and TEI XML for more than 60 million open-access works.
7. [Semantic Scholar Academic Graph API reference](https://api.semanticscholar.org/api-docs/graph): `/paper/search` (up to 1,000 relevance-ranked results), `/paper/search/bulk` (Boolean), `/snippet/search` (passages of about 500 words from title, abstract and body), and the `publicationTypes`, `minCitationCount` and `venue` filters.
8. [Exa search API reference](https://docs.exa.ai/reference/search): "The search endpoint lets you search the web"; the `publication` category covers research papers, preprints and journal articles.
9. [About Tavily](https://docs.tavily.com/documentation/about): "the web layer for AI agents".


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.