Retrieval for RAG and AI agents

A RAG pipeline or research agent is only as good as what its retrieval step puts in the context window. For scientific questions, that means four things from a search API:

The right papers in the top few results for a question written in plain language, because agents ask in sentences.
Full-text snippets, not just titles, so the model reads the passage that supports a claim.
Metadata to weigh the evidence, so the model can prefer a meta-analysis to a case report.
Stable identifiers, so every claim the model makes stays citable.

How the APIs compare for RAG

	Plain-language queries	Text for the context window
Consensus	Hybrid semantic and keyword search Ranked for scientific relevance	Up to 3 query-relevant passages per paper, labeled by section (paid plans) Abstract and a one-sentence takeaway
PubMed	Term matching with MeSH mapping¹	Abstracts. Full text lives in PubMed Central²
Europe PMC	Keyword search over abstracts and full text³	Whole open-access articles as XML, to chunk yourself⁴
OpenAlex	Semantic search, up to 50 results per query⁵	Whole open-access PDFs and TEI XML, to chunk yourself⁶
Semantic Scholar	Relevance search and snippet search⁷	Passages of about 500 words through snippet search⁷

Full-text snippets, ready for a prompt

Add include_full_text_chunks=true to a Consensus search on a paid plan. Each eligible result gains up to 3 passages from the body of the paper, chosen for your query, each starting with the section it came from. You get the doses, methods and measured values that abstracts leave out, without downloading and chunking PDFs.

Python

import os
import requests

resp = requests.get(
    "https://api.consensus.app/v1/search",
    headers={"x-api-key": os.environ["CONSENSUS_API_KEY"]},
    params={"query": "what dose of creatine improves working memory", "include_full_text_chunks": True},
    timeout=30,
)
resp.raise_for_status()

context = []
for paper in resp.json()["results"]:
    evidence = paper["full_text_chunks"] or [paper["abstract"]]
    for passage in evidence:
        context.append(f"[{paper['doi']}] ({paper['study_type']}, n={paper['sample_size']}) {passage}")

Snippets come from open-access papers. Partner full text improves which papers rank, but isn’t returned. When a paper has no snippet, fall back to its abstract and takeaway. See Full text in the API.

Or let Consensus run the agent

If you’d rather not build the retrieval loop, the Threads API runs the Consensus research agent for you. It does more than search: it plans a multi-step strategy, crawls citations backward and forward from key papers, looks up authors, finds specific papers and reads them in full. It returns a written answer with numbered citations and the quotes it relied on.

pro reads about 20 papers. Use it for most questions.
deep reads about 50 papers and writes a structured report. It needs a paid plan.

For assistants and agents that speak MCP, the Consensus MCP server offers the same search and research agent as tools.

Web search APIs vs a scholarly search API

Exa and Tavily are web search APIs built for AI agents. Exa’s search endpoint “lets you search the web”, and offers a publication category for research papers and journal articles⁸. Tavily calls itself “the web layer for AI agents”⁹. They’re the right tools for news, company pages and the open web. The biggest difference is full text. Consensus searches the full text of papers from 13 publisher partners, including 7 of the 10 largest academic publishers, as well as every open-access paper, and hands your agent the passage from the paper that supports the point. That text isn’t on the open web, so a web index can’t search it. Search results carry snippets from open-access papers, and the research agent’s citations carry the quotes it relied on. A scholarly API also gives your agent:

A defined corpus of papers, each with a DOI, journal and year, instead of a mix of papers, press releases and blog posts.
Evidence metadata, so the agent can tell a randomized trial from a case report and a peer-reviewed paper from a preprint.
Retraction handling. Consensus marks retracted papers and never uses them in its AI answers.

Many teams give their agent both: web search for current events and general facts, Consensus for anything that needs a citation to the literature.

In production

Owkin replaced a hand-maintained set of PubMed abstracts with the Consensus MCP server inside its AI scientist, and got 25x the paper coverage. See Swap a hand-maintained paper corpus.
Phylo’s agents fan each biology question out into parallel Consensus API searches to screen about 20,000 drug targets. See Parallel search angles.

Add an evidence layer to an agent

Give a working agent peer-reviewed literature as a tool.

Compare scholarly APIs

Why teams choose Consensus, and every API at a glance.

Sources

Facts about other providers were checked against their official documentation on 5 November 2026. Limits and prices change, so check the source before you build on a number.

PubMed User Guide: Boolean operators, field tags, MeSH and Best Match sorting.
About PMC: “more than 10 million full-text article records”.
About Europe PMC: 49.2 million abstracts and 12.5 million full-text articles; searches abstracts and full text; searches can be limited to clinical trials, preprints or article sections such as Methods; free to anyone.
Europe PMC developer resources: REST, SOAP, OAI-PMH and FTP access to open-access content and metadata; automated bulk download of other content isn’t allowed. The Articles RESTful API includes /{id}/fullTextXML.
OpenAlex semantic search: embeds title, abstract and venue name; up to 50 results per query; 1 request a second; 17 supported filters, and others such as cited_by_count return a 400 error.
OpenAlex full text: cached PDF and TEI XML for more than 60 million open-access works.
Semantic Scholar Academic Graph API reference: /paper/search (up to 1,000 relevance-ranked results), /paper/search/bulk (Boolean), /snippet/search (passages of about 500 words from title, abstract and body), and the publicationTypes, minCitationCount and venue filters.
Exa search API reference: “The search endpoint lets you search the web”; the publication category covers research papers, preprints and journal articles.
About Tavily: “the web layer for AI agents”.

​How the APIs compare for RAG

​Full-text snippets, ready for a prompt

​Or let Consensus run the agent

​Web search APIs vs a scholarly search API

​In production

​Related