Skip to main content
Consensus has published two blind evaluations of how relevant its search results are. Both used real, anonymized user queries and independent PhD-level judges who didn’t know which system returned each paper. Both measured the Consensus product rather than the APIs on the other comparison pages, so neither ranks PubMed, Europe PMC, OpenAlex or Semantic Scholar. To compare providers on your own domain, run a benchmark like the one Phylo ran before choosing Consensus.

​
Consensus vs Google Scholar

Published 5 December 20251. What was tested. Consensus Quick search, the single-pass ranked search, against Google Scholar. Consensus’s agentic modes weren’t part of this test. Method.
  • 500 real Consensus user queries, randomly sampled and anonymized, ranging from keyword lookups to natural-language questions.
  • The top 10 results from each system, about 10,000 query–paper pairs.
  • Three independent PhD researchers, with no affiliation to Consensus and no knowledge of which system returned each paper, scored every pair from 0 (irrelevant) to 4 (perfectly relevant) on its title and abstract.
Results. Consensus also had higher average relevance at every rank from 1 to 10. Published 4 August 20262. What was tested. Consensus Pro, the research agent that the Threads API runs in pro mode, against GPT-5.5 and Opus 4.8, each with web search enabled and given the same query. Method.
  • 100 real Consensus user queries, randomly sampled and anonymized.
  • The top 20 results from each system.
  • Four independent PhD-level researchers, blind to the source, scored each pair from 0 to 4. LLM-as-a-judge across several frontier models gave a second, independent signal.
  • Precision is the share of results scored 3 or 4. Latency is wall-clock time from query to a complete set of results.
Results. Consensus was more accurate than both frontier models, and far faster. Against GPT-5.5 the precision gain is modest, 1.8% in the top 10, but Consensus returned its results 8.6 times faster. Against Opus 4.8 it was both 11.6% more precise and 11.8 times faster. Even comparing Consensus’s slower queries with the frontier models’ faster ones, at the 75th percentile, Consensus was 3.8 and 5.1 times faster. The frontier models got there by brute force, running many more iterations and queries per result, so each answer also cost far more compute. Query by query, Consensus returned the most relevant results on 59 of the 100 queries, GPT-5.5 on 22 and Opus 4.8 on 6, with 11 ties. Consensus held relevance better deeper into the list: between rank 10 and rank 20 its precision fell 8.7 points, against 10.3 for GPT-5.5.

​
What these evaluations don’t show

  • Relevance is textual. Judges scored how well a paper matches the query, not its recency, citations or journal quality.
  • Judging used title and abstract, because some systems restrict full-text access.
  • The frontier models had no budget. They could spend as much time and as many tokens as they wanted on each query. Consensus returned more relevant results in a fraction of the time and compute. Time was measured directly; tokens and dollars weren’t.

​
Run your own benchmark

The most convincing evaluation is one on your own queries. Benchmark literature providers against your own ground truth walks through it: curate queries across your domain, list the papers an expert would insist on, and measure how many each provider surfaces in its top 20.

How search works

Hybrid search, ranking and reranking, step by step.

Compare scholarly APIs

Why teams choose Consensus, and every API at a glance.

​
Sources

Facts about other providers were checked against their official documentation on 5 November 2026. Limits and prices change, so check the source before you build on a number.
  1. Consensus Outperforms Google Scholar for Academic Search Retrieval, Consensus blog, 5 December 2025.
  2. Consensus Beats Claude and ChatGPT in Academic Search, Consensus blog, 4 August 2026.