> ## Documentation Index
> Fetch the complete documentation index at: https://docs.consensus.app/llms.txt
> Use this file to discover all available pages before exploring further.

# Track new papers across many topics on a schedule

> Sweep hundreds of topics on a cron, diff against the last run, and report only what changed.

## What you get

An unattended job that re-runs a fixed query set on a schedule, stores what it has seen, and produces a delta — the papers that are new since last time, ranked by whether they change a decision you have already made.

One research team runs this across 121 countries every six hours to track education-reform literature, feeding a public site. The shape generalizes to any entity list: compounds, competitors, materials, indications, regions.

## Who it's for

Engineers and technical researchers maintaining a living evidence base rather than answering a one-off question. Runs from Codex, Claude Code, a GitHub Action, or any scheduler.

## How it works

<Steps>
  <Step title="Fix the entity list and the query template">
    Separate the two. One template — "education reform outcomes in {entity}" — crossed with an entity list is far easier to maintain and audit than hundreds of hand-written queries.
  </Step>

  <Step title="Window every run">
    Set `year_min` and `month_min` to the last successful run rather than filtering client-side. You retrieve less, pay for less, and get a cleaner set.
  </Step>

  <Step title="Paginate to exhaustion per entity">
    Walk `page` from `0` while `is_end` is false. Pagination runs to page 49 and `page_size` caps at your plan's maximum, so one query can reach tens of thousands of records.
  </Step>

  <Step title="Diff on DOI against stored state">
    Persist every DOI you have reported. The run's output is the set difference, not the search results.
  </Step>

  <Step title="Log the empty runs too">
    A run that finds nothing is a record you need — it distinguishes "no new evidence" from "the job silently broke three weeks ago."
  </Step>
</Steps>

## The pipeline

```python theme={null}
import os, time, requests

API = "https://api.consensus.app/v1/search"
HEADERS = {"x-api-key": os.environ["CONSENSUS_API_KEY"]}

STRATEGY = dict(
    study_types=["rct", "meta-analysis", "systematic review"],
    exclude_preprints=False,   # early signals often appear as preprints first
    page_size=200,
)

def sweep(query, since):
    """Every record for one query since the last run."""
    page, out = 0, []
    while page <= 49:                         # 0-indexed; 49 is the ceiling
        resp = requests.get(API, headers=HEADERS, params={
            "query": query,
            "year_min": since.year,
            "month_min": since.month,
            "page": page,
            **STRATEGY,
        })
        if resp.status_code == 429:           # rate limited — back off and retry
            time.sleep(int(resp.headers.get("retry-after", 30)))
            continue
        resp.raise_for_status()
        body = resp.json()
        out.extend(body["results"])
        if body["is_end"] or body.get("next_page") is None:
            break
        page = body["next_page"]
    return out

def run(entities, template, since, seen_dois):
    new, counts = {}, {}
    for entity in entities:
        results = sweep(template.format(entity=entity), since)
        counts[entity] = len(results)
        for paper in results:
            doi = paper.get("doi")
            if doi and doi not in seen_dois:
                new[doi] = {**paper, "entity": entity}
    return new, counts        # persist counts even when new is empty
```

<Note>
  Pagination and page sizes above the default of 20 require a paid plan or an Enterprise API key. `page_size` caps at 300 on Pro and Teams, 750 on Deep, and higher on Enterprise keys. A request over your ceiling is clamped rather than rejected — read the `page_size` echoed back in the response.
</Note>

## Rate limits and backoff

Build backoff in before you widen the entity list, not after.

* **Pace to one search per second.** That is the limit on every plan except Enterprise, so build the pacing into the pipeline rather than firing entities in parallel. Honour `retry-after` and add jitter between entities.
* A **429 mentioning your monthly search allowance** is a usage limit, not a rate limit — retrying will not help. Turn on additional usage in your account settings to keep searching past the included allowance.
* Scheduled runs consume searches like any other request. A 121-entity sweep every six hours is roughly 484 entity-sweeps a day before pagination; size your plan for it.

## Reporting the delta

Rank the new papers by whether they change a decision, not by citation count — a paper published this week has none yet. A useful report is five items with one line each and a link, plus an explicit "nothing notable this run" state that actually gets used.

## What to check before you trust it

* **Alert on the job, not just the findings.** A silent pipeline looks identical to a quiet literature. Persist per-run counts and alert when a run returns zero results across *all* entities.
* **Version the query template.** When you change it, the delta becomes incomparable to previous runs. Record the template version alongside each run.
* **Watch for entity-name collisions.** Short or ambiguous entity names pull in unrelated work. Spot-check the highest-volume entities.
* **Cache if you re-query.** Heavy users hit this fast; cache on the full parameter set with a short TTL.

## Related

<CardGroup cols={2}>
  <Card title="Monitor new research about a drug, device, or brand" icon="satellite-dish" href="/use-cases/brand-and-device-monitoring">
    The same machinery, pointed at a commercial watchlist.
  </Card>

  <Card title="Build a reusable literature-review workflow" icon="wand-magic-sparkles" href="/use-cases/literature-review-skill">
    The interactive version of the same method.
  </Card>

  <Card title="Best practices" icon="shapes" href="/use-cases/best-practices">
    Exhaustive sweep and diff-against-last-run in detail.
  </Card>

  <Card title="All use cases" icon="grid-2" href="/use-cases">
    Browse the gallery by persona.
  </Card>
</CardGroup>