TL;DR
DocPull 5.0 is live on PyPI and GitHub. The release turns page capture into local-first evidence collection for agents. It collects context from web sources, keeps the source files behind each answer, and makes every switch to a provider or cloud browser explicit.
Start with local capture, check the site’s open discovery signals, then bring your own key. DocPull enforces that order across the CLI, Python SDK, policy files, providers, rendering routes, benchmarks, and MCP tools.
$0.
Budget boundary
Set --budget 0 to block paid-capable routes.
docpull https://docs.python.org/3/ --budget 0 -o ./evidenceWith --budget 0, DocPull blocks paid-capable provider and cloud routes before they run. It still allows cache, direct HTTP, sitemap discovery, local extraction, indexing, pack analysis, monitors, and local browser rendering.
You can still choose Tavily, Exa, Parallel, Vercel Sandbox, or E2B. Instead of relying on rules scattered across scripts, DocPull checks one budget policy before each run.
| Route class | Under --budget 0 | Examples |
|---|---|---|
| Local routes | Allowed | Cache, HTTP, discovery, extraction, indexing, local render |
| Paid-capable routes | Blocked before execution | Tavily, Exa, Parallel, Vercel Sandbox, E2B |
Discover before search
Many sites already publish useful discovery signals: a site-level llms.txt file, RSS or Atom feeds, OpenAPI references, sitemap indexes, or public documentation trees on GitHub.
docpull discover scan https://docs.python.org/3/ -o ./packs/discoveryThe scanner writes candidate_sources.ndjson in the same format used by provider imports. Local and bring-your-own-key discovery produce the same input for later steps.
Start with the signals a site already publishes, review the candidates, and pay for broader discovery only when those signals are not enough.
Evidence formats and OKF
A hosted response may answer a question today. A local evidence pack gives you files to inspect tomorrow.
DocPull stores sources in readable, searchable formats: Markdown with frontmatter, streamed or chunked NDJSON, SQLite with FTS5 search, Google Open Knowledge Format bundles, cached archives and mirrors, source metadata, manifests, indexes, and context packs for agents.
| Output shape | Preserves | Use when |
|---|---|---|
| Markdown + frontmatter | Readable source snapshots | Review, cite, diff, commit |
| NDJSON / JSONL | Page, chunk, candidate, and run records | RAG, agents, warehouses, evals |
| SQLite + FTS5 | Searchable local corpus | Offline search, QA, debugging |
| Google OKF | Portable Markdown/YAML bundle | Agent and human knowledge exchange |
| Archives + mirrors | Cached pages and snapshots | Reports, refreshes, stale checks |
| Metadata + manifests | URLs, route steps, timestamps, hashes | Provenance, accounting, audits |
| Context packs + skills | Curated sources and retrieval hints | Codex, MCP, agent workflows |
| Downstream outputs | Small exports with source trails | Sheets, n8n, Vercel AI, CrewAI, warehouses, launch media |
Choose OKF when you need a portable bundle. It writes Markdown concept files with YAML frontmatter, generated directory indexes, and the DocPull manifest. The result remains readable without a custom client.
docpull https://example.com --format okf -o ./site-okfThe format is opt-in. A conformance smoke test generated five bundles from public sites with zero output-shape errors. The test checks packaging only. It does not measure retrieval quality or answer accuracy.
The same source pack can feed other tools: CSV or TSV for review, NDJSON or Parquet for analytics, app context for Vercel AI and CrewAI, and media built from those sources, such as the clips in this article.
Agents can read the files, people can review them, and RAG systems can index them. Teams can cite the files in reports, compare them with newer captures, audit stale sources, and refresh them when the original pages change.
Keep the receipt
DocPull can add an accounting artifact to any run with a budget or paid-capable route: run.accounting.json.
It records non-secret route and cost data: budget limits, estimated cost, actual cost when known, and paid request counts. It also logs local browser time, HTTP requests, cache hits, blocked actions, and route steps.
The source files show which evidence DocPull collected. The accounting file shows how DocPull collected it. Together, they help teams trace a suspect answer, a failed refresh, or an unexpected bill.
Browser only when needed
Not every page needs a browser. DocPull can often capture static and server-rendered pages through direct HTTP and framework-aware extraction. That remains the default path.
docpull render https://example.com --runtime local --budget 0Some pages expose useful content only after JavaScript runs. For those cases, v5 adds a local renderer through an external, agent-browser-compatible CLI. The base package stays browser-free unless you choose the renderer. Cloud rendering is also opt-in, and --budget 0 blocks it.
Measure the free path
A zero-cost default matters only if it completes the task. DocPull 5.0 measures how often it does.
Run the zero-dollar benchmark:
docpull benchmark quick --zero-dollar --target-set zero-dollar --provider allThe benchmark assigns each target to one class: complete_for_0, complete_with_local_browser, partial_for_0, requires_provider, requires_cloud_browser, or blocked_by_policy.
Maintainers can now track one clear target: complete more tasks locally before adding another provider.
Provider routing benchmark
A fixed provider matrix compared DocPull core, Parallel Search, Parallel Context, Tavily Search + Extract, and Exa Search Contents. The benchmark ran each provider-target pair three times under the same crawl, search, and extraction limits.
| Measure | Result | What it means |
|---|---|---|
| Matrix | 8 targets × 5 routes × 3 runs | 120 runs across tool docs, a JavaScript-heavy site, archived docs, and a pricing page. |
| Direct crawl | Best or tied on 5 of 8 targets | For known URLs, direct crawl performed best or tied without paid discovery or extraction. |
| Reliability | 100% pass³ @80; 88% @90 | DocPull core met the fixed test thresholds most often. |
| Provider spend | $0.470 including credits | In this matrix, Parallel and Exa helped most with discovery, freshness, and archives. |
The results support this routing rule: crawl directly when you know the URL. Add a provider when you need discovery, fresh results, or archived pages. The weighted scores are a June 2026 snapshot based on heuristic coverage, cleanliness, source fidelity, freshness, and density. They do not rank providers for every setting.
docpull benchmark quick --target-set provider-matrix --provider all --max-pages 8 --max-depth 1 --max-search-results 5 --extract-limit 2 --max-estimated-cost 5.00Escalation ladder
Local-first does not mean local-only. Providers and cloud browsers can help when the local path falls short. DocPull 5.0 makes that boundary visible before a run spends money or sends work to external infrastructure.
| Route | Use when | What you keep | Cost / boundary |
|---|---|---|---|
| DocPull core | Public or authorized static and server-rendered pages, OpenAPI specs, feeds, filings, and vendor pages. | Markdown, NDJSON, metadata, manifests, indexes, evidence packs. | Works under --budget 0. No provider key. |
| DocPull local render | Direct HTTP returns a JavaScript shell, but local browser rendering can expose the content. | Rendered HTML via agent-browser plus DocPull conversion artifacts. | Works under --budget 0. Requires a local runtime. |
| Tavily | Fast web search, site mapping, broad discovery before local fetch. | Candidate sources or provider context DocPull can normalize. | Paid-capable. Dry-run first; --budget 0 blocks live runs. |
| Exa | Keyword search misses the intent, related pages, competitors, or long-tail sources. | Relevant URLs and extracts for downstream evidence packs. | Paid-capable. Use when keyword search falls short. |
| Parallel | The task needs live-web search, extraction, and context from multiple sources. | Context packs ready for DocPull scoring, citations, entities, and briefs. | Paid-capable. Escalate after local/open discovery. |
| Vercel Sandbox | The task needs isolated browser execution and cannot use the local renderer. | Same agent-browser JSON contract as local rendering. | Cloud route. Choose it explicitly; --budget 0 blocks it. |
| E2B Sandbox | The task needs an API-keyed sandbox, a prebuilt template, or file-based result transport. | Sandboxed render payload for DocPull conversion and artifacts. | Cloud route. Choose it explicitly; --budget 0 blocks it. |
| Full browser automation | Need clicks, login state, app workflows, CAPTCHA walls, or private dashboards. | Export rendered HTML or content, then pass it into the evidence pipeline. | Choose this route explicitly for interactive tasks. |
The intended escalation order is:
- Try open discovery: sitemaps, feeds, llms.txt, OpenAPI references, and public docs repositories.
- Try local rendering when direct HTTP only returns a shell.
- Dry-run a BYOK provider route before making a live request.
- Use a live provider only when its search or extraction justifies the external request.
- Use cloud rendering only when local infrastructure cannot complete the task.
Ask one question: what is the lowest-cost route that can complete this task and preserve its evidence?
Who it is for
DocPull 5.0 is for teams that need current web context and the source files behind it.
| Audience | Why it matters |
|---|---|
| Agent builders | Need fresh context with a source trail behind the answer. |
| RAG teams | Want durable, re-indexable source packs outside the vector database. |
| Researchers | Need local corpora they can cite, inspect, compare, and refresh. |
| Cost-conscious teams | Need budget boundaries before execution, not after a bill arrives. |
| Developers | Want local control while keeping providers available when they are useful. |
Limits and tradeoffs
DocPull will not capture every website locally. It does not provide a proprietary web-scale index or use stealth scraping and CAPTCHA bypass as hidden fallbacks. Use a paid provider or cloud browser when the local path falls short.
Try it
Install or upgrade:
pip install -U docpullThen try one zero-dollar capture:
docpull https://www.python.org/blogs/ --single --budget 0 -o ./python-newsThe first pass needs no provider key. Run with --budget 0, inspect the files, then decide whether to add a provider or cloud browser.
Start locally, use open signals first, escalate with evidence, and keep the receipt.