RAG Documentation Scraper

RAG Documentation Scraper

Turn any documentation site into a vector-database-ready knowledge base in one click.

Crawls documentation sites with Playwright, strips HTML navigation boilerplate, splits content on heading boundaries, and outputs clean Markdown or JSON chunks with embedded source metadata — ready to drop into Pinecone, Chroma, pgvector, or any LangChain / LlamaIndex / Haystack pipeline.

---

Why AI engineers choose this over generic scrapers

Generic scraper output	What you get here
Raw HTML with nav bars, sidebars, cookie banners	Clean text extracted from the main content area only — no noise
One blob per page	Heading-boundary chunks that map naturally to retrieval units
No metadata	Every chunk carries source_url, page_title, section breadcrumb, token_count, content_hash
Re-processes every page on every run	Optional DynamoDB delta tracking skips unchanged pages — up to 80% cost reduction on re-runs
Duplicate content from versioned docs	SHA-256 deduplication silently drops identical chunks regardless of URL

---

Benchmarks

Tested against the Crawlee.dev documentation (production run, 2026-10-18):

• 25 pages crawled → 94 chunks produced, 0 errors
• All chunks under 512 tokens (compatible with text-embedding-3-small, nomic-embed-text, bge-m3)
• Per-page time: ~1.8 s average (Playwright, images/fonts blocked)
• Noise stripped: nav, sidebar, footer, cookie banners, pagination, breadcrumbs, "edit this page" widgets

---

3-line Python quickstart

from apify_client import ApifyClient
chunks = ApifyClient("YOUR_APIFY_TOKEN").actor("YOUR_USERNAME/apify-rag-docs-scraper").call(
    run_input={"startUrls": [{"url": "https://YOUR_DOCS_SITE.com/docs/"}], "chunkSize": 500, "outputFormat": "json-chunks"}
)["defaultDatasetId"]
items = list(ApifyClient("YOUR_APIFY_TOKEN").dataset(chunks).iterate_items())
# items[0] → {"text": "...", "metadata": {"source": "...", "title": "...", "section": "...", "token_count": 487}}

See developer_quickstart/colab_rag_example.ipynb for a full end-to-end walkthrough: actor run → dataset pull → ChromaDB insert → semantic search query.

---

Key features

• 3 RAG-native output formats — markdown (default), json-chunks (Pinecone/Chroma/pgvector ready), text (bare text)
• 4 framework-specific formats — LangChain Document, LlamaIndex TextNode, Haystack Document, raw JSONL
• Flexible chunk size — Set chunkSize (tokens) to control granularity; works with any embedding model
• 4 chunking strategies — heading boundary (recommended), fixed token window with overlap, paragraph, whole-page
• PDF support — basic pdfjs extraction (free) or AWS Textract OCR for scanned/image PDFs
• Incremental crawling — DynamoDB delta tracking; re-runs skip unchanged pages entirely
• Version-aware URL filtering — restrict to a specific version path (e.g. /v2/) with one input field
• MCP-compatible — includes an MCP server wrapper so Claude Desktop and Cursor can call this actor as a live tool

---

Output formats

markdown (default) — generic RAG, vector-DB agnostic

{
  "markdown": "## Quick Start\n\nCrawlee is available as a set of NPM packages...",
  "source": "https://crawlee.dev/docs/introduction",
  "title": "Introduction | Crawlee for JavaScript",
  "section": "Introduction > Quick Start",
  "chunk_index": 0,
  "chunk_total": 12,
  "token_count": 312,
  "content_hash": "c4a7e891f23b5d10",
  "scraped_at": "2026-10-18T14:45:22.341Z"
}

json-chunks — Pinecone / Chroma / pgvector

{
  "text": "Quick Start\n\nCrawlee is available as a set of NPM packages...",
  "metadata": {
    "source": "https://crawlee.dev/docs/introduction",
    "title": "Introduction | Crawlee for JavaScript",
    "section": "Introduction > Quick Start",
    "chunk_index": 0,
    "chunk_total": 12,
    "token_count": 312,
    "last_modified": null,
    "content_hash": "c4a7e891f23b5d10",
    "scraped_at": "2026-10-18T14:45:22.341Z"
  }
}

langchain — drop-in LangChain Document

{
  "page_content": "Quick Start\n\nCrawlee is available as a set of NPM packages...",
  "metadata": {
    "source": "https://crawlee.dev/docs/introduction",
    "heading_path": "Introduction > Quick Start",
    "chunk_index": 0,
    "chunk_total": 12,
    "token_count": 312,
    "content_hash": "c4a7e891f23b5d10"
  }
}

---

Input parameters

Parameter	Type	Default	Description
startUrls	array	—	Required. Root URL(s) to crawl
urlPatterns	string[]	[]	Glob patterns to restrict crawl scope (e.g. /docs/**). Empty = all same-domain URLs
maxDepth	integer	5	Link levels to follow (0 = start pages only)
outputFormat	enum	markdown	markdown, json-chunks, text, langchain, llamaindex, haystack, raw_jsonl
chunkSize	integer	512	Target token count per chunk. 500 works well for most embedding models
chunkStrategy	enum	heading_boundary	heading_boundary, fixed_tokens, paragraph, page
chunkOverlapTokens	integer	50	Token overlap between adjacent chunks (for fixed_tokens strategy)
deduplicateContent	boolean	true	SHA-256 dedup — skip chunks already seen in this run
versionFilter	string	""	Only crawl URLs containing this path fragment (e.g. /v3/)
useDeltaTracking	boolean	false	DynamoDB incremental crawling — re-runs skip unchanged pages
pdfHandling	enum	skip	skip, extract_text, aws_textract
maxRequestsPerCrawl	integer	200	Hard page cap per run. 0 = unlimited
excludeSelectors	string[]	[]	Extra CSS selectors to strip (e.g. .announcement-bar)

---

Pricing

This actor uses Pay-Per-Result pricing — you pay per chunk pushed to the dataset.

Docs site size	Typical chunks	Estimated cost
50 pages	250–400 chunks	~$1.25–$2.00
100 pages	500–800 chunks	~$2.50–$4.00
500 pages	2,500–4,000 chunks	~$12.50–$20.00

Cut costs by up to 80%: Enable useDeltaTracking: true so re-runs only process pages that changed since the last run. Stable documentation sites rarely change more than 10–20% of pages per day.

---

Pre-configured Published Tasks

Five ready-to-run tasks are available in the Apify Store for the most popular documentation niches:

1. LangChain Documentation Scraper — Full LangChain Python docs, Markdown + RAG chunks
2. Stripe API Docs Ingestion Tool — Stripe API reference in json-chunks format for payment bots
3. Docusaurus / ReadTheDocs Crawler — Universal adapter for Docusaurus and ReadTheDocs sites
4. Cursor / Claude MCP Documentation Extractor — MCP spec + Cursor AI docs into structured chunks
5. GitHub Wiki & Markdown Repository Scraper — GitHub Wikis and Markdown repos

Run python scripts/create_published_tasks.py (after apify push) to create these tasks programmatically.

---

AWS Integration (optional)

AWS features — Textract OCR for PDFs and DynamoDB delta tracking — are fully optional. Set these environment variables in Apify Console → Actor Settings → Environment Variables:

Variable	Required for
AWS_ACCESS_KEY_ID	Delta tracking + Textract
AWS_SECRET_ACCESS_KEY	Delta tracking + Textract
AWS_S3_BUCKET	Textract only (temp staging)

The DynamoDB table is created automatically on first use (PAY_PER_REQUEST billing, 90-day TTL).

---

MCP / Agent integration

The repository ships a standalone MCP server (mcp-server/) that wraps the Apify actor as a live tool. Once configured, Claude Desktop, Cursor, or any MCP-compatible AI client can call it directly during a conversation — no API calls or copy-pasting required.

Tool exposed

Tool	Description
scrape_documentation_for_rag	Crawl a documentation URL and return semantically chunked, LLM-ready text with full source metadata. Accepts url, outputFormat, chunkSize, maxPages, maxDepth, urlPattern, and versionFilter.

Prerequisites

# Node 18+ required
git clone https://github.com/LernowaAdmin/rag-docs-scraper.git
cd rag-docs-scraper/mcp-server
npm install
npm run build          # compiles TypeScript → dist/index.js

Claude Desktop (local stdio)

Edit ~/Library/Application Support/Claude/claude_desktop_config.json (macOS) or %APPDATA%\Claude\claude_desktop_config.json (Windows):

{
  "mcpServers": {
    "rag-docs-scraper": {
      "command": "node",
      "args": ["/absolute/path/to/rag-docs-scraper/mcp-server/dist/index.js"],
      "env": {
        "APIFY_TOKEN": "apify_api_xxxxxxxxxxxxxxxxxxxxxxxxxxxx"
      }
    }
  }
}

Restart Claude Desktop. The tool appears automatically in the tool palette.

Cursor (local stdio)

Edit ~/.cursor/mcp.json:

{
  "mcpServers": {
    "rag-docs-scraper": {
      "command": "node",
      "args": ["/absolute/path/to/rag-docs-scraper/mcp-server/dist/index.js"],
      "env": {
        "APIFY_TOKEN": "apify_api_xxxxxxxxxxxxxxxxxxxxxxxxxxxx"
      }
    }
  }
}

Reload Cursor (Cmd/Ctrl + Shift + P → "Reload Window").

Remote / Docker (HTTP + SSE transport)

For hosted deployments, the server supports SSE over HTTP:

cd mcp-server
MCP_TRANSPORT=http MCP_PORT=3001 APIFY_TOKEN=apify_api_... node dist/index.js

Then point your MCP client at http://your-host:3001/sse.

The mcp-server/Dockerfile builds a production-ready image:

cd mcp-server
docker build -t rag-docs-mcp .
docker run -p 3001:3001 -e APIFY_TOKEN=apify_api_... -e MCP_TRANSPORT=http -e MCP_PORT=3001 rag-docs-mcp

Apify-hosted MCP gateway (no local server needed)

Apify exposes every actor's API as an MCP-compatible tool via their MCP gateway. Add this to your client config for a zero-install option:

{
  "mcpServers": {
    "apify": {
      "type": "sse",
      "url": "https://mcp.apify.com/sse",
      "headers": {
        "Authorization": "Bearer apify_api_xxxxxxxxxxxxxxxxxxxxxxxxxxxx"
      }
    }
  }
}

See developer_quickstart/mcp_client_config_example.json for all three variants side by side.

---

FAQ

The actor crawls only one page — why aren't more pages enqueued?

A redirect often changes the URL structure after navigation. Check the actual URL in the logs (it may be /en/docs/ instead of /docs/) and update startUrls and urlPatterns to match the post-redirect path.

Chunks are empty or very short on a JavaScript-rendered site.

The page content may not be rendered when Playwright first loads it. Add the main content selector to excludeSelectors's inverse — or simply let the actor wait by setting navigationTimeoutSecs to a higher value (default: 30 s).

I'm seeing nav menu text in every chunk.

Add the offending element's CSS selector to excludeSelectors (e.g. [".custom-nav", "#site-sidebar"]). The actor already strips the most common patterns automatically.

Versioned docs produce duplicate chunks.

Use versionFilter to restrict crawling to one version (e.g. /v3/), or leave deduplicateContent: true to drop identical chunks regardless of which version URL they came from.