Built for production crawling at any scale

Most crawlers were built before LLMs existed. They were designed to extract data into spreadsheets, monitor prices, or feed search engines that ran on keywords. crawlcrawl was designed around a different question: what does a modern data pipeline actually need from a crawler, and how do we deliver it in one call that does not require glue code on the customer side.

The answer is the crawler endpoint below. You provide a starting URL and a few constraints; we discover the link graph, fetch every page within your limits, render JavaScript transparently when it is needed, strip boilerplate, and return clean markdown plus structured signals. The dataset persists for retrieval, comparison, and re-ingestion. If a page needs anti-bot routing through our global network across 190+ countries, that happens without a separate API call. The price you see at signup is the price you pay; every feature is included at every paid tier.

Start a crawl

One POST. We queue the job, follow the link graph up to your max_pages limit, and return clean markdown plus metadata for every page found.

curl -X POST https://api.crawlcrawl.com/v1/crawls \
  -H "Authorization: Bearer crk_..." \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://docs.example.com",
    "max_pages": 100,
    "respect_robots": true
  }'

# returns
{ "id": 504, "status": "queued", "url": "https://docs.example.com" }

The response is a run identifier. The job runs asynchronously while your application stays responsive. You can poll for status, subscribe to a webhook for completion, or retrieve results once the run finishes. For continuous workloads, pair the same call with a cron expression and the crawl repeats on schedule indefinitely.

Aggressive mode for owned sites

For full-site rebuilds where you own the target (your own documentation, your own product catalog, your own marketing site), aggressive mode lifts concurrency, reduces inter-request delay, and skips robots.txt politeness checks. The result is meaningfully faster completion on sites where you have authority to crawl quickly.

curl -X POST https://api.crawlcrawl.com/v1/crawls \
  -H "Authorization: Bearer crk_..." \
  -d '{"url":"https://your-site.com","max_pages":500,"aggressive":true}'

Read what came back

Every crawl produces a retrievable dataset. The pages, the link graph, and the orphan list are all queryable through dedicated endpoints, so your pipeline can paginate, filter, or stream as needed.

GET /v1/crawls/{id}             → status + page count + timings
GET /v1/crawls/{id}/pages       → every page's markdown + meta
GET /v1/crawls/{id}/links       → per-page outbound link graph
GET /v1/crawls/{id}/orphans     → pages with no incoming internal links
GET /v1/crawls/{old}/diff/{new} → change-detection diff between two runs

The diff endpoint is the feature most production teams reach for first. Pass two crawl IDs of the same site (say, yesterday's and today's), and the response returns pages added, pages removed, and pages whose markdown content hash shifted. Your downstream pipeline indexes only the deltas rather than the entire corpus, which typically cuts re-ingestion cost by 80 to 95 percent on continuous content monitoring workloads.

What you get in each page response

Every page in a crawl response includes the markdown, the page metadata, and the structured-data signals that the source author already wrote into the HTML. Title, description, canonical URL, Open Graph tags, JSON-LD blocks, schema.org types, hreflang relationships, and the heading tree all arrive together. There is no second API call required to enrich a page for RAG ingestion.

markdown — boilerplate-stripped, heading-preserving, table-and-code-friendly output
metadata — title, description, OG, Twitter card, canonical, robots-meta, hreflang
schema — extracted JSON-LD blocks parsed and typed (Article, Product, FAQPage, HowTo, Organization, and more)
links — outbound link list with anchor text
headings — heading tree for chunking
content_hash — stable hash for change detection across runs

Use cases that fit the Crawler endpoint

Teams use the crawl endpoint for several distinct workloads. The most common are RAG ingestion (refreshing the corpus a vector database serves), content monitoring (alerting on pricing-page changes, regulatory updates, competitive feeds), and SEO audits (orphan detection, broken link discovery, weekly site diff for a portfolio of client sites).

What ties these workloads together is the multi-page shape. Single-URL scraping APIs cover a single fetch; the crawl endpoint covers the orchestration around it: depth, link discovery, dataset persistence, and change detection over time. Once a team is running production workloads at any non-trivial scale, the crawl endpoint is where most of the volume lives.

"We index documentation across forty vendor sites every week. crawlcrawl made it boring infrastructure, and that is the highest compliment I can give a tool."
— Amit Tanwar, Founder, Networkers Home

Pricing for crawls

Each page fetched counts as one credit. JavaScript rendering, structured-data extraction, and global routing are included; they do not multiply the credit cost. A 100-page crawl costs 100 credits on the canonical plan. The free tier covers 1,000 credits per month with no card required. Pro at $8/mo covers 5,000 pages. Studio at $42 covers 100,000. Agency at $167 covers 500,000. Scale at $300 covers 1,000,000. Every tier is exactly half of Firecrawl.

See full pricing →

Crawler.