Web Crawler API — IteraTools

About this tool

Crawl a website using breadth-first search (BFS) and extract structured content — title, markdown, and links — from multiple pages in a single API call. Ideal for documentation scraping, site analysis, and content extraction at scale.

✅ Up to 20 pages per job
✅ Same-domain filtering (default on)
✅ Include/exclude URL patterns (regex)
✅ Returns title + markdown + outbound links per page
✅ 60s timeout, skips failed pages gracefully

Quick Start

curl -X POST https://api.iteratools.com/crawl \
  -H "Authorization: Bearer YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com","max_pages":2}'

Parameters

Parameter	Type	Default	Description
`url` *	string	—	Starting URL to crawl
`max_pages`	integer	5	Max pages to crawl (1–20)
`same_domain`	boolean	true	Only follow links on the same domain
`include_pattern`	string	null	Regex: only crawl matching URLs
`exclude_pattern`	string	null	Regex: skip matching URLs

Response

{
  "ok": true,
  "data": {
    "pages": [
      {
        "url": "https://example.com",
        "title": "Example Domain",
        "markdown": "# Example Domain\n\nThis domain is for use in illustrative examples...",
        "links": ["https://www.iana.org/domains/reserved"]
      }
    ],
    "total": 1,
    "crawl_time_ms": 1423
  }
}

Pricing

$0.010 per job (up to 20 pages included) via x402 micropayment on Base (USDC).