Search indexing
Crawl and index entire sites to power search, RAG pipelines, and internal knowledge bases.
Web crawler that reaches every page
The web wasn't built for agents. Browserbase gives your web crawler real cloud browsers that render JavaScript, follow links across a site, and reach pages that turn away ordinary tooling. Describe what you need and the Stagehand SDK returns it as structured data, with no per-site tuning.
The Problem
Crawling the modern web is harder than it looks
- Static crawlers miss content on JavaScript-rendered and single-page sites.
- Bot detection blocks scripted requests within minutes of a crawl starting.
- Broken CSS selectors and custom parsers need constant maintenance per site.
- Rate limits and IP bans stall large crawls partway through.
- Infinite scroll, pagination, and login walls hide the data you actually need.
The Solution
How Browserbase powers a reliable web crawler
- Real browser rendering: full Chrome instances execute JavaScript, load SPAs, and handle infinite scroll and pagination.
- Fetch for speed: pull static pages over lightweight HTTP and fall back to a full browser only when a page needs it.
- Natural language extraction: describe the data you need in plain English. The Stagehand SDK finds and returns it as structured JSON.
- Self-healing selectors: when a site changes its layout, the crawler re-identifies the right elements automatically.
- Verified: every session reaches pages that turn away ordinary tooling, with residential proxies and managed CAPTCHA solving included.
- Agent Identity: Web Bot Auth signs your agent's requests, so sites can verify it instead of guessing.
- Parallel at scale: run thousands of browser sessions simultaneously to crawl large sites fast.
What you can build with a web crawler
Competitive intelligence
Crawl competitor catalogs, pricing, and content across hundreds of sites on a schedule.
Content aggregation
Follow links to collect articles, listings, and user-generated content from dynamic sources.
Site auditing
Crawl your own properties to check links, metadata, and accessibility at scale.
Frequently Asked Questions
What is a web crawler?
A web crawler is software that systematically browses websites, following links from page to page to collect content. Browserbase runs crawlers on real cloud browsers, so they render JavaScript and reach pages that static crawlers miss.
How does a web crawler work?
It starts from one or more URLs, loads each page, extracts the data and the links, then queues those links to visit next. On Browserbase, each page loads in a full Chrome session, so dynamic content renders exactly as a person would see it.
Do I need a full browser for every page?
No. Use Fetch for lightweight HTTP retrieval on static pages, and fall back to a full browser session for JavaScript-heavy ones. The smart-fetch-scraper template does exactly this, so large crawls stay fast and cost-efficient.
How do I build a web crawler?
With the Stagehand SDK in TypeScript or Python you describe the data you want and define a schema, and the crawler returns typed JSON. Browserbase handles the browsers, rendering, and access, so you focus on the crawl logic, not the infrastructure.
Is web crawling legal?
Crawling publicly available data is generally permitted, but you are responsible for respecting each site's terms and applicable laws. Browserbase gives your agent verified access through Agent Identity and Web Bot Auth, so sites can identify your crawler instead of guessing.
What sites can a Browserbase web crawler reach?
Any site a person can visit, including JavaScript-heavy single-page apps and pages behind login walls using persistent contexts. Verified access and residential proxies come with every session.
Can I crawl thousands of pages at once?
Yes. You can run thousands of browser sessions in parallel in the cloud and write results straight to clean, typed JSON, CSV, or your database.