Set crawl limits and pace#

This panel answers two questions: when should the crawl stop, and how hard should it push while it runs. The limits protect you from a crawl that never ends. The timing settings protect the target from a crawl that overwhelms it.

Select a crawl strategy#

Three strategies trade speed against fidelity.

• Fast: makes direct requests for each URL.
• Balanced: the default.
• Thorough: reproduces navigation state.

Navigation state matters on an application where a page behaves correctly only after the steps that precede it. Choose Thorough for a stateful application where a Fast crawl returns errors or login pages instead of content. Otherwise leave it alone.

When the crawl stops#

Four limits end a crawl, whichever is reached first.

Setting	Default	What it counts
Max depth	10	Navigation steps from the seed URL
Max locations	2000	Distinct URLs recorded
Max requests	10000	Total HTTP requests sent
Time limit	0, meaning none	Seconds of wall-clock time

Set max locations, max requests, or the time limit to 0 to remove that limit. Max depth has no such option, and cannot go below 1.

Depth is the limit to understand first. It counts steps away from the seed, not directory levels in the URL, so a page linked from the home page is depth 1 however deep its path looks. Ten steps reaches almost everything on a normal site. Lower it to two or three for a broad, shallow picture of a large application.

The location and request ceilings exist because a crawl can generate URLs faster than it consumes them. A calendar, a search form, or a paginated list with sort parameters produces new URLs without end. A crawl that completes on one of these limits, rather than on running out of links, has not finished mapping the site: the limit cut it off. Tighten scope rather than raise the ceiling.

Set concurrency and delay#

Three settings decide the load the target sees.

• Max concurrency: caps requests in flight at once across all hosts. Default 8.
• Per-host delay: sets the minimum gap between consecutive requests to the same host. Default 100 milliseconds.
• Request timeout: abandons an individual request that hangs. Default 30 seconds.

The defaults are polite. They suit a production system you do not control, and they match what a well-behaved search engine crawler does.

Against a test system you own, raise concurrency and drop the delay to zero to finish a crawl in a fraction of the time. Do not do this against anything shared or in production. A crawl that puts a site under load is an outage you caused, and it looks like a denial-of-service attempt from the other side.

Lower the request timeout when a few slow endpoints hold up an otherwise quick crawl.

Three worked combinations#

For a first look at an unfamiliar site, lower max depth to 3 and leave everything else alone. You get the shape of the application in a few minutes, and can then decide what deserves a full crawl.

For a complete map of a site you own, raise max locations, set the delay to 0, raise concurrency, and leave depth at its default.

For a crawl against production, change nothing except perhaps a time limit, so it cannot run longer than you intend.