Set the crawl boundary#

Scope is the boundary around a crawl. A site links outward constantly: to a documentation host, a status page, a content delivery network, a social profile. Without a boundary, the crawler follows all of it.

By default, the boundary is the host you seeded with. The crawler visits nothing else, so an empty scope is safe rather than unlimited.

Widen with include rules#

Add an include rule to widen the boundary. Each rule has two parts: a host suffix and a path prefix. You can set either part, or both.

The host suffix matches by suffix, not exactly, so example.com also admits api.example.com and staging.example.com. Use it to pull a whole estate into one crawl.

The path prefix restricts to a branch of the URL path, such as /app. A rule with both parts set requires both to match.

Any include rule replaces the default. A URL must then match at least one rule before the crawler visits it. If you add a rule for a second host but not for the original, the seed host falls out of scope. Add a rule for each host you want.

Narrow with exclude rules#

Exclude rules take precedence over includes. The crawler skips a URL that matches any exclude rule, even if an include rule also matches it.

Use them for the parts of a site that waste a crawl or cause harm.

• Logout endpoints: end the session and stop the crawl from reaching anything behind it.
• Search and filter pages: generate unlimited URL combinations.
• Export and report endpoints: are slow and expensive to serve.

Exclude the logout path on any authenticated crawl. It is the single most common reason a crawl of a logged-in application stops early.

Honour robots.txt#

Honor robots.txt is ticked by default. The crawler then respects the Disallow rules the target publishes.

Leave it on unless you own the target, or you hold written permission that covers it. If you turn it off, you deliberately visit paths that the site asks automated clients to stay away from. Sites often exclude those paths because they are expensive, destructive, or sensitive.

This setting is separate from Parse robots.txt in the Discovery section. Parse robots.txt reads the file to find the sitemaps it references. Honor robots.txt obeys the file's restrictions. You can read the file for discovery and still respect it, and that is the default.