Is Web Crawling Legal? Laws, Risks, and Responsible Crawling Practices

Web crawling is generally lawful when a crawler requests public pages in a controlled way. What changes that answer is everything around the request: access controls you bypass, terms you agreed to, personal data you collect, the load you put on the server, the country you are in, and what you do with the results. There is no universal yes or no.

This is general information, not legal advice. Have a lawyer who knows your jurisdiction review any collection plan that touches third-party systems or personal data. Last reviewed: October 12, 2026.

What "crawling" actually covers

A crawler is software that requests web resources automatically. A scraper collects and structures the returned content for another purpose. A renderer executes JavaScript to see the page a browser would produce. A search engine bot discovers pages, fetches them and stores information about them for an index. (For the SEO side of crawling, see what crawling is in SEO.)

Those labels describe behaviour, not legal status. Fetching a product page anyone can open is very different from entering a customer portal with someone else's credentials or defeating a CAPTCHA. The first can be a routine SEO audit. The second can be unauthorized access.

Crawling, indexing and republishing are also different acts. A search engine can index a title and URL without copying a whole article into a competing database. A scraper that stores full copyrighted pages, personal profiles or confidential records raises different questions. Intended use matters: analysing a site you own carries far less risk than reselling someone else's dataset.

In the United States, the Justice Department's CFAA guidance describes computer-fraud cases as fact-dependent: prosecutors weigh the technology, the sensitivity of the information and how evidence could be gathered lawfully. Other countries apply privacy, database, copyright, contract or computer-misuse rules differently, and a court decision in one jurisdiction does not settle your project in another.

The safest case is the simplest one: a developer crawls the public HTML of their own marketing site to find broken links. They control the server, can read its logs and set the rate. That same developer should stop if a third-party URL redirects into a login page, exposes personal records, or returns an access-denied response.

Where crawling creates legal or operational risk

Risk rises when a crawler crosses a technical boundary, gathers data it does not need, or causes measurable strain. robots.txt communicates the owner's crawling preferences; it is a technical convention rather than a legal permission document, but ignoring it is the fastest way to turn a quiet crawl into a dispute.

Behaviour	Likely concern	Safer practice
Bypassing a login, paywall, CAPTCHA or IP block	Unauthorized access, contract or computer-misuse claims	Stop; get written authorization or use an approved feed or API
Ignoring robots.txt or a direct no-crawl request	Policy conflict, escalation, contract claims	Honour the instruction and record the decision
Collecting names, emails, profiles or identifiers	Privacy, data-protection, retention and security duties	Collect the minimum, define a purpose, secure it, delete it
Downloading full articles, images or datasets	Copyright, database rights, unauthorized commercial use	Keep only the fields the analysis needs
Sending many concurrent requests	Server strain, service disruption, abuse complaints	Low rate, caching, bounded scope and a kill switch

A user-agent string does not create permission, and a public URL does not mean the owner welcomes bulk copying. Treat a complaint, an HTTP 429, or a jump in server load as a reason to pause.

The Ninth Circuit's 2022 hiQ Labs v. LinkedIn opinion is often cited as "scraping is legal". It is narrower than that: it turned on the CFAA, public profile pages and the case's procedural posture, and it left other claims open. Use court opinions as input to counsel's analysis, not as a licence.

How to crawl a website responsibly

Start with a written scope. Crawling your own site gives you control over policy, logs and infrastructure; a third-party site needs more care, especially one with accounts, user-generated content or unclear automated-access rules.

1. Identify the crawler. Use a truthful user-agent such as CompanyAuditBot/1.0 and a monitored contact address.
2. Set the boundary. Define hostname, URL patterns, maximum pages, file types and duration before the first request.
3. Read robots.txt. Cache it, follow applicable Disallow rules, and stop if the file cannot be parsed.
4. Control traffic. Start with one request at a time, add delays, retry sparingly, back off on 429 and 503, and cache identical responses.
5. Stop at barriers. Do not guess passwords, solve CAPTCHAs, rotate IPs to dodge limits, or follow a redirect into a restricted area.
6. Minimize data. Store status codes, headings, links and findings rather than personal records. Encrypt what you keep and set a deletion date.
7. Log decisions. Record the robots.txt you saw, the scope, the user-agent, errors, and why the crawl stopped.

Rendering crawlers need extra restraint. Executing JavaScript pulls scripts, stylesheets, API calls and images, so one URL can cost the server many times a plain HTML fetch. Block unneeded assets, cap browser concurrency, and test against a staging copy when you own the site.

A workable policy can fit in one sentence: crawl only public URLs on example.com, honour robots.txt, fetch at most 500 URLs with one browser session, retry each failure once, cache responses, and stop on any login wall or repeated 429. That is a technical control, not legal clearance.

How search engines crawl, and how to verify them

Search crawling separates controls that developers often blur together. robots.txt tells a compliant crawler which paths to avoid. The HTTP status says what happened. A canonical tag suggests the preferred version of similar pages. noindex asks the index to leave a page out. Server authentication decides whether the client gets the resource at all.

Verifying Googlebot. Do not trust the user-agent string or a copied IP list: anyone can claim to be Googlebot. Google documents a two-step check: a reverse DNS lookup on the requesting IP should return a googlebot.com, google.com or googleusercontent.com hostname, and a forward lookup on that hostname must resolve back to the same IP. Google also publishes its crawler IP ranges as JSON files you can match against. Reverse lookup alone is not enough.

Hard-coded allowlists age badly: cloud ranges change, proxies hide origins, and a wrong rule blocks legitimate traffic. Combine server logs, verified DNS and request behaviour, and never treat an allowlist as permission to serve private content.

When a page is not being crawled or indexed, work in this order:

1. Fetch /robots.txt and check whether the path, or a script or stylesheet the page needs, is blocked.
2. Inspect the status code, redirect chain, content type and final URL.
3. Read server logs for the real user-agent, source IP, latency and error rate.
4. Test the rendered page if JavaScript inserts links, metadata or the main content.
5. Check canonical, noindex, sitemap and internal links before judging indexability.

Crawlability asks whether a bot can fetch the resource; indexability asks whether the engine will store and show it. A page can be crawlable but excluded by noindex, or blocked before the bot ever sees its canonical tag.

What an SEO crawler should check on your own site

An SEO crawl tells you what a page does and how it responds. It cannot tell you whether a collection plan is lawful, so keep the technical report and the legal review separate.

Search operators help with public discovery without touching the server: site:example.com inurl:docs shows indexed documentation paths, site:example.com filetype:pdf shows indexed PDFs. They show what a search engine already made discoverable; they do not authorize requests to an exposed directory or account area.

On a site you control, an SEO crawler should flag:

• CSS, JavaScript, images or API responses blocked from the renderer.
• Broken or conflicting canonicals, duplicate parameter URLs, inconsistent trailing slashes.
• Redirect chains and loops, and redirects using the wrong status: a 301 for a permanent move, a 302 only for a genuinely temporary one.
• Missing titles, descriptions, headings, alt text and structured data.
• Slow Largest Contentful Paint, heavy pages and other performance problems.

A concrete case: a team finds its JavaScript-rendered pricing page serves a blank HTML shell, one of its scripts is blocked in robots.txt, and an old route 302s to it. They unblock the script, turn the permanent move into a 301, and re-crawl the rendered output. Every step happened on their own site.

A go / no-go framework before you crawl

Decide before you write code. The safe default is your own site, or a system you have explicit written authorization for.

Zone	Example	Action
Green	Your own deployed site or an authorized staging host	Define scope, rate limits, logging and retention; crawl
Yellow	Public third-party pages with unclear terms or personal data	Narrow scope, skip sensitive fields, read terms, ask first
Red	Login areas, paywalls, bypassed blocks, traffic that hurts service	Stop; get counsel and written authorization

Questions to take to counsel for a yellow or red project:

• Which jurisdictions' privacy, copyright, database, contract and computer-misuse rules apply?
• Does the crawl collect personal data, and what notice, purpose, security and deletion duties follow?
• Do the site's terms or an API agreement restrict automated access or reuse?
• Does the project copy protected text, images, records or a substantial part of a database?
• What evidence shows authorization, and which technical controls must the crawler respect?

How SEOAgent fits

SEOAgent crawls the site you own. The free Skill and local seoagent CLI run inside your coding agent (Claude Code, Cursor, Codex), crawl your live site, and write proposed fixes, such as metadata, redirects, sitemap and robots.txt changes, into your repository as a diff you approve before anything ships. See SEO optimization in your repo and SEOAgent for developers.

It does not give legal advice, grant permission to crawl third-party systems, or decide whether a data-collection plan complies with local law. Point it at your own domain, or one you are authorized to audit, and the green-zone rules above already cover you.

Before running any crawler, write the hostname, URL scope, request policy, data fields, retention period and stop conditions into the project notes, and have counsel review the plan whenever the target, the data or the access method leaves the green zone.

References

• Justice Manual 9-48.000: Computer Fraud and Abuse Act (U.S. Department of Justice)
• hiQ Labs, Inc. v. LinkedIn Corp., No. 17-16783 (9th Cir. 2022)
• Verifying Google crawlers and fetchers (Google)
• Introduction to robots.txt (Google Search Central)