Why Robots.txt is Critical for Technical SEO
The robots.txt file is the front door of your website for search engine web crawlers. It informs Googlebot, Bingbot, and AI data scrapers which directories and URLs they are allowed to index.
Frequently Asked Questions
What is a robots.txt file?
A robots.txt file is a plain text file placed in the root directory of a website that instructs search engine web crawlers (like Googlebot) which pages or files they can or cannot request from your server.
Where should the robots.txt file be uploaded?
The robots.txt file must be uploaded to the root directory of your website domain (e.g., https://example.com/robots.txt). It will not function if placed in subfolders.
How do I block AI crawlers (GPTBot, ClaudeBot) in robots.txt?
To block AI scrapers from training on your content, add User-agent blocks for GPTBot, ClaudeBot, PerplexityBot, and CCBot with 'Disallow: /'.
Does robots.txt guarantee that a page will not appear in Google search results?
No. Robots.txt prevents crawling, not indexing. If an external website links to a disallowed URL, Google may still index the URL snippet without crawling its content. Use noindex meta tags or password protection to guarantee removal.
What is the Crawl-Delay directive?
Crawl-delay specifies the number of seconds a web crawler should wait between successive server requests. Note that while Bingbot and Yahoo honor crawl-delay, Googlebot ignores it (use Google Search Console rate limits instead).
Why should I include a Sitemap URL in robots.txt?
Adding 'Sitemap: https://example.com/sitemap.xml' at the bottom of your robots.txt file helps search engines discover your sitemap index automatically without manual Search Console submissions.
Is the User-Agent wildcard (*) case-sensitive?
Robots.txt directives are generally case-insensitive for User-agent names, but path URL rules (Disallow: /admin/) ARE case-sensitive.
What happens if a website has no robots.txt file?
If no robots.txt file exists (returning a 404 HTTP status), search engine web crawlers assume full permission and will crawl all publicly accessible URLs on your domain.
Sources & References
Related Calculators & Tools
Browse Marketing & SEO →Generate tracking URLs with custom campaign UTM parameters, campaign presets, and instant QR code download.
Generate meta tags, Open Graph tags, and Twitter Cards with real-time Google and social preview mockups.
Preview how your page appears in Google search results on desktop and mobile. Checks pixel width clipping limits.
Test your email subject lines for character limits, spam trigger words, and get alternative A/B testing suggestions.