Home / Web tools / Robots.txt Generator & Tester

Robots.txt Generator & Tester

SEO

Build a valid robots.txt file for Googlebot and AI crawlers, add Allow/Disallow rules and Sitemap URLs, and test any URL path before you publish.

Quick presetsClick to load a template

Live URL Path Tester

Generated robots.txt


      

Upload this file to the root of your domain (https://yourdomain.com/robots.txt).

What this robots.txt generator builds

Pick a preset for WordPress, Next.js, Shopify, or AI crawler blocking, customize your User-agent rules and Sitemap URLs, and copy or download your ready-to-upload robots.txt file.

Before deploying, use the built-in URL Path Tester to verify whether a specific path (like /wp-admin/ or /wp-admin/admin-ajax.php) is allowed or blocked for Googlebot or any other crawler.

DirectiveWhat it tells crawlersCommon example
User-agentWhich crawler the rules apply to (* means all bots)User-agent: Googlebot
DisallowURL paths crawlers should not requestDisallow: /admin/
AllowExceptions inside a disallowed folderAllow: /wp-admin/admin-ajax.php
Crawl-delaySeconds to wait between requests (ignored by Googlebot, used by Bing/others)Crawl-delay: 5
SitemapFull URL to your XML sitemap index or fileSitemap: https://example.com/sitemap.xml

How to deploy robots.txt safely

  1. Download the generated file as robots.txt (all lowercase).
  2. Upload it to the root directory of your domain so it loads at https://yourdomain.com/robots.txt.
  3. Verify it in Google Search Console under Settings → robots.txt.
  4. Never use robots.txt to hide sensitive pages from Google index—use a noindex meta tag or password protection instead, because blocked URLs can still appear in search results without a snippet if other sites link to them.

Frequently asked questions

Where should I put my robots.txt file?

It must live at the root of your host, e.g. https://example.com/robots.txt. Placing it in a subfolder like /blog/robots.txt has no effect.

Does robots.txt stop Google from indexing a page?

No. robots.txt blocks crawling, not indexing. If other pages link to a disallowed URL, Google can still index the URL. Use a noindex meta tag or X-Robots-Tag header to keep a page out of search results.

How do I block AI crawlers like GPTBot without hurting Google SEO?

Keep User-agent: * allowing your public pages, and add separate blocks for User-agent: GPTBot, CCBot, or ClaudeBot with Disallow: /.

Is this robots.txt generator free?

Yes, 100% free and runs locally in your browser without sending your URLs or rules to any server.