Free Technical SEO Tool

Create a Clean Robots.txt File

Build a robots.txt file for search engine crawlers with custom user-agent rules, allow/disallow paths, sitemap location and optional crawler controls.

User-agent Rules Disallow Paths Allow Paths Sitemap Download TXT

Robots.txt Generator

Create crawl directives and preview the complete robots.txt file instantly.

Runs in your browser
Robots Settings Build crawl rules
Used to suggest the sitemap URL. This value itself is not required inside robots.txt.
Enter one path per line. Use / only if you intentionally want to block the selected crawler from the entire site.
Optional. Enter one allowed path per line.
Quick Presets
Robots.txt Preview Editable output

Free Robots.txt Generator

WebRankly Robots.txt Generator helps you create crawler directives for your website. A robots.txt file is normally placed at the root of a site, for example https://example.com/robots.txt.

User-agent Rules Create directives for all crawlers or selected bots such as Googlebot, Bingbot and other named crawlers.
Allow & Disallow Add crawl paths line-by-line and generate correctly formatted directives.
Sitemap Directive Add your XML sitemap URL to help crawlers discover the sitemap location.
Download File Copy the generated content or download it directly as a robots.txt file.

Where Should robots.txt Be Placed?

The file should normally be accessible from the root of the hostname you want to control, such as https://example.com/robots.txt. Rules for a different subdomain need to be served from that subdomain’s own robots.txt location.

Robots.txt Is Not a Security Tool

Disallowing a URL does not make it private or protected. Sensitive areas should use proper authentication and access controls. Also review important CSS, JavaScript and image paths before blocking them because search engines may need those resources to understand or render a page.

Robots.txt FAQs

Robots.txt controls crawling, not guaranteed indexing removal. If you need a page excluded from search results, use an appropriate indexing control and make sure the crawler can access the page when required to see that directive.
You can include one or more Sitemap directives using absolute sitemap URLs. It is a convenient way to advertise sitemap locations to crawlers that support the directive.
No. Support varies between crawlers. This generator lets you add it, but you should check the documentation for the crawler you are targeting.
You can create named User-agent rules for crawlers that publish robots.txt support. Whether a crawler honors the directive depends on that crawler’s behavior and current documentation.
Copied