Create a robots.txt file to control how search engine crawlers access your site. Manage Allow/Disallow rules easily.
The bot you want to give instructions to.
Disallow (Block)
Allow (Unblock)
Optional: Path to your XML sitemap.
Optional: Seconds between requests (supported by some bots).
Generated robots.txt
The Robots.txt Generator allows you to easily create the robots.txt file for your website. This file gives instructions to web robots (also known as crawlers or spiders) about which parts of your site they should or should not crawl.
The Robots Exclusion Protocol (REP), or robots.txt, is a text file residing in the root directory of your website. It tells search engine crawlers which pages or files the crawler can or can't request from your site.
*) means the rules apply to all bots.Using a robots.txt file helps you manage crawl budget by preventing bots from wasting time on unimportant pages (like admin panels, duplicate content, or temporary files). It is a crucial part of technical SEO.
Disallow: /admin/
User-agent: * Disallow: /admin/
Blocks all crawlers from /admin/.
Allow: /public/ User-agent: Googlebot
User-agent: Googlebot Allow: /public/
Allows Googlebot to crawl /public/.
Crawl-delay: 10
User-agent: * Crawl-delay: 10
Delays crawlers by 10 seconds between requests.
To communicate with web crawlers and other web robots.
Upload it to your site's root directory.
Yes, simply specify the subdomain in the site URL.
Yes, including crawl-delay and sitemap directives.
Major search engines like Google, Bing, and Yahoo support the standard robots.txt protocol.
Update it whenever your site's structure or crawl needs change.
While it can help, for comprehensive protection, consider additional security measures.