Robots.txt Generator

Default - All Robots are:	Allowed Refused
Crawl-Delay:	Default - No Delay 5 Seconds 10 Seconds 20 Seconds 60 seconds 120 Seconds
Sitemap: (leave blank if you don't have)
Search Robots:	Google	Same as Default Allowed Refused
	Google Image	Same as Default Allowed Refused
	Google Mobile	Same as Default Allowed Refused
	MSN Search	Same as Default Allowed Refused
	Yahoo	Same as Default Allowed Refused
	Yahoo MM	Same as Default Allowed Refused
	Yahoo Blogs	Same as Default Allowed Refused
	Ask/Teoma	Same as Default Allowed Refused
	GigaBlast	Same as Default Allowed Refused
	DMOZ Checker	Same as Default Allowed Refused
	Nutch	Same as Default Allowed Refused
	Alexa/Wayback	Same as Default Allowed Refused
	Baidu	Same as Default Allowed Refused
	Naver	Same as Default Allowed Refused
	MSN PicSearch	Same as Default Allowed Refused
Restricted Directories:	The path is relative to root and must contain a trailing slash "/"

Now, Create 'robots.txt' file at your root directory. Copy above text and paste into the text file.

About Robots.txt Generator

This free robots.txt generator builds a robots.txt file that tells search engine crawlers which parts of your site they may visit. Choose your settings, press the button, and copy or save the file.

Quick facts

• Default access: allow or refuse all robots
• Crawl delay: 5, 10, 20, 60 or 120 seconds, or none
• Sitemap: add your sitemap address
• Search robots: set rules for individual crawlers such as Google, Google Image, Google Mobile, MSN, Yahoo, Ask/Teoma and others
• Restricted directories: list folders you want robots to avoid
• Output: robots.txt text to copy, or save as a robots.txt file
• Cost: free, no sign-up

How to use the Robots.txt Generator

1. Choose whether all robots are allowed or refused by default.
2. Add a crawl delay and your sitemap address if you want.
3. Set rules for specific search robots if needed.
4. List any restricted folders, such as /admin/.
5. Press Create Robots.txt, then copy the text or save it as a file.
6. Upload robots.txt to the root of your site, so it opens at yourdomain.com/robots.txt.

Important notes

• Robots.txt asks well-behaved crawlers to stay away. It does not hide pages from people and is not a security tool.
• Blocking a page here does not remove it from Google. To keep a page out of search results use a noindex tag.
• Google ignores the crawl-delay setting, though some other crawlers follow it.
• Be careful not to block CSS, JavaScript or important pages by mistake.

After you upload it

Test the file in Google Search Console and add your sitemap there. You can create a sitemap with the XML sitemap generator.

Privacy

The file is generated in your browser from the settings you choose.