robots.txt is one of those files every website is supposed to have, yet most people only think about it once something’s already gone wrong — usually a page accidentally blocked from Google, or the opposite: a staging site that got indexed because nothing told crawlers to stay out. It’s a plain text file, sitting at yourdomain.com/robots.txt, that tells well-behaved search engine bots which parts of your site they’re welcome to crawl.
What it does NOT do is hide a page from search results — that’s a common misunderstanding. Blocking a URL in robots.txt stops bots from crawling it, but if other sites link to that URL, Google can still show it in search results (just without a description, since it was never allowed to read the page). If you actually want a page kept out of search entirely, that’s a job for a noindex meta tag, not robots.txt.
What it IS good for: keeping bots away from admin areas, internal search result pages, duplicate filtered/sorted product listings, and anything that would otherwise waste your crawl budget on pages with no SEO value. It’s also where you point search engines to your XML sitemap, which helps them discover your important pages faster.
To generate one, open the robots.txt Generator, specify which bots you want to allow or block (or leave it open to all, which is the default for most sites), list any paths you want disallowed, and add your sitemap URL. The tool outputs a correctly formatted file — one small syntax mistake (a missing slash, a typo in “User-agent”) can accidentally block your entire site, so generating it rather than hand-typing it is the safer move.
Once you have the file, upload it to your site’s root directory (not a subfolder — it has to be at yourdomain.com/robots.txt exactly) and give it a day or two; crawlers don’t re-check it instantly. You can verify Google has picked up the right rules through Google Search Console’s robots.txt report.
Related reading: