How to use robots.txt?

Updated: 28 September 2026 · Applies to: Google Search Console robots.txt report

A robots.txt file in the top folder of your website tells search engine crawlers which parts of the site they may crawl, and where your sitemap is.

Create it

Create public_html/robots.txt (for your main domain) with rules like these:

User-agent: *
Disallow: /admin/
Disallow: /cart/

Sitemap: https://www.example.com/sitemap.xml

• User-agent: * applies the rules to all crawlers.
• Disallow lists paths that should not be crawled; Allow makes exceptions.
• Sitemap gives the full address of your XML sitemap.

Note: robots.txt is public and only a request: well-behaved crawlers follow it, others may not. To keep a page out of search results, use a noindex tag instead, and protect private content with a password. Check your file with the robots.txt report in Google Search Console.