How to Stop Google Indexing Pages and Files with .htaccess (X-Robots-Tag noindex)

Applies to: Apache 2.4 with mod_headers (cPanel hosting, VPS and dedicated servers)

A noindex rule tells search engines not to show a page or file in their results. For HTML pages you can put it in a meta tag, but PDFs, images and whole folders have no HTML head. For those, send it as an HTTP header, X-Robots-Tag, from your .htaccess file. Google and Bing both support this header.

Before you start: don't also block the same URLs in robots.txt. If crawlers are not allowed to fetch a page, they never see the noindex header, and an already-indexed URL can stay in the results.

Which rule to use

• noindex: don't show this page or file in search results.
• nofollow: don't follow the links on it.
• noindex, nofollow (or none): both.

Edit .htaccess in cPanel

1. Open File Manager in cPanel and go to the folder you want to change (public_html for the main site).
2. If you don't see .htaccess, turn on hidden files (see showing hidden dotfiles in File Manager).
3. Right-click .htaccess and choose Edit, or create the file if it doesn't exist. More detail: how to edit .htaccess in File Manager.
4. Add one of the examples below, save, and test (see the last section).

Examples

Noindex a whole folder (for example /private/): create .htaccess inside that folder with:

<IfModule mod_headers.c>
    Header set X-Robots-Tag "noindex, nofollow"
</IfModule>

The rule applies to every page and file in the folder and its subfolders.

Noindex PDF and Office files anywhere on the site: add to the .htaccess in public_html:

<IfModule mod_headers.c>
    <FilesMatch "\.(pdf|docx?|xlsx?|pptx?)$">
        Header set X-Robots-Tag "noindex, nofollow"
    </FilesMatch>
</IfModule>

Noindex one file:

<Files "thank-you.html">
    Header set X-Robots-Tag "noindex"
</Files>

Noindex a whole site, such as a staging copy on its own subdomain: add the folder rule above to the .htaccess in that site's document root. Remove it before the site goes live, or it will drop out of search results.

We tested these rules on Apache 2.4.68, the version on our cPanel servers: pages and files matched by a rule return X-Robots-Tag: noindex, nofollow, and the rest of the site is unchanged.

Check that it works

Look at the response headers from your computer's terminal:

curl -I https://example.com/private/
curl -I https://example.com/files/brochure.pdf

The output should include X-Robots-Tag: noindex, nofollow. If it doesn't, check that the rule is in the right folder and that your browser or a CDN is not showing a cached copy. In Google Search Console you can then run URL Inspection on the URL and request indexing, so Google sees the header sooner. Otherwise it can take a long time before Google crawls the page again.

Other ways to noindex

• Single HTML page: add <meta name="robots" content="noindex"> inside its <head>.
• WordPress: Settings › Reading › Search engine visibility hides the whole site; SEO plugins can noindex single posts or pages.
• Stop crawling rather than indexing: that is what robots.txt does (see how to create a robots.txt file in cPanel).

A noindex header does not protect private content: anyone with the link can still open it. To lock a folder, use password protection (Directory Privacy in cPanel) instead.

Official documentation: Block search indexing with noindex (Google Search Central).