TL;DR
- robots.txt controls crawling; a blocked URL can still appear in Google if other pages link to it.
- A noindex rule keeps a page out of search results, but Google has to crawl the page to see it.
- Don’t block a page in robots.txt and add noindex at the same time; Google won’t see the noindex.
- Google ignores noindex lines in robots.txt and reads only the first 500 KiB of the file.
Both tools say “stay away,” but to different parts of the search process. Getting them right takes a minute; getting them wrong can hide your whole site or leave pages you wanted gone sitting in search results.
What each one does
| robots.txt | noindex | |
|---|---|---|
| Where it lives | A file at the root of your domain, such as example.com/robots.txt | A meta tag in the page’s HTML, or an X-Robots-Tag HTTP header |
| What it controls | Whether crawlers may fetch a URL | Whether a page may appear in search results |
| Can the URL still show in Google? | Yes, without a description, if other pages link to it | No, once Google has crawled the page and seen the rule |
| Good for | Saving crawl effort on endless filter or search URLs, admin areas | Thank-you pages, internal search results, thin tag pages |
The classic mistake
A site owner wants a page out of Google, so they add noindex to the page and also Disallow it in robots.txt for good measure. Because crawling is blocked, Google never fetches the page, never sees the noindex, and can keep the URL in its index based on links alone.
The fix: remove the robots.txt block, leave the noindex in place, and let Google recrawl.
Common mistake: Staging sites are often launched with a site-wide Disallow: / or a “discourage search engines” setting still switched on. After any launch or redesign, check robots.txt and a page’s source for noindex before anything else.
robots.txt rules worth knowing
- Google supports four fields: user-agent, allow, disallow and sitemap. Other lines, including
noindexandcrawl-delay, are ignored by Google. - Size limit: Google reads up to 500 KiB of the file and ignores anything after that.
- Don’t block CSS and JavaScript your pages need to render; Google renders pages much like a browser.
- List your sitemap with a
Sitemap:line; see XML sitemaps.
Checking your work
Use the URL Inspection tool in Search Console to test a specific URL. It shows whether crawling is allowed and whether Google found a noindex rule. Then watch the page indexing report for “Blocked by robots.txt” and “Excluded by noindex” entries that you didn’t intend.
FAQ
How do I remove a page from Google quickly?
Add a noindex rule or remove the page, and use Search Console’s Removals tool for a temporary hide while Google recrawls. For sensitive content, put it behind a password instead.
Can I noindex a PDF?
Yes, with the X-Robots-Tag HTTP header, since PDFs don’t have an HTML head. Your web server or hosting panel can usually add headers for specific file types.
Does robots.txt keep private pages private?
No. The file is public and only well-behaved crawlers follow it. Anything private should require a login.
Search engines update their systems and documentation often. This guide reflects Google’s published guidance at the time of writing; check Google Search Central for current details, and test changes on a staging copy of your site before making them live.