Google Clarifies Supported Fields in Robots.txt Files

Google Search Central recently updated its documentation to clarify which fields are supported in robots.txt files. This clarification comes as a response to recurring questions about unsupported fields. The new update provides webmasters with a clear understanding of what can be included in their robots.txt files to ensure proper interaction with Google's crawlers.


What is Robots.txt?

Robots.txt is a simple text file placed on a website's server that guides search engine crawlers on which parts of the site they are allowed to crawl. It serves as a useful tool for controlling web traffic and optimizing search engine behavior on your site.


Valid Robots.txt Format

A valid robots.txt line consists of a field, a colon, and a value. Spaces are optional but recommended for readability. For example:

field: value

Any additional spaces at the beginning or end of the line will be ignored, and comments can be included by using the # symbol, with everything after it being ignored.

Example Format:

user-agent: Googlebot  # Target Google's bot
disallow: /private/    # Block crawling of private folder

Supported Fields in Robots.txt

Google has made it clear that only specific fields are supported, and anything outside these won't be recognized. Here are the fields you can use:

  • user-agent: This field identifies which web crawler (or bot) the rules apply to. For example, using user-agent: Googlebot will target Google’s crawler specifically.
  • allow: This field defines a URL path that may be crawled, overriding any broader disallow rules.
  • disallow: This field is used to block specific URL paths from being crawled. For example, you might use disallow: /admin/ to prevent search engines from accessing your admin directory.
  • sitemap: This field provides the full URL of a sitemap, making it easier for search engines to discover and index all relevant pages on your site.

Example of a robots.txt File:

user-agent: *
disallow: /private/
allow: /public/
sitemap: https://www.example.com/sitemap.xml

In this example, all crawlers (user-agent: *) are blocked from accessing the "private" folder but allowed to crawl the "public" folder. A sitemap URL is also provided for easier discovery of all pages.


Unsupported Fields: What You Should Avoid

Google does not support fields like crawl-delay or any others not mentioned in their official robots.txt documentation. The inclusion of unsupported fields will be ignored by Google’s crawlers, and they will not affect how your site is indexed or crawled. To avoid confusion and ensure effective communication with search engines, it's essential to stick to the supported fields.


Why This Clarification Matters

Webmasters often encounter confusion when adding unsupported fields like crawl-delay, expecting Google to follow them. This update is aimed at minimizing such misunderstandings. It reinforces the importance of adhering to Google’s supported syntax, ensuring that your robots.txt file is both valid and effective in controlling site indexing.

By using only the officially supported fields, you can ensure that Google and other search engines interpret your robots.txt file correctly and that your crawling preferences are respected.


Conclusion

This clarification is part of Google’s ongoing effort to streamline SEO best practices, helping website owners achieve better control over their site's visibility on search engines. For more detailed information, you can visit Google's official robots.txt documentation.


References:

Google’s Latest Documentation Updates


Previous Updates

Google Introduces Certification Markup for Merchant Listings.

Google Updates on Best Practices for Dynamically-Generated Product Markup.

Google's Big Shift: Cache Operator Gone, But There’s a New Way to Access Archived Pages.