Google’s Advice on AMP Redirects for Better Crawl Efficiency

Managing AMP subdomains and ensuring that Google properly handles URL redirects can be challenging, especially for large-scale websites. This issue was raised by a website owner on Reddit, where they questioned whether returning a 410 status for old AMP URLs would have saved crawl budget, or if relying on 301 redirects was sufficient. In response, Google’s Search Advocate John Mueller provided valuable advice on managing AMP subdomains, crawl budget, and redirect strategies.

The Situation

Several years ago, the user created an AMP subdomain (amp.example.com) to optimize their website for mobile users. Three years ago, they decided to stop using AMP and implemented 301 redirects for all AMP URLs back to their main domain (example.com). Despite this, Googlebot continues to crawl hundreds of thousands of AMP URLs:

HTTPS AMP URLs: 100,000 URLs redirect, with 7,000 still marked as “crawled but not indexed.”
Non-HTTPS AMP URLs: 8,000 URLs redirect, with 1,000 “crawled but not indexed.”

The user is worried that this crawling might consume resources, specifically the crawl budget for their main domain, and is considering switching from 301 redirects to 410 Gone status to stop Google from crawling these URLs altogether. So they took to Reddit looking for guidance.

The Solution

In response to this situation, John first explained that since the AMP URLs are on a separate subdomain, they likely have their own dedicated crawl budget:

"If it's a separate subdomain, it'll generally have its own 'crawl budget.' Also, with 500k pages, I don't know if you'd really need to worry about crawl budget, even if it were not on a separate host name." - John Mueller

This reassures the user that the crawling of the AMP subdomain is unlikely to impact the crawl budget of the main domain. Given that the site contains 500,000 pages, the main domain likely has sufficient resources to ensure efficient crawling of important URLs.

As for the site owner’s suggestion of returning a 410 status code for all AMP URLs, this would explicitly tell Google that the content no longer exists. This status would signal to Googlebot to stop crawling those URLs and could theoretically free up resources more quickly than waiting for 301 redirects to take full effect.

However, Mueller mentioned that while a 410 status code would achieve this, it may not be necessary in this case. Since the AMP URLs already have 301 redirects, it’s only a matter of time before Google reduces its crawling activity naturally.

“Personally, I'd either leave the setup as-is (redirects), or just remove the hostname from DNS and force it like that.” - John Mueller

By choosing either to keep the 301 redirects or remove the AMP subdomain from DNS, the site owner can prevent unnecessary crawling without needing to implement 410s.

One of the more direct solutions suggested by Mueller was to remove the AMP subdomain from DNS entirely. This would essentially prevent Google from crawling the subdomain since the domain would no longer resolve.

Removing the DNS entry is a more aggressive approach, but it can be highly effective in stopping crawling immediately. It also removes the need to return any status codes (like 301 or 410) because Googlebot would not be able to access the subdomain at all. This method could be particularly useful for site owners looking for a faster resolution to stop crawling.

Conclusion

John Mueller’s advice provides valuable insight into handling AMP subdomain redirects. Whether you choose to stick with the 301 redirects, return 410 status codes, or remove the AMP subdomain from DNS, each option comes with its own implications.

If you’re not in a hurry: Stick with the 301 redirects, and Google will gradually reduce its crawling over time.
If you want faster results: Consider removing the subdomain from DNS to stop crawling immediately.
If you’re concerned about crawl budget: Know that AMP subdomains typically have their own crawl budgets, so the impact on your main domain should be minimal.

Mueller also pointed to helpful resources on managing crawl budgets:

"The docs are findable under 'Large site owner's guide to managing your crawl budget.'

To learn more, you can refer to Google’s Large site owner’s guide to managing your crawl budget.

References:

AMP Subdomain Still Crawled After 301 Redirects.

Social Media Marketing Company