On Tuesday the 2nd of July, Google announced that they will stop supporting three rules that can sometimes be found in a robots.txt file. From the 1st September 2019 the following rules will be ignored if they are in your website’s robots.txt file:
- Nofollow
- Crawl-delay
- Noindex
What does this mean for website owners?
Google announcing a change such as this may not be relevant to those that don’t use these rules, however, for those that do the impact could be huge. Once Google stops respecting these rules, this could have a drastic effect on your traffic.
Nofollow rule in robots.txt
This rule simply tells Google not to follow the links on a specific page. If you are using this rule in your robots.txt it could mean that you don’t want Google to follow links of that page.
This means that if Google does follow the links of a page they could discover a page that you don’t want indexed in Google. This could have a negative effect on your crawl budget and make your website less search engine friendly.
The best way to implement nofollow is to add this to the links you don’t want Google to follow rather than in the robots.txt file.
Crawl-delay rule in robots.txt
Restricting how often Google can crawl your website probably isn’t the best course of action for SEO. However, some websites may have this implemented as not to overload the server.
Unfortunately, after the 1st of September 2019, these rules will be ignored. If your server can’t handle the increased activity you may need an upgrade.
Noindex rule in robots.txt
This is the most commonly used robots.txt rule that will no longer work. If there are pages on your website that you do not want indexed, then they must include a noindex meta tag in the <head> of each page that you don’t want indexed by Google.
It may have been easier in the past to add rules to robots.txt to remove these pages from Google, however, on the 1st of September 2019, there could be tens, hundreds or even thousands of pages from your website that suddenly end up indexing in Google that you don’t want. This, in turn, could cause duplicate content issues resulting in a loss of traffic and sales.
If you are unsure how this change from Google will affect your website, contact our SEO team for a free review.
