SEO is a vital part of your online marketing strategy and to make the most of this you need to make your website as friendly to search engines as possible.
The first file a search engine will access on your website is the robots.txt file as this contains the rules of which pages and files a search engine is allowed and disallowed from reading.
Controlling how search engines read your website in this way can be very beneficial when utilised correctly, or disastrous if you don’t know what you’re doing.
What to look for in your robots.txt
Here are three simple things to check in your robots.txt file to ensure it is healthy and your SEO strategy is off to the best possible start:
1. Check search engines can read your website
This may seem like obvious advice however if search engines can’t read your website then your website can’t be indexed by them. To check your website is accessible to search engines ensure the following line is not present in your robots.txt
Disallow: /
This rule will stop search engines reading your website completely, causing no pages to be indexed or if there are already pages indexed then search engines will be blocked from crawling these.
2. Check robots.txt contains a reference to your XML sitemap
Being the first file a search engine checks this is a great place to provide search engines with a list of all the pages you want indexed by pointing search engines to your XML sitemap file.
The XML sitemap should contain all of your most important pages and include new pages that you add to your website so the search engines can quickly and easily find and index new pages.
Check your robots.txt to see if it includes a reference to your XML sitemap, it should look like this:
Sitemap: https://www.example.com/sitemap.xml
3. Ensure CSS and JavaScript files are accessible by search engines
Years ago search engines only used to view your website as text with all of the styles and scripts removed as these were seen as dressing for users rather than content needed by search engines at the time.
Today however search engines are much more advanced and realise that the way websites are built has changed. This is why it is very important for search engines to view a page the same as a user would (to detect issues such as hidden content and cloaking which could lead to a penalty).
Why you should monitor your robots.txt file
As the robots.txt is such an important file on your website (it can block search engines reading all of your web pages and files) it is important to monitor this file for changes. Imagine you suddenly drop traffic from your website and can’t work out why, this could be caused by a single change to the robots.txt that could have easily been avoided.
Your SEO company should be monitoring your robots.txt and making changes where necessary to improve how your website is read by the search engines. All Colewood SEO packages include this as this is a critical part of SEO. Contact us today for a free SEO audit which includes a review of your robots.txt file.
