Few days ago, I logged into my Facebook account and upon getting to my inbox, someone asked me to explain what I know about Robots.txt files and how to use Robots.txt files for SEO. I replied him immediately and decided to share with you guys all you need to know about Robots.txt file and how you can use it for SEO on your website. Many questions like;
- What is robot.txt file?
- How I use robot.txt file for SEO?
- What is robot.txt used for?
- What is robot.txt in wordpress?
- How do I stop Google from indexing some web pages on my website?
- How do I stop Google from indexing my admin folders?
- Why the slow crawl rate on my blog?
One thing about Using Robot.txt file is that every website irrespective of the platform either WordPress, Joomla, Drupal, etc needs it. And helps improve your website Index rate making the SEO effort on your website or blog to be better. Let me start by asking
What is Robot.txt File?
The robots exclusion protocol (REP), or robots.txt is a text file created by webmasters to instruct robots/bots (typically search engine robots/bots) on how to crawl and index pages on their website. see more from the wikipedia page
With the Robots.txt file on your website, you tend to gain full authority as a webmaster on your website. This will make you to stop the search engine robots from going to certain places on your website according to the file demand you created. The Robots.txt resides at the root of a domain. For example; domain.com/Robots.txt
What is Robots.txt file Used for on a Website?
Before we talk about the uses of robot txt file, you must know it that different search engines have its own bots or search spider to crawl and indexing your website. Don’t forget in my post about how to get Google and other search engines to crawl your website, I made mention that Web Crawling and Web Indexing are two different things. The search engine bot (Google Bot, Bing Bot, Yandex Bot and other search engine Bot) come to your website through a Site Map link from your webmaster dashboard of through a link which can be backlink or page URL you submitted to Google Crawler to index your new blog post.
Sitemap.xml, just as Robot.txt dwells at the root domain of your website. Below is the big caption.
SITEMAP.XML: The road/route to every part of your website
ROBOT.TXT: The points on the Sitemap that cannot be accessed by bots, The places or positions on the website that has “no access” rule, and tell the robot not to go there.
So below are the uses of robot.txt file on your website.
- The robot txt files, gives the search engine bots some restrictions while crawling the webpages on your website: Yes its true if they can’t crawl all the pages on your website with the access they have, they will stop crawling and your site index rate will eventually fall, also at the same time, they are some certain pages or folders (like your admin dashboard, wp-admin, plugins and the rest of them) on your website that you don’t want the search engine robot to crawl since it’s of no use to the robot. With the use of robot.txt, you tell these search engine bots no to crawl such areas on your website.
- Helps increase the Crawl speed of your blog: Since not everywhere on your website is crawled, the crawl time of your website will as well reduce which will helps you in your SEO effort and deep crawling of your inner pages which in general, helps your online content marketing strategy.
- Privacy: The robot.txt file in your website give you room for privacy on your website between you and search engines.
- The robots.txt file of your website, controls how search engine spiders see and interact with your webpages
Note: Robot txt files are not used for Noindexing in anyway, all it does is to give the search engine bots authority and control over how they crawl and go about on your website.
How to Check Your Robots.txt file
Now you’ve known what and the uses of robot.txt file is, the next question on your mind is “how do I check the robots.txt file of my website”? To check yours,
Kindly enter www.domain.com/robots.txt (replace domain with your own site domain name) on your browser and you will see a result like the blank image result below for Techliet.com.
You can also login to your Google webmaster Tool to check. To do that,
- From your GWT account, Go to “Crawl”
- Click on “Robot.txt tester” see image below
Robot files only stops crawling and not indexing. Google Indexes every page and post and may require you to take a Noindex action if you want to avoid some posts and pages from being crawled.
Using robot txt file on your website, give you total control of your platform from search engines and gives them restrictions to certain places on your website. Places like Comments, trackbacks, feeds and pages can be prevented from crawling. Since Robot.txt file stops only crawling and not indexing, you will have to use the Noindex tag to stop Google from indexing some posts and pages on your website. To use the Noindex tag, you will have to use any of the SEO plugins like Yoast SEO plugin to achieve this.
If there is any part of your blog or website that has been indexed already and you want to De-Index, don’t use robots.txt to block access to that part, else this will prevent bots from crawling that part of your blog.
If you know you have not been using robots.txt file on your website or blog, head straight and do right away, it will help your greatly, if you have any else to chip in to this article, recommendation question or feedback, please use the comment section and please do subscribe to our mailing list before leaving. You will always get the best tips and updates on SEO, traffic, increased earning and online marking tips.