Detail of user-agent, allowed and disallowed URLs of your website
With this free tool, you can analyze, test, and validate your website robots.txt file with just one click. The tool will help you to check which of your page URL is blocked by the user-agent or which are not. In simple words, by using this (robots.txt analyzer) tool you can check the allowed and dis-allowed list of URLs.
This is nothing but a simple text file filled with the information that which web pages or links of your website are crawled by google bot or search engines and which are not. You can also specify XML sitemap in robots.txt file to tell the URL structures of your web pages to google bots or search engine bots.
Whenever search engine bots want to index any website, it first searches the robots.txt file of the website and follows the instructions specified in the robots.txt file given by webmasters. For this purpose, you must have the robots.txt file saved in your website root directory. You should always check the correctness of your robots.txt content before uploading it to the root directory. Because this file contains the instructions that which URL of your website is to be crawled by search engine bots. That is why you must care about this.
This free tool (Analyze And Test Robots Txt Files On Large Scale) will help you to validate and test your created robots.txt file. You only need to enter the website URL in the above form and click on Analyze robots.txt Data button.
The robots.txt document can essentially be made utilizing any content manager text editor. Each record comprises two values. Initial, one indicates the client specialist (user-agent) to which the instructions are to be applied, and the second one is dis-allowed instructions to be avoided to crawl mentioned URLs.
Structure of robots.txt file
User-agent: *
Disallow:
The above code permits to search engine bot to crawl all the web pages without any restriction. If you want to disallow some URLs or specific web pages then you should add the following instruction in robots.txt.
User-agent: *
Disallow:/third-party/
The above code will instruct the google bot or search engine bot, not to crawl the pages which include this (/third-party/) path in the URL.