Text file containing website access permissions for search engine bots. Search engine bots that come to crawl websites first examine this file and perform crawling as per the permissions in this file. This text file placed in the website source code provides information about the web pages that search engine bots are to index. Therefore, webmasters should pay special attention while creating or editing the robots.txt file.
