The process of systematically examining and collecting information from a website by search engine bots. The process of crawling may take place for an entire website or for a limited number of web pages on a given website. During crawling, all items including images, content, and links of web pages are examined in detail. Crawling can be done by search engine bots as well as crawling tool bots.

Related Articles