Bot Protocol Guide

Robots.txt is a text file used by websites to communicate with web robots (such as search engine crawlers) about which parts of the site should be crawled or excluded from indexing.