Robots txt configuration

Answer: Robots.txt configuration is a root-level plain text file that instructs web crawlers which site paths to crawl or avoid, using User-agent, Disallow, All