Robots.txt Tester

Fetch robots.txt, inspect crawl rules, and test which paths are allowed for search bots.