RobotFileParser.read
- RobotFileParser.read()
Reads the robots.txt URL and feeds it to the parser.
© Python Software Foundation. PSF License Version 2 · Official docs ↗
1 Example
>>> import urllib.robotparser >>> rp = urllib.robotparser.RobotFileParser() >>> rp.set_url("http://www.pythontest.net/robots.txt") >>> rp.read() >>> rrate = rp.request_rate("*") >>> rrate.requests 1 >>> rrate.seconds 1 >>> rp.crawl_delay("*") 6 >>> rp.can_fetch("*", "http://www.pythontest.net/") True >>> rp.can_fetch("*", "http://www.pythontest.net/no-robots-here/") False
from the Python docs