RobotFileParser.crawl_delay
- RobotFileParser.crawl_delay(useragent)
Returns the value of the Crawl-delay parameter from robots.txt for the useragent in question. If there is no such parameter or it doesn’t apply to the useragent specified or the robots.txt entry for this parameter has invalid syntax, return None.
© Python Software Foundation. PSF License Version 2 · Official docs ↗
1 Example
>>> import urllib.robotparser >>> rp = urllib.robotparser.RobotFileParser() >>> rp.set_url("http://www.pythontest.net/robots.txt") >>> rp.read() >>> rrate = rp.request_rate("*") >>> rrate.requests 1 >>> rrate.seconds 1 >>> rp.crawl_delay("*") 6 >>> rp.can_fetch("*", "http://www.pythontest.net/") True >>> rp.can_fetch("*", "http://www.pythontest.net/no-robots-here/") False
from the Python docs