RobotFileParser.request_rate
- RobotFileParser.request_rate(useragent)
Returns the contents of the Request-rate parameter from robots.txt as a named tuple RequestRate(requests, seconds). If there is no such parameter or it doesn’t apply to the useragent specified or the robots.txt entry for this parameter has invalid syntax, return None.
© Python Software Foundation. PSF License Version 2 · Official docs ↗
1 Example
>>> import urllib.robotparser >>> rp = urllib.robotparser.RobotFileParser() >>> rp.set_url("http://www.pythontest.net/robots.txt") >>> rp.read() >>> rrate = rp.request_rate("*") >>> rrate.requests 1 >>> rrate.seconds 1 >>> rp.crawl_delay("*") 6 >>> rp.can_fetch("*", "http://www.pythontest.net/") True >>> rp.can_fetch("*", "http://www.pythontest.net/no-robots-here/") False
from the Python docs