RobotFileParser.can_fetch
- RobotFileParser.can_fetch(useragent, url)
using the parsed robots.txt decide if useragent can fetch url
© Python Software Foundation. PSF License Version 2 · Official docs ↗
1 Example
>>> import urllib.robotparser >>> rp = urllib.robotparser.RobotFileParser() >>> rp.set_url("http://www.pythontest.net/robots.txt") >>> rp.read() >>> rrate = rp.request_rate("*") >>> rrate.requests 1 >>> rrate.seconds 1 >>> rp.crawl_delay("*") 6 >>> rp.can_fetch("*", "http://www.pythontest.net/") True >>> rp.can_fetch("*", "http://www.pythontest.net/no-robots-here/") False
from the Python docs