RobotFileParser
- RobotFileParser(url='')
This class provides a set of methods to read, parse and answer questions about a single robots.txt file.
© Python Software Foundation. PSF License Version 2 · Official docs ↗
Members of RobotFileParser
c
- can_fetch
- using the parsed robots.txt decide if useragent can fetch url
- crawl_delay
- Returns the value of the Crawl-delay parameter from robots.txt for the useragent in question.
m
p
- parse
- Parse the input lines from a robots.txt file.
r
- read
- Reads the robots.txt URL and feeds it to the parser.
- request_rate
- Returns the contents of the Request-rate parameter from robots.txt as a named tuple RequestRate(requests, seconds).
1 Example
>>> import urllib.robotparser >>> rp = urllib.robotparser.RobotFileParser() >>> rp.set_url("http://www.pythontest.net/robots.txt") >>> rp.read() >>> rrate = rp.request_rate("*") >>> rrate.requests 1 >>> rrate.seconds 1 >>> rp.crawl_delay("*") 6 >>> rp.can_fetch("*", "http://www.pythontest.net/") True >>> rp.can_fetch("*", "http://www.pythontest.net/no-robots-here/") False
from the Python docs