RobotFileParser.parse

urllib.robotparser

method of RobotFileParser (source)

RobotFileParser.parse(lines)

Parse the input lines from a robots.txt file.

We allow that a user-agent: line is not preceded by
one or more blank lines.

© Python Software Foundation. PSF License Version 2 · Official docs ↗