:mod:`!urllib.robotparser` --- Parser for robots.txt

فهرست فصل‌ها دانلود آرشیوناشر

Python 3.14 Docs · نسخهٔ 3.14.10 · PSF License

:mod:!urllib.robotparser --- Parser for robots.txt

:synopsis: Load a robots.txt file and answer questions about fetchability of other URLs.

Source code: Lib/urllib/robotparser.py

single: WWW single: World Wide Web single: URL single: robots.txt

--------------

This module provides a single class, RobotFileParser, which answers questions about whether or not a particular user agent can fetch a URL on the website that published the robots.txt file. For more details on the structure of robots.txt files, see 9309.

This class provides methods to read, parse and answer questions about the robots.txt file at *url*.

Sets the URL referring to a robots.txt file.

Reads the robots.txt URL and feeds it to the parser.

Parses the lines argument.

Returns `True if the *useragent* is allowed to fetch the *url* according to the rules contained in the parsed robots.txt` file.

Returns the time the `robots.txt file was last fetched. This is useful for long-running web spiders that need to check for new robots.txt` files periodically.

Sets the time the `robots.txt` file was last fetched to the current time.

Returns the value of the `Crawl-delay parameter from robots.txt for the *useragent* in question. If there is no such parameter or it doesn't apply to the *useragent* specified or the robots.txt entry for this parameter has invalid syntax, return None`.

Returns the contents of the `Request-rate parameter from robots.txt as a named tuple RequestRate(requests, seconds). If there is no such parameter or it doesn't apply to the *useragent* specified or the robots.txt entry for this parameter has invalid syntax, return None`.

Returns the contents of the `Sitemap parameter from robots.txt in the form of a list. If there is no such parameter or the robots.txt entry for this parameter has invalid syntax, return None`.

The following example demonstrates basic use of the RobotFileParser class::

توجه: >> import urllib.robotparser >> rp = urllib.robotparser.RobotFileParser() >> rp.set_url("http://www.pythontest.net/robots.txt") >> rp.read() >> rrate = rp.request_rate("*") >> rrate.requests

1

توجه: >> rrate.seconds

1

توجه: >> rp.crawl_delay("*")

6

توجه: >> rp.can_fetch("*", "http://www.pythontest.net/")

True

توجه: >> rp.can_fetch("*", "http://www.pythontest.net/no-robots-here/")

False