python-reppy2: python-reppy2 (Robots Exclusion Protocol Parser for Python)
python-reppy2:
python-reppy2: Replaces the built-in robotsparser with a RFC-conformant
python-reppy2: implementation that supports modern robots.txt constructs like
python-reppy2: Sitemaps, Allow, and Crawl-delay.
python-reppy2:
python-reppy2:
python-reppy2:
python-reppy2:
python-reppy2:
python-reppy2: