Robots.txt meant for search engines don’t work well for web archives | Internet Archive Blogs

http://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/

This URL is bookmarked by 1 user.

First saved: 2017-Apr-21 (Fri)
Last checked: 2017-Apr-21 (Fri), response 200