Apparently there's no limit on recursively downloading the site with wget and never reading any warning. Or on reading it via Google's cache (682,000 results): User-agent: Googlebot Disallow: /axs Disallow: /cgi-bin/ User-agent: * Disallow: /cgi-bin/ Disallow: /axs/
adaviel has not submitted any stories.
Submit a story now.
robots.txt at northcountrygazette.org
Apparently there's no limit on recursively downloading the site with wget and never reading any warning. Or on reading it via Google's cache (682,000 results):
User-agent: Googlebot
Disallow: /axs
Disallow: /cgi-bin/
User-agent: *
Disallow: /cgi-bin/
Disallow: /axs/