add a robots.txt

Yahoo! has been pounding on ikiwiki.cgi again. While I'd prefer ikiwiki to generate sites that avoided robots hitting it in other ways, I'm adding a robots.txt, at least temporarily.
2008-11-10 18:08:09 -05:00 · 2008-11-10 18:08:09 -05:00 · 1ea66c3d3f
parent f0e58faefa
commit 1ea66c3d3f
1 changed files with 2 additions and 0 deletions
--- a/doc/robots.txt
+++ b/doc/robots.txt
@ -0,0 +1,2 @@
+User-Agent: *
+Disallow: /ikiwiki.cgi