| Re: Hey youz guyz [message #259810 is a reply to message #259798] |
Sun, 20 May 2007 23:26   |
 |
Goztow
Messages: 9737
Registered: March 2005
Location: Belgium
Karma:
|
General (5 Stars)
Goztoe |
|
|
| havoc9826 wrote on Mon, 21 May 2007 05:10 |
Can't you have robots.txt tell them to leave, or would that prevent legitimate Google/Archive.org caching? Or are these crawlers the kind that ignore robots.txt?
|
Robots.txt works like a charm.
Robots can take a LOT of your bandwith. Might be good to configure your robots.txt to only allows google bot or only allow them to search the index.
You can find me in The KOSs2 (TK2) discord while I'm playing. Feel free to come and say hi! TK2 discord
[Updated on: Sun, 20 May 2007 23:26] Report message to a moderator |
|
|
|
Search
Help
Members
Register
Login
Home
