Fortunately, most robots on the web follow a simple protocol by which you can keep them off your server if you wish, or keep them out of portions of your server which are robot traps (ie, they contain an infinite number of possible links). Read the document World Wide Web Robots, Wanderers and Spiders (URL is <URL:http://web.nexor.co.uk/mak/doc/robots/robots.html> ) and learn about the emerging standards for exclusion of robots from areas in which they are not wanted. You can also read about existing robots there, including useful cataloging robots you probably do not want to keep off your server.