Spider

An automated program that crawls widely through the Internet and collects and indexes information, usually on behalf of a search engine or a monitoring company. A spider can often by identified through the User Agent field of a log file, or through its Internet Protocol address.