** This snippet checks incoming UserAgent against a known bot list, and time-stamps the visit if a match. The admin page botspottingadmin.mgi should be visited first to generate the database, and this code should only be used occasionally for bot sniffing -- as the database could potentially balloon on heavy traffice sites.
To prevent runaway monitoring, we recommend using an include on the desired monitored page like this where the day of data collection is in the form of yyyymmdd:
20040707 psv **
AbachoBOT
abcdatos_botlink
AESOP_com_SpiderMan
ah-ha.com crawler
ia_archiver
Scooter
Mercator
Scooter2_Mercator_3-1.0
roach.smo.av.com-1.0
AltaVista-Intranet
FAST-WebCrawler
Wget
Acoon Robot
antibot
Atomz
Buscaplus Robi
CanSeek
ChristCRAWLER
Crawler
RaBot
contact/jylee@kies.co.kr
RaBot
DeepIndex
DittoSpyder
Jack
Speedy Spider
ArchitextSpider
ArchitectSpider
Arachnoidea
EZResult
Fast PartnerSite Crawler
FAST Data Search Crawler
KIT-Fireball
GalaxyBot
geckobot
(Genealogical Search Engine)
Googlebot
moget/2.0
Aranha
Slurp.so/1.0
Slurp/2.0j
Slurp/2.0-KiteHourly
Slurp/2.0-OwlWeekly
Slurp/3.0-AU
Hubater
(research centre)
IlTrovatore-Setaccio
IncyWincy
UltraSeek
InfoSeek Sidewinder
Mole2/1.0
MP3Bot
C-PBWF-ip3000.com-crawler
kuloko-bot/0.2
LNSpiderguy
NetResearchServer
MantraAgent
NetResearchServer
Lycos_Spider_(T-Rex)
HenryTheMiragoRobot
MSNBOT/0.1
Gulliver
PicoSearch
PJspider
nttdirectory_robot
griffon
Spider/maxbot.com
gazz/1.0
NationalDirectory-SuperSpider
dloader(NaverRobot)
Openfind piranha,Shark
Openbot
psbot
CrawlerBoy Pinpoint.com
AlkalineBOT
Fluffy the spider
Scrubby
asterias
speedfind ramBot xtreme
Kototoi/0.1
Searchspider
SightQuestBot
Surfnomore Spider v1.1
teoma_agent1
ESISmartSpider
Tutorial Crawler
UK Searcher Spider
appie
Nazilla
marvin/infoseek
MuscatFerret
WhizBang! Lab
ZyBorg
WIRE WebRefiner:
WSCbot
Yandex
pet-based search engine
libwww-perl
Iron33
Gotcha!