怎样知道蜘蛛是否来过自己的网站

要想知道搜索引擎蜘蛛是否来过自己的网站，其实很简单，只要你的服务器开启了日志，就可以通过日志文件查看哪些蜘蛛来过自己的网站，例如：

2009-11-20 05:33:13 W3SVC9119222 61.191.191.120 GET /show.asp id=411 80 - 220.181.7.51 Baiduspider+(+http://www.baidu.com/search/spider.htm) 200 0 0 17811
2009-11-20 05:33:17 W3SVC9119222 61.191.191.120 GET /show.asp id=410 80 - 220.181.7.103 Baiduspider+(+http://www.baidu.com/search/spider.htm) 200 0 0 17423
2009-11-20 05:33:21 W3SVC9119222 61.191.191.120 GET /show.asp id=408 80 - 220.181.7.20 Baiduspider+(+http://www.baidu.com/search/spider.htm) 200 0 0 19268
2009-11-20 05:33:24 W3SVC9119222 61.191.191.120 GET /show.asp id=375 80 - 220.181.7.45 Baiduspider+(+http://www.baidu.com/search/spider.htm) 200 0 0 20700
2009-11-20 05:41:09 W3SVC9119222 61.191.191.120 GET /show.asp id=424 80 - 220.181.7.31 Baiduspider+(+http://www.baidu.com/search/spider.htm) 200 0 0 19049
2009-11-20 05:41:12 W3SVC9119222 61.191.191.120 GET /show.asp id=308 80 - 220.181.7.17 Baiduspider+(+http://www.baidu.com/search/spider.htm) 200 0 0 19063
2009-11-20 05:51:51 W3SVC9119222 61.191.191.120 GET /show.asp id=403 80 - 220.181.7.52 Baiduspider+(+http://www.baidu.com/search/spider.htm) 200 0 0 21880
2009-11-20 06:02:52 W3SVC9119222 61.191.191.120 GET /index.html - 80 - 220.169.61.3 Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1;+SV1;+.NET+CLR+2.0.50727) 200 0 0 40483
这个不用说大家都知道


2009-11-17 06:00:34 W3SVC9119222 61.191.191.120 GET /robots.txt - 80 - 61.135.249.12 Mozilla/5.0+(compatible;+YoudaoBot/1.0;+http://www.youdao.com/help/webmaster/spider/;+) 200 0 0 357
2009-11-17 06:00:36 W3SVC9119222 61.191.191.120 GET /index.html - 80 - 61.135.249.12 Mozilla/5.0+(compatible;+YoudaoBot/1.0;+http://www.youdao.com/help/webmaster/spider/;+) 200 0 0 40819
2009-11-17 06:00:54 W3SVC9119222 61.191.191.120 GET /index.html - 80 - 61.135.249.12 Mozilla/5.0+(compatible;+YoudaoBot/1.0;+http://www.youdao.com/help/webmaster/spider/;+) 200 0 0 40819
这说明有道蜘蛛来过自己的网站


2009-11-17 13:37:29 W3SVC9119222 61.191.191.120 GET /robots.txt - 80 - 67.195.111.31 Mozilla/5.0+(compatible;+Yahoo!+Slurp;+http://help.yahoo.com/help/us/ysearch/slurp) 200 0 0 376
2009-11-17 13:37:33 W3SVC9119222 61.191.191.120 GET /index.html - 80 - 67.195.111.31 Mozilla/5.0+(compatible;+Yahoo!+Slurp/3.0;+http://help.yahoo.com/help/us/ysearch/slurp) 200 0 0 40776
这是雅虎蜘蛛


2009-11-17 13:49:35 W3SVC9119222 61.191.191.120 GET /show.asp id=310&thisPage=3 80 - 203.208.60.206 Mozilla/5.0+(compatible;+Googlebot/2.1;++http://www.google.com/bot.html) 200 0 0 18031
2009-11-17 13:50:01 W3SVC9119222 61.191.191.120 GET /show.asp id=310&thisPage=4 80 - 203.208.60.205 Mozilla/5.0+(compatible;+Googlebot/2.1;++http://www.google.com/bot.html) 200 0 0 18619
2009-11-17 13:51:27 W3SVC9119222 61.191.191.120 GET /show.asp id=310&thisPage=2 80 - 203.208.60.207 Mozilla/5.0+(compatible;+Googlebot/2.1;++http://www.google.com/bot.html) 200 0 0 18398
2009-11-17 13:53:30 W3SVC9119222 61.191.191.120 GET /show.asp id=333 80 - 203.208.60.206 Mozilla/5.0+
这是谷歌蜘蛛


等等。。。。。。。。
   一般情况下，搜索引擎不会匿名去抓取，所以会在你的服务器上留下蛛丝马迹，大家可以多看看自己的网站日志文件，就可以知道哪些蜘蛛来过自己的网站了。