| Top 500 Web Crawlers (Total 105) |
|
Name
 |
Hits
 |
1. |
Web Crawler - Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) |
75614 |
2. |
Web Crawler - Mozilla/5.0 (compatible; DotBot/1.1; http://www.dotnetdotcom.org/, crawler@dotnetdotcom.org) |
23909 |
3. |
Web Crawler - msnbot/2.0b (+http://search.msn.com/msnbot.htm) |
19026 |
4. |
Web Crawler - msnbot/1.1 (+http://search.msn.com/msnbot.htm) |
10909 |
5. |
Web Crawler - Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp) |
10275 |
6. |
Web Crawler - Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) |
6934 |
7. |
Web Crawler - msnbot/1.0 (+http://search.msn.com/msnbot.htm) |
4187 |
8. |
Web Crawler - msnbot-media/1.0 (+http://search.msn.com/msnbot.htm) |
1743 |
9. |
Web Crawler - Mozilla/4.0 (compatible; Powermarks/3.5; Windows 95/98/2000/NT) |
1415 |
10. |
Web Crawler - Mozilla/4.7 (compatible; OffByOne; Windows 2000) Webster Pro V3.4 |
1393 |
11. |
Web Crawler - FAST MetaWeb Crawler (helpdesk at fastsearch dot com) |
888 |
12. |
Web Crawler - ia_archiver |
420 |
13. |
Web Crawler - msnbot-media/1.1 (+http://search.msn.com/msnbot.htm) |
298 |
14. |
Web Crawler - Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html) |
237 |
15. |
Web Crawler - TurnitinBot/2.1 (http://www.turnitin.com/robot/crawlerinfo.html) |
222 |
16. |
Web Crawler - FAST Enterprise Crawler 6 used by Singapore Press Holdings (crawler@sphsearch.sg) |
211 |
17. |
Web Crawler - WebAlta Crawler/2.0 (http://www.webalta.net/ru/about_webmaster.html) (Windows; U; Windows NT 5.1; ru-RU) |
184 |
18. |
Web Crawler - MSR-ISRCCrawler |
180 |
19. |
Web Crawler - DotBot/1.0.1 (http://www.dotnetdotcom.org/#info, crawler@dotnetdotcom.org) |
179 |
20. |
Web Crawler - Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html) |
148 |
21. |
Web Crawler - msnbot/2.0b (+http://search.msn.com/msnbot.htm)._ |
119 |
22. |
Web Crawler - ia_archiver (+http://www.alexa.com/site/help/webmasters; crawler@alexa.com) |
99 |
23. |
Web Crawler - WebAlta Crawler/1.3.23 (http://www.webalta.net/ru/about_webmaster.html) (Windows; U; Windows NT 5.1; ru-RU) |
92 |
24. |
Web Crawler - Cityreview Robot (+http://www.cityreview.org/crawler/) |
69 |
25. |
Web Crawler - disco/Nutch-1.0-dev (experimental crawler; www.discoveryengine.com; disco-crawl@discoveryengine.com) |
61 |
26. |
Web Crawler - Nutch/Nutch-1.0-dev (A Nutch-based crawler.; http://lucene.apache.org/nutch/bot.html; nutch-agent AT lucene.apache.org) |
52 |
27. |
Web Crawler - EnaBot/1.1 (http://www.enaball.com/crawler.html) |
50 |
28. |
Web Crawler - EnaBot/1.2 (http://www.enaball.com/crawler.html) |
42 |
29. |
Web Crawler - SimilarPages/Nutch-1.0-dev (SimilarPages Nutch Crawler; http://www.similarpages.com; info@similarpages.com) |
35 |
30. |
Web Crawler - SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html) |
33 |
31. |
Web Crawler - adidxbot/1.1 (+http://search.msn.com/msnbot.htm) |
33 |
32. |
Web Crawler - WebAlta Crawler/1.3.25 (http://www.webalta.net/ru/about_webmaster.html) (Windows; U; Windows NT 5.1; ru-RU) |
31 |
33. |
Web Crawler - my-heritrix-crawler(+http://mywebsite.com) |
23 |
34. |
Web Crawler - ConveraCrawler/0.9e (+http://ews.converasearch.com/crawl.htm) |
21 |
35. |
Web Crawler - Mozilla/5.0 (compatible; Yahoo!-AdCrawler; http://help.yahoo.com/yahoo_adcrawler) |
19 |
36. |
Web Crawler - Mozilla/5.0 (iPhone; U; CPU like Mac OS X; en) AppleWebKit/420+ (KHTML, like Gecko) Version/3.0 Mobile/1A543a Safari/419.3 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html) |
16 |
37. |
Web Crawler - DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html) |
14 |
38. |
Web Crawler - Mozilla/5.0 (compatible; archive.org_bot/1.13.1x +http://crawler.archive.org) |
14 |
39. |
Web Crawler - Mozilla/5.0 (compatible; KaloogaBot; http://www.kalooga.com/info.html?page=crawler) |
13 |
40. |
Web Crawler - Mozilla/5.0 (Slurp/cat; slurp@inktomi.com; http://www.inktomi.com/slurp.html) |
12 |
41. |
Web Crawler - msnbot/2.0b |
11 |
42. |
Web Crawler - NextGenSearchBot 1 (for information visit http://www.zoominfo.com/About/misc/NextGenSearchBot.aspx) |
10 |
43. |
Web Crawler - Googlebot/2.1 (+http://www.google.com/bot.html) |
10 |
44. |
Web Crawler - voyager/2.0 (http://www.kosmix.com/crawler.html) |
10 |
45. |
Web Crawler - DomainCrawler/1.0 (info@domaincrawler.com; http://www.domaincrawler.com/domains/view/shuttlesinorlando.com) |
9 |
46. |
Web Crawler - Mozilla/5.0 (compatible; Nigma.ru/3.0; crawler@nigma.ru) |
9 |
47. |
Web Crawler - CyberPatrol SiteCat Webbot (http://www.cyberpatrol.com/cyberpatrolcrawler.asp) |
9 |
48. |
Web Crawler - SheenBot/SheenBot-1.0.4 (Sheen web crawler) |
7 |
49. |
Web Crawler - Nokia6820/2.0 (4.83) Profile/MIDP-1.0 Configuration/CLDC-1.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html) |
6 |
50. |
Web Crawler - kalooga/KaloogaBot (Kalooga; http://www.kalooga.com/info.html?page=crawler; crawler@kalooga.com) |
5 |
51. |
Web Crawler - CazoodleBot/CazoodleBot-0.1 (CazoodleBot Crawler; http://www.cazoodle.com/cazoodlebot; cazoodlebot@cazoodle.com) |
5 |
52. |
Web Crawler - Mozilla/4.0 compatible ZyBorg/1.0 (ZyBorg@WISEnutbot.com; http://www.WISEnutbot.com) |
5 |
53. |
Web Crawler - Microsoft URL Control - 6.00.8862 |
4 |
54. |
Web Crawler - Black Hole Finder |
4 |
55. |
Web Crawler - http://domino.research.ibm.com/comm/research_projects.nsf/pages/sai-crawler.callingcard.html |
4 |
56. |
Web Crawler - GingerCrawler/1.0 (Language Assistant for Dyslexics; www.gingersoftware.com/crawler_agent.htm; support at ginger software dot com) |
3 |
57. |
Web Crawler - SimilarPages/Nutch-1.0-dev (SimilarPages Nutch Crawler; http://www.similarpages.com; info at similarpages dot com) |
3 |
58. |
Web Crawler - Googlebot/2.1 (+http://www.googlebot.com/bot.html) |
3 |
59. |
Web Crawler - ArabyBot (compatible; Mozilla/5.0; GoogleBot; FAST Crawler 6.4; http://www.araby.com;) |
3 |
60. |
Web Crawler - Willow Internet Crawler by Twotrees V2.1 |
3 |
61. |
Web Crawler - Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html) |
3 |
62. |
Web Crawler - Mozilla/2.0 (compatible; NEWT ActiveX; Win32) |
3 |
63. |
Web Crawler - VisBot/2.0 (Visvo.com Crawler; http://www.visvo.com/bot.html; bot@visvo.com) |
3 |
64. |
Web Crawler - asterias/2.0 |
3 |
65. |
Web Crawler - Clustera Crawler/Nutch-1.0-dev (Clustera Crawler; http://crawler.clustera.com; cluster@clustera.com) |
3 |
66. |
Web Crawler - SheenBot/SheenBot-1.0.2 (Sheen web crawler) |
2 |
67. |
Web Crawler - Mozilla/5.0 (compatible; Googlebot/2.1;+http://www.google.com/bot.html) |
2 |
68. |
Web Crawler - UFAM-crawler-0.1 |
2 |
69. |
Web Crawler - NetMechanic V4.0 |
2 |
70. |
Web Crawler - SheenBot/SheenBot-1.0.1 (Sheen web crawler) |
2 |
71. |
Web Crawler - lwp-trivial/1.41 |
2 |
72. |
Web Crawler - ichiro/2.0 (http://help.goo.ne.jp/door/crawler.html) |
2 |
73. |
Web Crawler - Mozilla/4.0 (compatible; Vagabondo/4.0; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/; http://www.wise-guys.nl/) |
2 |
74. |
Web Crawler - renlifangbot/1.0 (+http://search.msn.com/msnbot.htm) |
2 |
75. |
Web Crawler - librabot/1.0 (+http://search.msn.com/msnbot.htm) |
2 |
76. |
Web Crawler - RedBot/redbot-1.0 (Rediff.com Crawler; redbot at rediff dot com) |
2 |
77. |
Web Crawler - netEstate NE Crawler |
2 |
78. |
Web Crawler - SapphireWebCrawler/Nutch-1.0-dev (Sapphire Web Crawler using Nutch; http://boston.lti.cs.cmu.edu/crawler/; mhoy@cs.cmu.edu) |
2 |
79. |
Web Crawler - multicrawler (+http://sw.deri.org/2006/04/multicrawler/robots.html) |
2 |
80. |
Web Crawler - BrightCrawler (http://www.brightcloud.com/brightcrawler.asp) |
2 |
81. |
Web Crawler - Teemer (NetSeer, Inc. is a Los Angeles based Internet startup company.; http://www.netseer.com/crawler.html; crawler@netseer.com) |
1 |
82. |
Web Crawler - ICC-Crawler/2.0 (Mozilla-compatible; ; http://kc.nict.go.jp/project1/crawl.html) |
1 |
83. |
Web Crawler - ichiro/4.0 (http://help.goo.ne.jp/door/crawler.html) |
1 |
84. |
Web Crawler - SheenBot/SheenBot-1.0.3 (Sheen web crawler) |
1 |
85. |
Web Crawler - NetWhatCrawler/0.06-dev (NetWhatCrawler from NetWhat.com; http://www.netwhat.com; support@netwhat.com) |
1 |
86. |
Web Crawler - KSCrawler/Nutch-1.0 (http://www.kindsight.net/en/kscrawler; crawler@kindsight.net) |
1 |
87. |
Web Crawler - msnbot/1.0+(+http://search.msn.com/msnbot.htm) |
1 |
88. |
Web Crawler - Jayde crawler |
1 |
89. |
Web Crawler - Bloodhound/Nutch-0.9 (Testing Crawler for Research - obeys robots.txt and robots meta tags ; http://balihoo.com/index.aspx; robot at balihoo dot com) |
1 |
90. |
Web Crawler - www.fi crawler, contact crawler@www.fi |
1 |
91. |
Web Crawler - nutch.us/Nutch-1.0 (www.nutch.us; crawler@nutch.us) |
1 |
92. |
Web Crawler - PRCrawler/Nutch-0.9 (data mining development project; crawler@projectrialto.com) |
1 |
93. |
Web Crawler - Mozilla/4.0 (compatible; BorderManager 3.0) |
1 |
94. |
Web Crawler - VWBOT/Nutch-0.9-dev (VWBOT Nutch Crawler; http://vwbot.cs.uiuc.edu; vwbot@cs.uiuc.edu) |
1 |
95. |
Web Crawler - Mozilla/5.0 (compatible; Yahoo-AdCrawler; http://help.yahoo.com/yahoo_adcrawler) |
1 |
96. |
Web Crawler - MQBOT/Nutch-0.9-dev (MQBOT Nutch Crawler; http://vwbot.cs.uiuc.edu; mqbot@cs.uiuc.edu) |
1 |
97. |
Web Crawler - ICC-Crawler(Mozilla-compatible; http://kc.nict.go.jp/icc/crawl.html; icc-crawl-contact(at)ml(dot)nict(dot)go(dot)jp) |
1 |
98. |
Web Crawler - SheenBot/SheenBot-1.0.0 (Sheen web crawler) |
1 |
99. |
Web Crawler - ilial/Nutch-0.9 (Ilial, Inc. is a Los Angeles based Internet startup company.; http://www.ilial.com/crawler; crawl@ilial.com) |
1 |
100. |
Web Crawler - Mozilla/5.0 (Yahoo-MMCrawler/4.0; mailto:vertical-crawl-support@yahoo-inc.com) |
1 |
101. |
Web Crawler - Spock Crawler (http://www.spock.com/crawler) |
1 |
102. |
Web Crawler - PR Crawler/Nutch-1.0 (data mining develpment project; crawler@projectrialto.com) |
1 |
103. |
Web Crawler - hitcrawler_0.1 ringken@gmail.com |
1 |
104. |
Web Crawler - Mozilla/5.0 (compatible; Megaglobe Crawler/1.0; http://www.megaglobe.com) |
1 |
105. |
Web Crawler - ichiro/3.0 (http://help.goo.ne.jp/door/crawler.html) |
1 |