I have a database logging all website visits. How can I separated between real visitors and crawlers?
You can use Google Analytics for the best results, and the most data regarding your visitors. To have an idea what information they can offer you: http://www.icantinternet.org/2009/01/stats-show-you-what/
If you want to track it yourself, then you should use the “user agent string”, which holds the identification of the browser (or the spider in that case). I've written an article on how to use this string to identify an iPhone visitor, but by simply changing the string to look for, it also works for spiders: http://www.icantinternet.org/2009/01/recognizing-the-iphone-for-webpages/
This entry was posted
on Saturday, June 20th, 2009 at 1:22 pm and is filed under website traffic.
You can follow any responses to this entry through the RSS 2.0 feed.
You can leave a response, or trackback from your own site.
2 comments so far
Leave a reply