Search Spiders and Robots Tools and Tips

AWStats free
Of AWStats is a full featured, free, powerful log analyzer. It is loaded with most of the features that you need to track your site visits, but what I think it is most useful for is tracking googlebot, yahoo Slurp (inktomi), MSN bot, and all other search robots and Spiders. It gives you a list of all spiders that have visited your site, how many hits, bandwidth, and the last visit. This information is invaluable when trying to get indexed. Simply put, no robot - spider visits means you wont get indexed.

This tool is also useful once you are indexed. You can then see how often the spiders - bots are visiting your pages.

Google Cache free
Use google to find out if your site is indexed.
type 'cache:www.yourdomain.com/yourpage.html'

This gives you the last indexed version of your site. It also gives you a date that it was indexed.