Google currently is reported to have indexed 8 billion pages and counting. Google utilizes an array of bots A.K.A. spiders or crawlers. Among the specialized bots Google uses include: The web spider Googlebot, the Adsense spider MediaBot, the image spider ImageBot, the AdWords spider AdsBot, the RSS feed spider FeedFetcher-Google, and Googlebot-Mobile spider for mobile devices. MSN & Yahoo, the other two of the 'big three' have their own proprietary versions of spiders. Why is it important for an Internet Marketer to know how spiders crawl your website? A search engine crawler is your best visitor. Giving a crawler easy and uninhibited movement in your website is necessary for good search engine rankings.

Your website must be spider (search engine) friendly if you want any traffic from the search engines. A search engine spider does not read your website the way we humans do. The spider reads web-page source code (HTML) that renders your page, therefore ‘bad code’ can be an impediment to the spider, sometimes causing it to give up crawling your website. Spiders love content (text) and do not read JavaScript at all, therefore a website that is packed with images with no ALT tags to assist the spiders, and heaps of JavaScript may not be indexed successfully. So, when designing your website you must incorporate structural website design principals that elicit search engine friendliness. An astute marketer should also desire to see how search engines see his or her site. This may be accomplished by a Lynx Viewer which is a text-mode web browser. Additionally, a Lynx Viewer can help you determine if your web pages are accessible to the vision impaired, an assemblage of visitors that should not be ignored ---yes, there are millions of visually impaired people surfing the Internet regularly.