Have you ever wondered how search engines find websites and rank their content for search result pages? The major sites like Google, Yahoo and Bing all have their own spiders that crawl the entire net hopping from one link to another.

A search engine spider is an automated software program that is designed to follow links from one page to another gathering information on each page it visits. The spider reads the HTML or other files that contains the coding for your business website design. All the clickable links and the keywords from the readable text are extracted. This includes other information such as meta tag content and the last time the page was modified.

The search engine spiders tend to ignore any scripts or external information contained on the page. For example, any links on the page coded in Javascript are ignored by most or all search engines. Also, any embedded displays like iframes or embedded videos will not be relayed by the spiders to the search engines.

The data that is extracted will be analysed by the search engine’s page ranking algorithm. Each page is ranked for relevancy according to the keywords found on the page.

Webmasters can instruct the search engine spiders not to look at certain files or folders on their website by using a file known as the robots.txt. The robots.txt file is also known as the Robots Exclusion Standard and the Robots Exclusion Protocol. In the form of an ordinary .txt file, the robots.txt is placed in the root directory of the website’s directory. The site owner specifies what files or directories that search engine spiders should not include in search engine results.

The robots.txt file is used together with the website sitemap to help search engine spiders crawl through a website. Sitemaps are also stored in the website’s root and act as an inventory of all pages on the web site that the site owner wants included in search engine results. Sitemaps are particularly helpful in showing spiders how to find dynamic web pages that may be difficult for web crawlers to find on their own. There are a number of software programs that will generate sitemaps automatically on a routine basis. However, having a sitemap does not guarantee that search engines will include any specific page in search results.

All ClickWebDesign business website packages include correct robots.txt files, automatic sitemap generation, and google friendly design.