There are basically two types of search engines. The first are robots called web crawlers or spiders. Search Engines use spiders, sometime called a spiderbot to index websites. When you submit your website pages to a search engine by completing their required submission page, the search engine spider will index your entire site. A ‘spider’ is an automated program ran by a search engine system. A spider “crawls” a website. When a Spider crawls a website it reads the content, the Meta tags and also follows the links. The crawler then returns the information to a central depository that indexes the data. The spider will visit each link you have on your website and index those sites as well. Some spiders only index a certain number of pages on your website, so don’t create a site with 500 pages!
Web Crawlers usually discover pages from links within the site and from other sites. Sitemaps supplement this data to allow crawlers that support Sitemaps to pick up all URLs in the Sitemap and learn about those URLs using the associated metadata.
Spiders or Web Crawlers
A spider is almost like a book where it contains the table of contents, the actual content and the links and references for all the websites it finds during its search, and it may index up to a million pages a day.
Example: Bing and Google.
The spider will periodically return to the website to check for any information that has changed. The moderators of the search engine determine how often the spider returns.
What Happens When you use a search engine
When you ask a search engine to locate information, it is actually searching through the index which it has created and not actually searching the Web. Different search engines produce different rankings because not every search engine uses the same algorithm to search through the indices.
One of the things that a search engine algorithm scans for is the frequency and location of keywords on a web page. The algorithm can also detect artificial keyword stuffing or spamdexing. Then the algorithms analyze the way that pages link to each other.
A search engine can determine what a page is about. Especially if the keywords of the linked pages are similar to the keywords on the original page.
NEXT READ: Submitting Your Website to Search Engines
1 Comment
Pingback: The Importance Of Search Engines - Babysteps2onlinesuccess