'Crawlers' & 'Indexing' In SEO

Today we'll take a look behind the scenes at one of our most dependable life companions.

The search engine.

What is the search engine?

A search engine is an online program that searches for items in a database that link to keywords specified by a user and is used especially for finding specific sites on the World Wide Web. The most famous search engine is Google, others being 'Bing', 'Yahoo', 'Ask' etc. Search engines are so popular that research suggests there are 100 billion global searches being conducted each month and which can access over 1 trillion unique webpages available online.

A quick question before we begin!

How many of you have relied on 'Google' for example, to help with your homework, to answer your embarrassing questions or to simply do a bit of research?

I'm guessing there are quite a few and so, in this post we'll explore how we receive these answers, where they come from and who is responsible.

The online experience has improved over the years and the expectations of its users has increased. Unfortunately, most search engines may struggle to understand the sophistication of a websites content, making it very difficult to produce the credible, relevant results we seek.

Subsequently, search engines use programmes called 'Crawlers' alternatively known as 'Bots' or 'Spiders'.


As you may have guessed, Crawlers are used to crawl.

Where? - They travel across the entire Web in search of online activity and updates.

How? - Through links, which direct them to countless pages, websites and documents.

Why? - In order to identify and store any changes made to new or existing sites. Some of the most active Crawlers will visit the same site several times a day, checking for any updated content.

What's the point? - So when a user conducts a search query, the answers they need are easily accessible and retrievable.


Crawlers to the rescue!

They're able to store found information in a huge Index. Much like a library. They accumulate pages and 'save' them so search engines know how and where to find information for search queries, fast!

Search engine companies have datacentres all over the world so when it comes to an online search, information can be retrieved within a matter of seconds. Research tells us, speed does matter! An experiment was taken to measure the user's reaction when a search results page had a slower delivery. The experiments demonstrate that slowing down the search results page by 100 to 400 milliseconds has a measurable impact on the number of searches per user. - http://googleresearch.blogspot.co.uk/2009/06/speed-matters.html

With the Web continuously growing and developing, Crawlers can be very valuable to a search engines performance.

Next week, we'll take a look at the 'Algorithmic process' Crawlers follow and the foundations of a crawlers journey.

Sign up for our newsletter and see you next week!

Published: 17 Aug 2015

Author: Alix McGregor