Internet means Google (at least for most of us!). It is the default page and first place to look for information. “Google It” as they say. You just type literally anything and Google won’t disappoint you!
Here’s a screenshot of a simple search query that we ran. The results it is showing are about 1,98,00,000 results and that too in 1.03 seconds. That’s crazy!
So do you have an idea how Google or any other search engine is able to find so many relevant results (read websites) for you in such a short duration?
The reason why it is important to have knowledge about this is that it helps you in managing your SEO better. Only if you have a working knowledge of how search engines work, you’ll be able to make Digital Marketing strategies work the way you want it.
In a nutshell there are three stages associated with any search engine:
- Crawling– content is discovered;
- Indexing– where content is analysed and stored in databases
- Retrieval– where the query submitted by a user fetches relevant results in the form of pages.
The journey of search engine begins with crawling. Crawling includes scanning the site and getting every information available which includes page title, images, keywords in the website and pages the website links to.
And all this is done using crawlers or bots or spiders. This bot or a spider searches every website available, very very very quickly! It then adds the list to a database in addition to re-crawling the websites to double check. It catches any related websites with matching keywords, websites linked to other sites. But if your content is not in the homepage but is present after many clicks, the bot may five up and your website may not show up the search pages.
The next and perhaps one of the most crucial tasks is taking all the data which is crawled and put it in a huge database. Just to give an idea how big the database is for every
search, imagine a library big huge number of books. Now you have to make a list of the books, authors, and their titles. And now imagine you doing this for all the libraries in the world! Sounds mind boggling. Right? But this example is a very small scale version of what Google does!
All the data is stored in vast data centers around the world. Hers an image of a data center.
This is the last step and the one which determines whether users come to your website of not. It is what you see after you type in the search query. The search engine tries to display the results based on the crawlers’ judgement of relevancy of the content to the user.
Different search engines differ in this step. Some use keywords as the deciding factor while others work with advanced options of keyword proximity, age of the content.
Few years back this was a very simple process following a simple transparent logic. After entering the keywords, search engines looked at website titles, post titles, keywords within the website content and superficially looked at the number of links leading to the website.
But now the algorithms that drive search rankings are quite refined. To give you an idea here’s a video by Google explaining how search works.