How do search engines actually work?
Search engines are used by people when they have a query and are searching online for the answer.
Search engine algorithms are computer programmes that look for clues to give searchers the exact results they are looking for. Search engines rely on algorithms to find web pages and decide which ones to rank for any given keyword. Remember there’s also social media algorithms to consider for search.
There are three steps to how search engines work: crawling, indexing and ranking.
Step 1: Search Engine Crawling
The first step is crawling. Search engines send out web crawlers to find new pages and record information about them. We sometimes call these web crawlers ‘spiders’,‘robots’ or Googlebots.
Their purpose is to discover new web pages that exist, and to periodically check the content on pages they’ve previously visited to see whether they’ve changed or been updated.
Search engines crawl web pages by following links they’ve already discovered. So if you have a blog and it’s linked from your homepage, when a search engine crawls your homepage, it will then look for another link and may follow the link to your new blog post.
Step 2: Search Engine Indexing
The second step is indexing. Indexing is when a search engine decides whether or not it is going to use the content that it has crawled. If a crawled web page is deemed worthy by a search engine, it will be added to its index.
This index is used at the final ranking stage. When a web page or piece of content is indexed, it is filed and stored in a database where it can later be retrieved. Most web pages that offer unique and valuable content are placed into the index. A web page might not be placed in the index if:
The content is considered duplicate
The content is considered low value or spammy
It couldn’t be crawled
The page or domain lacked inbound links
Top tip: You can check your indexed pages by typing ‘site:yourdomain.com’ in the search bar (see our example below). This will show you the pages showing up on Google. For a more detailed report you can look at the ‘Index Coverage report’ in Google Search Console.
Google Search Console site search
Google Search Console site search
Step 3: Search Engine Ranking
The third step is really the most important, and that is ranking. Ranking can only happen after the crawling and indexing steps are complete. So once a search engine has crawled and indexed your site, your site can be ranked.
There are more than 200 ranking signals that search engines use to sort and rank content, and they all fit under the three pillars of SEO: technical optimization, on-page optimization, and off-page optimization.
Some examples of signals that search engines use to rank web pages are:
Keyword in title tag – Whether the keyword or a synonym was mentioned on the page and within the title tag
Loading speed – Whether the web page loads quickly and is mobile-friendly
Website reputation – Whether the web page and website is considered reputable for the topic being searched for
Backlinks – the quantity and quality of backlinks to a web page
Content quality and relevance – Is the content valuable and relevant to the website’s target audience?
Mobile friendliness – Are pages, content and images optimized for mobile?
