22/07/2015
If I told you that there are billions of creepy crawlies all over the world that aren’t found in any biology book – tiny little monsters so small you can’t see even them but so clever – so intelligent -that they have become vital to human civilisation, you would probably think I was mad, silly, or at least had a ‘vivid imagination’.
While at least one (and possibly all three) of these are true, it is actually quite an apt analogy.
However… Let’s start from a slightly different angle – all this talk of creepie crawlies is giving me the heebie jeebies! Instead of an analogy, how about a question:
How does Google work?
It seems that nobody can even think of talking about the internet without mentioning Google in the same breath. This is because the internet itself is HUGE – it is the size of every computer connected to it – far too big to imagine really. This creates a bit of a problem – it constantly changes, shifts and morphs into something different, all of the time. A piece of text or media which was in one place on one computer yesterday might not be there today, and it might be replaced by something else entirely tomorrow.
To solve this problem of size and changeability, Google does two things which make life amazingly simple (which is the reason it’s so popular). The first one you probably know about already – the search engine – the ability to type in a question, phrase or comment and have returned (almost instantly) a long list of relevant results from all over the world, categorised and sorted by preferences.
The second thing Google does is just as important though. Perhaps you have wondered yourself – how it does it? How does it know all of the information on every website all around the world?
Well, believe it or not, Google has little spies called web crawlers.
They are designed to work just like humans visiting websites – they take a look around, indexing the information on each page, noting who else is connected and making little notes, then… they leave.
After a long days hard web crawling, they then report all of this information back, which Google then use to populate a section within a huge database. And when I say huge… I mean HUGE! To give you some idea of just how big… if you wanted to display all of Google’s indexed websites on one monitor, the screen would have to be 6 million miles from corner to corner – roughly 241 times the length of the equator.
Google then, is an enormous index of information collected by these little web crawlers.
Robots took my job!
So far, most of the information is common sense – of COURSE Google don’t employ millions (or billions) of people to constantly update their website index! They are a software company – it’s the perfect job for indexing robots which are literally made to fetch data very fast when asked. Again, when I say fast, I mean FAST – if you took one minute to search each page on Google it would take 38, 026 years to look at them all. Google does this in under 0.5 seconds.
What might not be quite so obvious is how these little bits of indexing software can be used to your advantage!
First – here is a tip – web crawlers are designed to notice differences, and indeed to be sensitive to even slight changes. If they visit a website and find a difference compared to the last time they visited, two things happen. First – they do a full scan of the site (rather than just a partial check if nothing has changed) this means they update and refresh all of the relevant keywords and calculate a brand new ‘Quality Score’ (a rating from 1-10 on topical relevance and authority).
Second, they decide to revisit the site soon to see if there are any more changes. Being robots and not humans, they don’t rationalise very well – meaning they can’t decide if things are complete or not. So, because they can’t say that ALL of the changes that were going to be made have been made already, they check back much sooner than they would if there had been no changes.
To give you an idea, web crawlers typically visit a site every 3-6 months. If, however, there are regular changes to the site (say once a week) then the web crawler will constantly remind itself to keep on visiting!
So, how does that affect me?
Well – let’s use an example.
Let’s say you are a lawyer, working for one of the largest law firms in the world. You are in the middle of a really large international case and you find out a vital piece of information that sets a new precedent for national and international law. You want to make the most of this information and show your business is at the forefront of your field. So, you go onto your firm’s website and make a post about the case, explaining how it will change the future!!! You update your SEO with the new words, get your partners to link from their website to yours.
And…
Nothing happens….
The information is there, as plain as day, but nobody is visiting the firm’s website!
An hour passes. Still nothing. Two hours. Nope. Four hours. Nada.
When you type into Google about the case, the first hit is a news page which (after only 5 minutes of being live) has over 500 thousand likes! Outraged, you visit the media website and realise that your firm has not even been named!!!
Yes, people are busily talking about the details of the case, but you can’t help feeling cheated – even though you beat them (literally) to the ‘post’, they were able to get the scoop, and your business didn’t even get mentioned.
So why did the media company get so many visits, and how did the story come up on Google right away? Well, the law firm does not update their information very often, whereas the media company are constantly talking about new things – posting hourly or even less – meaning the web crawlers are working overtime trying to keep up!
The last thing to say about web-crawlers is that, while they are very small, they are also pretty clever – if they realise a website changes their information very regularly, they monitor it more and more closely. So – the more changes you make to a website, or the more often a site has new content, the more often a web crawler will visit – ready to update the Google index.
In many ways this ability to be discovered is quickly becoming one of the most valuable commodities on the internet – far from holding just information, websites which have specific elements (such as being tracked closely by Google web crawlers instantly) make it more suitable and applicable for advertising and back-linking investment.
Okay, so, with all this in mind – what can I do?
Well, a little bit of information goes a long way. If you have something urgent and important to say then use sites or channels that are likely to be updated quickly – if you use a site that does not post regularly, it may be months before you can find it on Google.
In addition, if you want your site to update quickly, for example to promote competitions or deliver key information to customers, then preparation is key. The easiest, fastest and cheapest way to ensure that your site is indexed regularly by Google is to create a blog – around 1000 words in a post, preferably posted weekly.
If you are able to post every week for three months then you can be sure that information that you post on the site will appear on Google within hours or days, not weeks or months!
