There aren’t many in Malta talking about crawl budgets and it’s no wonder, given that it is one of the most technical SEO concepts, sometimes ignored and frequently misunderstood.
What is the crawl budget?
The crawl budget, as the word itself says, is a real budget, in terms of resources, that Google dedicates to crawl the pages of our site and then add them to its index.
From a business point of view, the practical activity of periodically scanning the billions of websites on the internet to monitor any updates and changes, represents a cost for Google. It is therefore clear that it tries to optimise and minimise these costs by dedicating different resources to each website based on a series of factors.
We can therefore say that the crawl budget can be defined as the number of times Google downloads and crawls our files, pages and images in a given period of time.
There may be sites that are visited by the Google bot every day, several times a day and sites that are crawled once a week or a month. The higher the crawl budget, the more the site matters for Google and deserves to be crawled.
How important is it to monitor the crawl budget?
Typically, for a small 10-15-20 page corporate site, the crawl budget is not an important issue, it can be important though, in the presence of large sites such as eCommerce with thousands of products, but also blogs and editorial sites with thousands of articles. In this case it makes sense to know how to manage this budget, what to scan and when and what not.
The situations in which the crawl budget can generally be “ignored” are:
- Small websites
- When new pages or new articles are indexed immediately
- The site has less (indicatively) than a thousand URL
It is important to pay attention to the crawl budget instead in the presence of:
- A site with thousands of URLs
- A lot of duplicate content
- Many URLs with “query string parameters”
- Changes to the site are not quickly indexed
- Many 404 errors
- Infinite url (there may be elements like a calendar that can create these situations)
Can you optimise your crawl budget?
The best way to increase the crawl budget is certainly to carry out all basic SEO activities, such as:
- Publishing useful and valuable content
- Obtaining natural and quality links
- Good publication frequency
- Optimisation of server site performance
All factors that contribute to increasing site trust. However, there are other technicalities on which one can intervene:
- Robots.txt file
- Xml sitemap
- Parameters
- Broken links
- Canonicals
- Redirects
Conclusions
Many SEO specialists and web masters, unfortunately, either for the high technical aspect of the topic and for the lack of information, tend to ignore the crawl budget. However, with the increase in online stores and large websites it is an aspect that must not be underestimated and that can make the difference and determine the success of positioning.