Microsoft has announced a ‘major evolution’ in search-engine indexing. IndexNow lets website owners notify search engines about new content or updates. A simple ‘ping’ from a specific URL lets participating search engines know that content has been added, updated or deleted.
Takeaways
- In the statement announcing IndexNow, Microsoft highlighted that one of the ‘biggest pain points’ for website owners is the time it takes search engines to index website changes.
It can take days or even weeks for new URLs to be discovered and indexed in search engines, resulting in loss of potential traffic, customers, and even sales.
- When content is published to a website, or updated, IndexNow notifies all participating search engines, dramatically reducing the time that search-engine indexing takes, according to SearchEngine Journal’s Roger Montti.
- The open-source indexing protocol – developed by Microsoft Bing and leading Russian search engine Yandex – is open to all search engines with a ‘noticeable presence’ in at least one market. IndexNow will begin sharing up to 10,000 URLs a day in November 2021.
From pull to push
Currently, most search engines index web content following a site visit from a search-engine crawler. The crawler visits the website, requests pages and ‘pulls’ data from the web server into the index.
- IndexNow turns this ‘pull’ process on its head, allowing web publishers to ‘push’ content changes out to participating search engines. The advantage for publishers is that they control the process and no longer have to wait for search engines to discover content changes.
- Microsoft is presenting the IndexNow protocol as an initiative for a more efficient Internet. With website owners prioritising URLs where content has been added or amended, the need for speculative crawls to test if the content has changed is greatly reduced.
- Large companies including eBay, LinkedIn, and GitHub are said to be planning to migrate to IndexNow from the Bing URL submission API. Content Management Systems are also being encouraged to adopt the protocol, as are Content Delivery Networks and the SEO industry.
How to adopt IndexNow
Developers will be able to generate an API key using Bing’s online key generation tool. They should host the key in a text file named with the value of the key at the root of their website and submit an HTTP request containing the URL and key when content is added, updated, or deleted. You can submit one URL or a set of URLs per API call.
Non-developers will need to wait for their content platform to adopt IndexNow.
While Wix and Duda have acknowledged the importance of IndexNow and plan to adopt the protocol, WordPress hasn’t committed yet. Microsoft has submitted the IndexNow open-source code to WordPress for inclusion into the core, but there has been no decision yet.
WordPress aside, Roger Motti at SEJ sees IndexNow as a big deal.
It’s a major change in how search engines can discover updated and newly published documents. It benefits publishers with fast indexing and less server load from bots constantly crawling their web pages.Adoption by major search engines will be an evolution in the relationship between publishers and search engines that benefits both.
