Thursday, November 13, 2008

On-Demand Indexing for a fast-moving web

Tom Duerig and Nicholas Weininger, Software Engineers

Earlier this year when we launched Google Site Search, and AdSense for Search started using the Custom Search platform, we created a special Custom Search Engine (CSE) index for enhanced indexing. Webmasters could submit Sitemaps in Webmaster Tools, and the Custom Search platform indexed URLs from these Sitemaps into a special CSE index for more comprehensive coverage.

This solved an important problem: the need for enhanced index coverage for site search. The special CSE index, in addition to the Google index, enabled us to search deeper on selected sites, providing higher search quality.

Today, with our launch of On-Demand Indexing, we're addressing the need to provide fast indexing for your important, frequently-changing content. On-Demand indexing allows anyone with a Google Custom Search Engine or Google Site Search to identify and tell us about new pages or recent site changes by submitting a Sitemap to our Webmaster Tools. You can then select this specific Sitemap in the Indexing tab of your CSE control panel and hit the new "Index Now" button. We will immediately schedule the relevant pages for crawling and indexing, and these pages will be included in your search results usually within 24 hours - often much faster.





We realize that new or frequently updated content is often crucial to a website. In order to give you more control over how you use the On-Demand Index resources allocated to you, we will honor the priority and lastmod attributes that you provide in your Sitemap - this means you can identify what URLs should be given preference. Please ensure that your Sitemap metadata is up-to-date.

On-Demand Indexing uses a special index that is designed just for Custom Search. The main Google search indexing process is separate and uses different selection criteria, so submitting pages for On-Demand Indexing will not make them appear any faster in the main Google index, or impact ranking on Google.com. For more information on On-Demand Indexing, as well as the allocation per CSE, please refer to our FAQ.

We are always listening at our Custom Search discussion group, so please continue to tell us what features you'd like to see added to Custom Search.