Google Site Search On-Demand Indexing Triggers Re-Crawling

    November 16, 2008
    Chris Crum

Google has introduced some new functionality to its Site Search product, which allows customers to search within your own site. The feature that everyone is talking about is the ability to add pages on-demand to be indexed by site search, so customers have access to these new pages as soon as they’re ready.

Google Site Search

Some wonder why such a tool isn’t available for Google Web Search. "OK, Google, so when do we get the same functionality for instant indexing by Google web search? That would be cool," writes Andy Beal at Marketing Pilgrim. InfoWorld reports however, that the feature gives control to web publishers over the frequency with which Google refreshes its index of the site’s content, or in other words triggering a re-crawl.

"On demand indexing is like supercharging the Site Search product so that our customers — the Web site owners — can come to Google as their Web sites are changing and say ‘I just added 100 or 1,000 pages — please index them now,’" says Google Lead Product Manager for Enterprise Search, Nitin Mangtani.

Andrew Goodman at Traffick spoke with Mangtani further about Google Site Search:

I asked Nitin specifically if "smart" algorithm ranking factors like clickstream and behavioral data would be used to bubble relevant results to the top. By all means, the advantages of Google’s proprietary technology are brought to bear on every query, was his answer. But here is where some customers might get uneasy. There is an uncertain mix between rank weightings you can customize yourself, and black box ranking factors that will run in Googly fashion and never be revealed to you. Those who wish to control or understand their site search algorithm won’t enjoy this black box product, and this is where Google’s status as a public search engine that must guard its ranking secrets against spammers becomes as much of a liability as a strength, given that now, you’re guarding the algorithm against potential developers of custom ranking solutions for localized use, as opposed to would-be rank mavens trying to game the system in the case of the adversarial indexing environment of the public Web.

Even still, web publishers should be happy about the on-demand indexing as to get their sites crawled as often as possible. With On-Demand indexing:

– Site owners get an "Index Now" button to quickly and easily update their site search results with new and updated content.

– New pages are searchable within hours – taking no longer than a day to appear within site search results.

Adobe is one company that has utilized Site Search to great advantage, thus far. “On-Demand Indexing was essential for our recent launch of Adobe Creative Suite 4, the biggest software release in the company’s history,” said Tanya Wendling, senior director for Learning Resources at Adobe. “Google Site Search made it easy to implement search across our Creative Suite product line and online sites, and we are now able to index thousands of new pages and make them available to millions of users worldwide within hours. For more on Site Search,  Google provides resources to learn the ins and outs.