Live Search Tweaks CrawlerBy: WebProNews Staff - February 15, 2008
The Live Search Webmaster Center team announced earlier this week that Microsoft made some improvements to its Live Search webcrawler to improve indexing efficiency.
The two main improvements, blogs, Fabrice Canel, are with HTTP Compression and with Conditional Get:
HTTP Compression: HTTP compression allows faster transmission time by compressing static files and application responses, reducing network load between servers and Microsoft’s crawler. Microsoft says it supports the most common compression methods: gzip and deflate as defined by RFC 2616.
Conditional Get: Live Search now supports conditional get as defined by RFC 2616, meaning that Live Search will not download a page unless it has changed since the last time it was crawled. The crawler includes the "If-Modified-Since" header and time of last download in the GET request and when available includes the "If-None-Match" header and the ETag value in the GET request.
More information and links can be found at the Live Search Webmaster Center blog .