Got Crawl Issues? Live Search Can Help

    August 6, 2008
    WebProNews Staff

Microsoft launched its first major update to Live Search Webmaster Tools since November, adding crawl issue feedback, backlink data, improved filtering, and the ability to export data in CSV format to Excel for offline analysis. Update: Five minutes after publishing, the Live Search Webmaster Team let us know they were delayed a little, and that the update should go live at 5:00 EDT.

WebProNews got a sneak peak of the update from Live Search’s Jeremiah Andrick and Nathan Buggia, who say, "search has become mission critical" for websites. This update to the Webmaster Tools service is intended to help webmasters achieve better rankings in Live Search, and to provide a two-way dialog between webmasters and Live Search.

In other words, it answers a lot of FAQs, provides a lot of data, and eliminates a lot of theorizing in terms of penalties and/or achieving higher placement. "No other search engine provides this data," Buggia assured us.

The update revolves around a few frequently posed questions about improving a website’s relationship with a search engine. Is my site being crawled and indexed? Did the Live Search bot encounter any issues while crawling my site? Have I incurred any penalties? Who’s linking to my site? Is my robots.txt file configured properly?

Naturally, in response to questions, Live Search identified four common "crawl issues"

  • Broken links/File Not Found (404): Webmaster Tools will list URLs Live Search tried to crawl but was unable to due to removed content or misspelled link.
  • Robots Exclusion Protocol: Obviously, if robots.txt, a meta tag, or HTTP header attribute is used to block a search crawler, it’s generally on purpose. On the off chance a URL is blocked inadvertently, Webmaster Tools will provide a list of links not indexed for this reason.
  • Your link’s too long: Live Search flags URLs with "exceptionally long query strings" for fear of an infinite loop scenario (Google calls them "infinite spaces"). The "Long Dynamic URLs" report helps webmasters identify links they should consider shortening. How long is too long? The team suggests that if it won’t fit into an email subject line, it’s too long.
  • Unsupported Content-Types: Live Search likes content types like HTML, XML, Powerpoint, etc., and maintains a list of acceptable content types. Unsupported content-types, like .exe files, are not supported and will not be indexed by Live Search.

Buggia says that, unlike Google, Live Search does not penalize for backlinks coming from shady neighborhoods. Nevertheless, a webmaster likes to know where his or her traffic originates. For that reason, Webmaster Tools now has a filter functionality that allows webmasters to track backlinks from specific domains or subdomains ( for example), and from top level domains (.fr, .uk, etc.).

Sometimes that kind of data can be overwhelming for websites with a lot of pages, which is why the filtering functionality allows webmasters to filter results by up to two subdomains or subfolders.

An interesting new feature is the ability to export data in CSV (comma separated values) form to Excel or other reporting tool for examination offline. You list of crawl issues or backlinks, for example, can be put directly into Excel.

The Live Search team also said they now provide penalty notifications and descriptions.

Listen to the audio