Google Press Day – Eustace On Crawling

    May 10, 2006
    WebProNews Staff

Alan Eustace is Google’s Senior Vice President, Engineering & Research, and during his talk at Google’s media event he handled a few questions and expounded on the virtues of the crawl.

Regular WebProNews readers know all about search engine crawling. For those just joining us, welcome, and enjoy some of the material discussed by Dr. Eustace, including the crawl, as related to us by managing editor Mike McDonald.

“Crawling is just computer-speak for processing a page and all the links on that page, and all the links on that page, etc,” he said. “The important thing to remember is that speed matters.”

“The last time we posted a size of the index it was 8 billion. It’s a lot bigger now.”

Along with speed, Eustace cited the importance of avoiding duplicate results and keeping the index fresh. Eustace estimated that 10 to 20 percent of the Web changes month by month. “Index size matters, since queries have a very long tail. What you’re searching for is completely different than what someone else is searching for,” he said.

“We want the result that you want to be number one. We want people to be able to push the “I’m Feeling Lucky” button and get the best result every time.”

For new companies aiming for Google’s perch atop the search world, think again. Eustace said that as many as a quarter of queries Google sees are ones they have not seen before. “This is why small indexes are not going to be successful,” he said. “A lot of people don’t realize it, but thousands of machines work in concert for every query.”

Spam causes Google problems, and Eustace cited the importance of Google having algorithms that understand site quality very well. He also cautioned on link trading due to spammers getting involved with that activity.

Eustace handled a few questions during his time on stage, and we have some of them here:

Q: What’s being done to improve mobile search?
A: One thing we’re working on is transcribing, taking a page from a large screen and making it available for small screen

Q: How do you make video search a better experience?
A: In video, you have less text associated so therefore you have to ask users to label things for you, using popularity and clicks and things like that. What people expect out of video is different than what people expect out of search.

Q: Datacenters work great for search, but why don’t they seem to handle other services very well?
A: We use the same infrastructure for everything. We don’t make any guarantees that all of our products have the same degree of reliability. We of course try to maintain uptimes but there may be a product in beta, for example, that isn’t on all data centers.

Q: How are you thinking about the use of personal information in making search better?
A: If you understand what people have done over time, search can improve. For example, bass and jaguar: if we know a little more about you we have a better idea of what you’re looking for. My view is some level of knowledge built up over queries is useful.

Q: How do you measure internally the quality of your search?
A: I can’t go into details about how we measure quality. We feel like our search experience is improving, and that we feel like we have a lead over all of our competitors.

Add to | DiggThis | Yahoo! My Web | Furl

Bookmark WebProNews:

David Utter is a staff writer for WebProNews covering technology and business.