“Google News recently updated our infrastructure to crawl with Google’s primary user-agent,Googlebot. What does this mean? Very little to most publishers,” says Google News Product Specialist David Smydra. “Any news organizations that wish to opt out of Google News can continue to do so: Google News will still respect the robots.txt entry for Googlebot-News, our former user-agent, if it is more restrictive than the robots.txt entry for Googlebot.”

“Although you’ll now only see the Googlebot user-agent in your site’s logs, no need to worry: the appearance of Googlebot instead of Googlebot-News is independent of our inclusion policies,” says Smydra. “You can always check whether your site is included in Google News by searching with the “site:” operator. For instance, enter “site:yournewssite.com” in the search field for Google News, and if you see results then we are currently indexing your news site.”

As far as analytics, you’ll still be able to differentiate traffic from Google Search and traffic from Google News, Google says.

Sites using Google’s metered subscription model or the first click free model won’t have to make any changes, but sites that require registration, payment or login before reading the full article, Google News will only be able to crawl and index the title and snippet that’s shown on the page.

Google stresses that the change will not affect how it crawls your News sitemaps.

More info in the help center.

Original Article: Google has created a new web crawler specifically for Google News. What this means is that publishers who do not want Google News to index their content can more easily control that. That also applies to publishers who don’t wish to completely cut out indexing, but wish to limit/restrict certain elements of their content from being indexed.

Google offers this new crawler at a time when Google’s relationship with online news is a heavy focus of discussion throughout the industry, with the FTC’s meeting of the media minds taking place. This week Google already announced some changes to how it handles paid content (by offering a five-article limit for the "first click free" plan). Now the company appears to be further extending its olive branch to concerned publishers (whether or not that will be enough is another discussion).

In the past, publishers have been able to block Google from content via robots.txt and the Robots Extension Protocol (REP). They have also been able to keep content out of Google News and stay in Google Search, by using a contact form provided by Google. Now, Google is making it so publishers don’t even have to contact them.

"Now, with the news-specific crawler, if a publisher wants to opt out of Google News, they don’t even have to contact us – they can put instructions just for user-agent Googlebot-News in the same robots.txt file they have today," says Google News Senior Business Product Manager Josh Cohen. "In addition, once this change is fully in place, it will allow publishers to do more than just allow/disallow access to Google News. They’ll also be able to apply the full range of REP directives just to Google News. Want to block images from Google News, but not from Web Search? Go ahead. Want to include snippets in Google News, but not in Web Search? Feel free. All this will soon be possible with the same standard protocol that is REP."

"While this means even more control for publishers, the effect of opting out of News is the same as it’s always been," says Cohen. "It means that content won’t be in Google News or in the parts of Google that are powered by the News index. For example, if a publisher opts out of Google News, but stays in Web Search, their content will still show up as natural web search results, but they won’t appear in the block of news results that sometimes shows up in Web Search, called Universal search, since those come from the Google News index."

Cohen says Google News users shouldn’t notice any difference in their experience with the service. It will be interesting to see the reaction from disgruntled publishers, and whether or not this will make any significant difference in how they view Google News.

Related Articles:

> Google Changes How it Handles Paid Content

> Minds of the Media Gather to Discuss Future of News

> Google Okay With Blocking News Corp.

> Is it Really Crazy to Block Google?

Robots.txt holds the key for site publishers to gain proper indexing of their content by search engines. Scrupulous crawlers obey the tenets of robots.txt, spidering what the file allows them to do and avoiding paths disallowed by the webmaster.

Google posted a note about their robots.txt generator, housed in their Webmaster Tools. It permits the creation of blanket robots.txt files, or ones with more granular designations about robots and where certain ones can and cannot go.

Of course, a knowledge of robots.txt syntax and a few minutes in vi does the same thing, but there may be webmasters who prefer the comfort of a clean graphical interface to a text editor.

Google also has the advantage of a robots.txt analyzer in Webmaster Tools. This allows site publishers to test out the file and see if any of its contents could be problematic for arriving spiders.

They also noted a couple of caveats about robots.txt. First, not every search engine supports all of the possible extensions to the robots.txt standard. Second, there are unscrupulous crawlers that will ignore the file and grab whatever they can. Sensitive content should be either password protected if it needs to be online.

“ACAP will give content owners the confidence to allow search engines to index their content under clear terms of use,” the organization behind ACAP’s development said in their FAQ.

“To date, many aggregation websites have chosen to adopt a liberal attitude to copyright – ‘it’s OK until someone tells us it isn’t’ – which means there is an enormous amount of infringing material being hosted by major companies,” said the group.

That “liberal attitude” has been Google’s regular contention about not just online content, but that of its book search project. Google’s book scanning has angered publishers due to the search giant’s position that publishers need to actively opt-out, rather than Google proactively seeking permission to index.

Publishers consider robots.txt too simplistic with its allow/disallow choices for spiders and directories or types of content. “These simple choices are inconsistently interpreted,” ACAP claimed; search engines will likely find that opinion a surprising one.

ACAP extends what robots.txt presents to crawlers, in a standard form. For example, a time limit value can be defined, telling the crawler the publisher wants certain content to expire and be removed from an index after a given date or period of time.

From a cursory review of the technical documentation, ACAP looks like a way for publishers to establish usage guidelines that they can utilize in lawsuits against search engines. Though publishing groups have backed the standard, major search engines like Google and Yahoo are not represented in ACAP’s supporters.

This could be a step toward gearing up for a fight with the search engines to force them to comply with ACAP as they crawl available online content. We won’t be surprised if the search engines respond by disallowing the spidering of any ACAP-enhanced sites until everyone reaches a public common ground where they accept ACAP as an extension of the robots.txt standard.

follow me on Twitter

Moderator:

• Danny Sullivan, Conference Co-Chair, Search Engine Strategies San Jose

Speakers:

• Peter Linsley, Sr. Product Manager, Ask.com
• Evan Roseman, Software Engineer, Google, Inc.
• Sean Suchter of Yahoo! Search.
• Eytan Seidman, Microsoft

First to speak is Eytan Seidman from Microsoft. He shows a presentation on Microsoft’s Live Webmaster Portal which explains how Microsoft’s crawler will index your site. Live Webmaster Portal supports map submissions and one can also view their website’s statistics. Microsoft has many search engine crawlers and all their names begin with "MSNBot" -

• web search
• news
• academic
• multimedia
• user agent

Microsoft also supports "NOODP" and "NOCACHE" tags.

Next is yahoo! Search’s Sean Suchter who also has a presentation about Yahoo’s crawler.

dynamic URL rewriting via Site Explorer "Robots-nocontent" tag. Yahoo! employs crawler load improvements (reduction and targeting). The new Yahoo! search engine crawler targets better and has a comparatively low volume.

Google’s Evan Roseman steps up to explain and discuss webmaster central’s features. He recommends taking advantage of Webmaster central’s submit a site option so that Google’s search engien crawler can index all your content.

Next up is Ask.com’s Peter Linsley who discusses catering to the search engine robot as many times in catering to the actual human visitor, the robot is forgotten. Some problems include requiring cookies. He points out that Ask does accept site map submissions but points out that they’d rather be able to crawl naturally.

Peter uses the Adobe site to demonstrate some issues that they may have with multiple domains and duplicate content. He then uses the Mormon.org site and shows that they are disallowing crawlers to index the root page. This creates problems with crawling.

Q & A

• Q: First question is for the Google rep. Wants to know whether they will allow users to see supplemental results within Webmaster Central now that they are no longer tagging them in search results.
• A: Evan stated that being in supplemental is not a penalty but did not provide a definite answer as to whether they would allow users to discover if or not results are supplemental.

  Danny interjects that all engines have a two-tier system and Eytan, Sean and Peter confirmed that. So… they all have supplemental indices but people only seem to be concerned with Google’s, most likely because they used to identify them as such in the regular search results.

• Q: What can a competitor actually do if anything to hurt your site?
• A: Evan says that there is a possibility where a competitor could hurt your site but did say it is extremely difficult. Hacking, domain hi-jacking are some of the things that can occur.
• Q: Question relates to scenario when you re-publish content to places such as eBay but the sites you re-publish to rank better than original. How can a webmaster identify original source of information?
• A: Peter answers that one could try to get places they republish content to use robots.txt to block spidering of content. Another thing to do is have link back to original site. However on a site such as eBay, that is not always possible. The response to that is to create unique content for these sites that this person is re-publishing content on.
• Q: Robert Carlton asks if all engines are moving towards having things like Webmaster Centrals. Also asks how they treat 404s and 410s.
• A: As for 404s and 410s, Ask, Google and Yahoo! treat them the same. Robert points out that they should treat them differently as a 410 indicates the file is gone whereas 404 is an error.
• Q: Question regarding getting content crawled more frequently.
• A: Evan suggest to use the Site Map feature in Webmaster Central and keep it up to date. He also suggest promoting it by placing a link to it on the home page of their site.
• Q: How can one use site maps more effective for very larges site that have information changing on a regular basis? Also inquired how to get more pages indexed when only a portion are being indexed.
• A: Submitting a site map with Google is not going to cause other URLs to not be crawled. Evan also points that they are not going to be able to crawl and include ALL the pages that are out there. Again suggests that webmaster promote them such as listing them on home page. However when dealing with hundreds of thousands of pages, that is not always feasible.
• Q: How do engines interpret things like AJAX, JavaScript, etc.?
• A: Eytan answered that if webmaster wants things interpreted, they are going to have to represent those in a format the engine can understand, AJAX and JavaScript currently not being one of them.
• Q: Question regarding rankings in Yahoo! disappearing for three weeks but then they get back in. Is his due to an update?
• A: Sean answers that it certainly could be and suggests using Site Explorer to see if there is some kind of issue.
• Q: How many links will engines actually crawl per page? How much is too much?
• A: Peter says there is no hard and fast rule but keep the end user in mind. Evan echoes the same feeling.
• Q: Do the engine use meta descriptions?
• A: All engines use them and may use them if the algorithm feels they are relevant.
• Q: For sites that are designed completely in Flash, can you use content in a "noscript" tag or would that be considered as some type of cloaking?
• A: Sean said IP delivery is a no-no but if the content is the same as Flash, he’d rather see content in noscript than traditional cloaking. Evan suggests avoiding sites in complete Flash but rather use Flash components.
• Q: Is meta keywords tag still relevant?
• A: Microsoft – no, Yahoo! – not really, Google – not really, and Ask – not really. All read it but it is has so little bearing. For a really obscure keyword where it only appears in the keyword tag and no where else on the web, Yahoo! and Ask are the only ones that will show a search result based on it.
• Q: How do engines view automated submission/ranking software?
• A: Evan – don’t use them.

Comments

Tag:

Unfortunately a robots.txt file or robots meta tag won’t do that, but they can help the crawlers to index your site better and block out the unwanted ones. First a little definition explaining:

Search Engine Spiders or Crawlers – A web crawler (also known as web spider) is a program which browses the World Wide Web in a methodical, automated manner. Web crawlers are mainly used to create a copy of all the visited pages for later processing by a search engine, that will index the downloaded pages to provide fast searches.

A web crawler is one type of bot, or software agent. In general, it starts with a list of URLs to visit. As it visits these URLs, it identifies all the hyperlinks in the page and adds them to the list of URLs to visit, recursively browsing the Web according to a set of policies.

Robots.txt – The robots exclusion standard or robots.txt protocol is a convention to prevent well-behaved web spiders and other web robots from accessing all or part of a website. The information specifying the parts that should not be accessed is specified in a file called robots.txt in the top-level directory of the website.

The robots.txt protocol is purely advisory, and relies on the cooperation of the web robot, so that marking an area of your site out of bounds with robots.txt does not guarantee privacy. Many web site administrators have been caught out trying to use the robots file to make private parts of a website invisible to the rest of the world. However the file is necessarily publicly available and is easily checked by anyone with a web browser.

The robots.txt patterns are matched by simple substring comparisons, so care should be taken to make sure that patterns matching directories have the final ‘/’ character appended: otherwise all files with names starting with that substring will match, rather than just those in the directory intended.

Meta Tag – Meta tags are used to provide structured data about data.

In the early 2000s, search engines veered away from reliance on Meta tags, as many web sites used inappropriate keywords, or were keyword stuffing to obtain any and all traffic possible.

Some search engines, however, still take Meta tags into some consideration when delivering results. In recent years, search engines have become smarter, penalizing websites that are cheating (by repeating the same keyword several times to get a boost in the search ranking). Instead of going up rankings, these websites will go down in rankings or, on some search engines, will be kicked off of the search engine completely.

Index a site – The act of crawling your site and gathering information.

How can the robots.txt file and meta tag help you?

In the robots.txt you can tell the harmful ‘web crawlers’ to leave your web site alone, and give helpful hints to the ones you want to crawl your site. Below is an example on how to disallow a web crawler to search your site:

# this identifies the wayback machine User-agent: ia_archiver Disallow: /

ia_archiver is the crawler name for the wayback machine that you may have heard of, and the / after disallow tells ia_archiver not to index any of your site. The # allows you to write comments to yourself so you can keep track of what you typed.

Type the above three lines into notepad from your computer and save it to the root directory of your web site as robots.txt. Web crawlers look for this document first at a web site before doing anything else. This helps the crawler to do its job, and helps the web site owner tell the spider what to do. Say for instance you have some data that you don’t want the crawlers to see. (Like duplicate content for other browser referrer pages)

You can deter crawlers from indexing the ‘duplicate’ directory by typing this into your robots.txt file.

User-agent: * Disallow: /duplicate/

The * after user-agent says that this action applies to all crawlers and /duplicate/ after disallow tells all crawlers to ignore this directory and not search it. For each user-agent and disallow line there must be a blank space between them in order for it to function correctly. So this is how you would create the above two commands into a robots.txt file:

# this identifies the wayback machine User-agent: ia_archiver

User-agent: * Disallow: /duplicate/

One thing to note that is very important: Anyone can access the robots.txt file of a site. So if you have information that you don’t want anyone to see don’t include it into the robots.txt file. If the directory that you don’t want anyone to see is not linked to from your web site the crawlers won’t index it anyway.

An alternative to blocking indexing of your site is to put a meta tag into the page. It looks like this:

You put this into the tag of your web page. This line tells the robot crawlers not to index (search) the page and not to follow any of the hyperlinks on the page. So as an example tells the robot crawlers to not index the page, but follow the hyperlinks on this page.

Did You Know That Google Has Its Own Meta Tag?

It looks like this: . This tells the Google robot crawler not to index the page, not to follow any of the links, and not to keep from storing cached versions of your web site. You will want this done if you update the content on your site frequently. This prevents the web user from seeing outdated content that isn’t refreshed because of storage in the cache.

You can use the meta tag to specifically talk to Google’s robots to avoid complications or if you are optimizing your site for Google’s search engine. Recommended software tools to automate submitting and link creation : “http://blog-submitter.cafe150.com” – Blogs AutoFiller

Maksym Nesen is leading programmer of the Oksima team. he developed great product – Blogs Auto Filler which saves time and money for people who used to blogs advertising

http://blog-submitter.cafe150.com

A new breed of crawler’s are coming to a site near you!

Many higher caliber web development teams are spawning off a breed of crawlers that seem to index your valuable and copywrighted content for their own use and not for the searchers themselves.

I’m talking about millions of pages online that are being generated in order to attract more search engines and visitors to their site.

Have you ever noticed a portion of your content listed on some obscure page that has 50 other companies listed with yours and are evenly stuffed with the same key words???

Everywhere you search, you’re bound to come across one of these pages. Some would say that this is great for their business by being listed within these pages but I truly believe that this style of indexing hurts our industry as a whole.

This is nothing short of spam and keyword stuffing!

I mean, what are we teaching the newcomers to the search marketing industry? That this style of “FFA Pages” is acceptable and in order to attract 1000′s of visitors, you too should stuff a ton of pages with someone else’s content!

For a long time FFA pages (Free For All) were a thing of the past. But, in the midst of battle, a new breed of FFA has sprung up and taken the search marketing industry by storm.

I call this new breed; “TFFA” – Targeted Free For All!

These TFFA pages can be found on some of the most prominent websites world wide. It’s not just the little guys that indulge themselves and hide behind this form of search marketing spam.

Unfortunately, the industry is turning a blind eye!

Hey, if the big guys are doing it, then it’s ok!? NO! On the other hand, until these TFFA pages become more and more exposed as nothing short of spam tactics, we will continue to see our personal and valuable content being ripped right out from underneath our noses.

Martin Lemieux is the owner of Smartads. We help companies like yourself to market your business online and offline.

For more of Martin’s articles, go here: http://www.smartads.info/newsletter/archive

The Martin Report – eBusiness News!: http://www.smartads.info/the-martin-report/

SEM XML Feed: http://www.article99.com/rss-feeds/Search-Engine-Marketing.xml

Rob Sullivan put together a delightful article at SearchEngineJournal on the background and current life of search engine crawlers. He covers their wee beginnings at MIT up to the hordes crawling the sprawling internet we know and love today.

He also tries to explain a few things regarding what people might see when the do searches based on their particular topic:

You may also notice, upon reviewing your reports, that crawlers like Googlebot will visit repeatedly and request the same page(s) repeatedly. This is common as crawlers also want to be sure the site is stable and also to measure the page’s change frequency.

He goes on to explain crawler behavior and how they work, including certain behavior patterns and even breaks it down somewhat based on individual companies like Yahoo or AskJeeves.

If you want some incite on how crawlers work, this is an excellent article.

John Stith is a staff writer for WebProNews covering technology and business.

The “Meet The Crawlers” session was no exception.

Join the discussion on web crawlers here.

Weaving the Web

Craig Neville-Manning, senior research scientist at Google, had some great advice for webmasters (and a pointed barb for Yahoo – we’ll get to that though).

“We don’t like pointing users to pages that you change,” said Craig, describing the practice of cloaking. You should avoid cloaking at all costs. As Tim Mayer of Yahoo pointed out during our lunch chat, there’s a legitimate use for every potential spam technique.

For example, you can use cloaking to show search engines an optimized page and then, say, a Flash intensive page when users arrive. Despite cloaking’s legitimate uses though, authorities recommend that you don’t do it.

Concerned you might be cloaking? Read this page cloaking article.

Craig revealed a good rule of thumb for optimizers – Google’s algorithm values text and links that your site visitors can see more highly than anything they can’t see. This means focus your efforts more on explicit, helpful, and keyword-focused links, as well as copy that informs your visitors.

For those concerned that the Google bot uses too much bandwidth he mentioned that it can detect when your server is slowing down and it will back off. Also, the Google bot follows the robots.txt file to the letter.

If you have content you don’t want the bot to find be sure to put the robots.txt file up to keep it out. The bot, says Craig, can find content that’s unlinked. That’s right, the Google bot can find single pages dangling unlinked in space. He didn’t explain how this happens.

The question and answer session revealed a bit of how the Google looks at keywords in the url. Someone from the audience asked about how Google views words in the url, whether you should hyphenate them or not for added relevancy and ranking.

Craig said that Google does index words from the url, but they don’t have as much weight as text links. He added quickly though that you should not engineer your links for the algorithm – it’s better to have your url meaningful to your visitors than use it to affect your ranking.

Tim Mayer of Yahoo seconded this. He said focusing too much on url engineering can get you into the realm of over-optimization. “As a user,” said Tim, “if I see a domain with lots of hyphens it’s usually a low quality site.” He advised that you not push your filenames too hard, and that you have intuitive directory structures.

In the Link Building Strategies session, prominent seo guru Greg Boser said “hyphenated domains have come and gone.”

The big talk at this conference was the new Yahoo paid inclusion program, which allows webmasters to pay to show up in Yahoo’s primary search results. At the close of Google employee Craig’s presentation he declared, in a comment obviously leveled at fellow presenter Tim Mayer of Yahoo, “our search results are not for sale.”

Tim, ever the gentleman, let it slide.

Garrett French is the editor of iEntry’s eBusiness channel. You can talk to him directly at WebProWorld, the eBusiness Community Forum.