John Mueller placed a useful post from Zurich on the Webmaster Central blog at Google, where he delves into the issue of meta tags. Back in the day, meta tags like ‘keywords’ helped webmasters get their sites indexed appropriately.

It took next to no time for spammers to start clogging ‘keywords’ to the point where they became useless. As Mueller reminded everyone in an answer to a comment, Google isn’t looking at them for indexing purposes:

(W)e generally ignore the contents of the "keywords" meta tag. As with other possible meta tags, feel free to place it on your pages if you can use it for other purposes – it won’t count against you.

Plenty of tags do work favorably for pages, as do Sitemaps, which enjoy support from Google, Yahoo, and Microsoft. Meta tags can control robot behavior, or in the case of an individual crawler like Google’s Googlebot, affect a single robot’s actions with certain directives like meta name=robots or name=googlebot:

Google understands the following values (when specifying multiple values, separate them with a comma):

noindex: prevents the page from being indexed (see "Block or remove pages using meta tags")

nofollow: don't follow links from this page when looking for new pages to crawl (also see "Block or remove pages using meta tags")

nosnippet: don't show a snippet of this page when displaying it in the search results (see "Prevent or remove snippets")

noodp: don't use text from ODP (The Open Directory Project a.k.a. dmoz.org) to generate a title or snippet for this page (see "How do I change my site's title and description?")

noarchive: don't display a "Cached" link for this page in the search results (see "Prevent or remove cached pages")

unavailable_after:[date]: remove this page from the search results after the specified date and time (see "Robots Exclusion Protocol: now with even more flexibility")

Mueller noted the default rule is "index, follow" when this meta tag is not in place on a page, or if it is not targeted to the visiting spider when it arrives.

follow me on Twitter

The Google Public Policy blog noted the Senate Homeland Security and Government Affairs Committee will consider S. 2321, a Reauthorization of the E-Government Act. This extension will have OMB draw up guidance and best practices for agencies to enable search crawlers to index their websites effectively.

Google recommended Sitemaps as an easy way for agencies to comply with the Act’s requirements. The search company has helped several federal and state agencies, and the Library of Congress, implement Sitemaps for improving the site’s search visibility.

Website architecture at government sites could be hindering people from finding the information they need, in Google’s view. Google estimates about half of government websites suffer from these architecture issues, and Sitemaps can help alleviate the problem.

]]> http://www.webpronews.com/senate-considers-better-spidering-as-law-2007-11/feed 0 http://www.webpronews.com/paid-search-engine-inclusion-programs-the-details-inktomi-altavista-2004-03 http://www.webpronews.com/paid-search-engine-inclusion-programs-the-details-inktomi-altavista-2004-03#comments Mon, 15 Mar 2004 14:02:34 +0000 Dirk Brockhausen http://www.webpronews.com/?p=9233 Following our overview of current paid search engine inclusion (PFI) programs, we will now take a closer look at the technical procedures involved, spidering activity, quality of service, etc. This first instalment deals with Inktomi and AltaVista, more will follow in a later article.

Inktomi

Inktomi’s current submission partners are: ineedhits.com, Outrider, Position Technologies and VeriSign.

Outrider is PositionTech’s London based partner. Personnel connections exist between PositionTech and Outrider via Detlef Johnson, the former Supervisor Search Engine Optimization Services with Outrider, now functioning as VP of Technology with PositionTech.

We registered new domains with all four partners and tracked the ensuing spider visits.

ineedhits.com

The Inktomi spider arrived the day following page submission:

#UA Mozilla/3.0 (Slurp/si; slurp@inktomi.com;
http://www.inktomi.com/slurp.html)
j6002.inktomi.com
209.185.141.249

Regularly, this spider will crawl your pages every second day. However, we detected exceptions from the rule. E.g. following the turn of the month, the spider took 4 days to come back.

Beyond the pages registered for paid inclusion, the domain URL (e.g. http://www.domain.com/ ) was spidered.

This will normally occur every 5 days, with the site’s robots.txt file being crawled as well.

However, the domain URL hasn’t to date been included in the Inktomi index.

About two weeks following paid submission, this Inktomi spider started to fetch the pages in irregular intervals:

#UA Mozilla/5.0 (Slurp/cat; slurp@inktomi.com;
http://www.inktomi.com/slurp.html)
wm3018.inktomi.com
216.35.116.103

Beyond Spidering

Spider visits apart, what’s important for the submitting webmaster is, of course, the PFI partners’ Management Area or control center.

In ineedhits.com’s “Subscription Management Area” you will see your URLs’s current status.

The “Click-thru hits report” displays the number of hits and the search terms which generated them. You can click on the links to either AOL, HotBot or MSN to see the pertinent search results page. According to ineedhits.com: “Click-Thru reports currently only reflect a small percentage of the entire click-thrus Inktomi Search Partners refer to your URL.”

This is no exaggeration: in our case, only about 75% of hits and keywords actually generated were displayed in this report. Obviously, this calls for some serious improvement.

Outrider

About 1 to 2 hours after submission, our site was visited by the PositionTech spider:

#UA Mozilla/4.0 (compatible; MSIE 4.01; Windows NT)
169.207.238.180

It was a one-hit affair only: we haven’t seen this spider visit our site again since.

Another two hours later, the Inktomi crawling cycle commended with a visit from this spider:

#UA Mozilla/3.0 (Slurp/si; slurp@inktomi.com;
http://www.inktomi.com/slurp.html)
j6002.inktomi.com
209.185.141.249

Following this, the spider started visiting the site in the same rhythm as with ineedhits.com.

Beyond Spidering

In Outrider’s “Account Service” section you will see your URLs’s current status and the number of generated clicks. According to Outrider: “Click-Thru reports reflect less then half of all traffic generated by Inktomi.”

This can be confirmed on our part: on average, less than 50% of generated clicks were reported.

PositionTech

The PositionTech spider arrived one hour after submission:

#UA Mozilla/4.0 (compatible; MSIE 4.01; Windows NT)
169.207.238.180

Another hour later, the Inktomi spider cropped up:

#UA Mozilla/3.0 (Slurp/si; slurp@inktomi.com;
http://www.inktomi.com/slurp.html)
j6002.inktomi.com
209.185.141.249

Ensuing spider behavior was identical to submissions via ineedhits.com and Outrider.

Beyond Spidering

In PositionTech’s subscribers area you will see your URLs’s current status and the number of generated clicks as well as the keywords entered by visitors. In our case, only 40% of actually generated hits and keywords were displayed.

Here, there is no pointer to the fact that not all clicks will be correctly displayed. This calls for improvement.

Click-thru data can also be checkd in a graphical view.

A very nifty feature exclusively offered by PositionTech is their “Pure Web Search” function. Under URL < http://169.207.238.189/search/ > a search form for pure Inktomi is offered. Search results are uncluttered by “enhancements” such as Overture PPC links, ODP stuff, ads, etc. This URL is accessible to the general public (at least, currently).

VeriSign

One day after submission, the Inktomi spider arrived:

#UA Mozilla/3.0 (Slurp/si; slurp@inktomi.com;
http://www.inktomi.com/slurp.html)
j6002.inktomi.com
209.185.141.249

Ensuing spider behavior was identical to submissions via the other partners.

Beyond Spidering

In VeriSign’s subscribers area you will see your URLs’s current status and the number of generated clicks as well as the keywords entered by visitors. However, this did not work in our case. Although our access log files registered several hits, VeriSign did not display so much as a single one of them.

Inktomi PFI Summary

* Pricing is identical, regardless of submission partner.

* Submitting your pages to any of the partners will trigger the Inktomi spider, which will then crawl your pages following its own rhythm, independent of chosen submission partner. Partners do seem to hold a certain influence on commencement of the spidering cycle, though.

* With PositionTech and Outrider it took a mere 1-2 hours for the first Inktomi spider to arrive. They are also the only submission partners checking sites with their own (shared) spider.

* Inktomi seems to have devoted a single spider for crawling pages under the PFI program.

* Crawled pages will normally show up in the Inktomi index about one day after spidering.

* PositionTech’s subscribers area control center offers the most comprehensive functionality, making it the superior service.

However, seeing that no submission partner will currently process 100% of generated hits, a systematic analysis of your own server traffic logs cannot be discarded.

AltaVista’s infoSpider

Paid inclusion for AltaVista is handled by their independent partner infoSpider.

Shortly after submission, a spider will check the submitted URL:

#UA infoSpider URL Checker
208.185.243.149.proboost.com
208.185.243.149

Four days after, the first AltaVista spider arrived on site:

#UA Scooter-3.0.3
scooter2.sv.av.com
209.73.162.172

Pages were included in AltaVista’s index the following day.

The spidering cycle is determined by AltaVista’s index runs for paid inclusions, which is about once per week.

The AltaVista spider(s) will visit submitted pages one, two or three days prior to this index run. There is currently no regular, predictable rhythm discernible as with Inktomi.

AltaVista has devoted at least one other spider to crawling PFI pages:

#UA Scooter-3.2
scooter3.sv.av.com
209.73.162.143

Beyond Spidering

In infoSpider’s Service section, you can check on your URLs’ current status. No hits and keywords data is being offered.

Compared to Inktomi’s PFI program, AltaVista’s refresh cycles are considerably longer, and the service area offered features only very basic functionality.

Dirk Brockhausen is the co-founder and principal of
fantomaster.com GmbH (Belgium), < http://fantomaster.com/ >,
a company specializing in webmasters software development,
industrial-strength cloaking and search engine positioning
services.

He holds a doctorate in physics and has worked as an SAP
consultant and software developer since 1994. He is
also Technical Editor of fantomNews, a free newsletter
focusing on search engine optimization, available at:
< http://Fantomaster.com/fantomnews-sub.html >
You can contact him at
mailto:fntecheditor@fantomaster.com