|
Probably the most important step in getting your site found in a search engine is the one in which the search engine crawls it. There are things that can be done and things that can be avoided to make this process as painless as possible for the search engine, which will in turn, make it as painless as possible for the webmaster.
Since Google dominates the search market share by such a large market share, it is always a good idea to listen to what they have to say about such matters. So when they post a presentation with tips on optimizing crawling and indexing, you’ll probably want to pay attention.
Google has done just that, highlighting things to stay away from, and things you can do to enhance your site’s crawlability. Here is that presentation with specific examples of URLs.
"The Internet is a big place; new content is being created all the time," says Google Webmaster Trends Analyst Susan Moskwa. "Google has a finite number of resources, so when faced with the nearly-infinite quantity of content that’s available online, Googlebot is only able to find and crawl a percentage of that content. Then, of the content we’ve crawled, we’re only able to index a portion."
"URLs are like the bridges between your website and a search engine’s crawler: crawlers need to be able to find and cross those bridges (i.e., find and crawl your URLs) in order to get to your site’s content," continues Moskwa. "If your URLs are complicated or redundant, crawlers are going to spend time tracing and retracing their steps; if your URLs are organized and lead directly to distinct content, crawlers can spend their time accessing your content rather than crawling through empty pages, or crawling the same content over and over via different URLs."
If you want to get crawled faster by Google, you should remove user-specific details from URLs. Specifics of this can be viewed in the slideshow. Basically, URL parameters that don’t change the content of the page, should be removed and put into a cookie. This will reduce the number of URLs that point to the same content, and speed up crawling.
Google says infinite spaces are a waste of time and bandwidth for all, which is why you should consider taking action when you have calendars that link to infinite numbers of past/future dates with unique URLs, or other paginated data.
Tell Google to ignore pages it can’t crawl. This includes things like log-in pages, contact forms, shopping carts, and other pages that require users to perform actions that crawlers can’t perform themselves. You can do this with the robots.txt file.
Finally, avoid duplicate content when possible. Google likes to have one URL for each piece of content. They do recognize that this is not always possible though (because of content management systems and what have you), which is why the canonical link element exists to let you specify the preferred URL for a particular piece of content.
These are very generic tips. I guess all SEO professionals would know about it. I expected Google to shell out something more exclusive…
I think sometimes it helps to consider that not everyone is a seasoned pro. There are always going to be people who still have to learn. Lots of webmasters who can’t afford to hire SEOs may appreciate tips for how they can improve their crawlability on their own.
Yeah… I suppose so. Just that when I read the title of the post, I got all excited about what would follow. And was disappointed after I read through all of it. Anyways, you are doing a good job with your posts
Yes google is lot of information resource. Great information Thank U
Great tips. The cost of hiring someone for SEO has gone through the roof. Thanks for the tips on this I look for more. As mentioned not everyone is a pro so these type of releases are worth their weight in gold.
What is the consensus opinion regarding hyphenated URLs? Good idea or not? Thanks in advance.
Chris what you meant by ” user-specific details “. Can you please tell me?
Advance thanks
I’m glad Google is trying to help and I’m sure as someone pointed out the tips are useful, but I agree they are generic and I was hoping for something more in depth. I don’t think we’ll ever get that from Google though. Thanks, though, Chris for keeping us up on the new information coming out on Google and search engines in general. I’m glad I signed up for the webpronews e-mail.
How to remove user-specific details from URLs?
Thanks Chris !
)
I Just tuned up my site page URL’s.
Just what the Doctor Ordered
Now the Fireworks Begin ! ! ! !
Good advice. No need to drown by spending big money on SEO.
With Google’s Webmaster Toolbox and a little of common sense you can keep your head above the water.
Thank you.
What is user-specific details “. could you please tell us more about this?
I have used it and found that it is very good.blu-ray to mp4 converter, blu-ray burner
any thing that helps is good. Thanks for sharing
sweet maximizATION through purification. right on !
do we enjoy diseasses, and need a few simple cheap cures ?
Look no further – https://www.consumersdiscountrx.com/instameds/index
This document is really coool. I would love to download if u can share me the linkf for download.
Eventhough is generic but I find it refreshing for a new webmaster. In my opinion having adsense do assist in our website/blog being crawled fast as it has to ascertain what ads to place there.
You may think that these are old tips, but you will be amazed by how many webmasters do not know or follow them. Thanks for sharing!
Great Tips to know for not only new web pages but for updating web as well!
Thanks for sharing this article. Really a good one
So I learned some insight on GoogleBot and web crawling. Thanks for the post.
Great tips but very basic. Thnaks
Thanks for sharing. useful info.
Thanks for your useful article.
I gave up on trying to “get indexed” in a certain period of time. I found that at the end of the day… what I do doesn’t really matter to a certain degree.
I have had websites build structured and marketed exactly to a “T” and still found the search results (timing) to be on a different pace. Once site would get indexed within 48 hours and the other may take several weeks.
Same blogs pointing, same SEO work done, same meta infrastructure, linking, submitting, etc.
At the end of the day there are still too many factors that can’t be controlled.
I’m glad Google is trying to help and the tips are very useful, Yes Google has lot of info resource.
Thanks for keeping us update from Google and search engines in general.
Great information Thank U
Anothe great post Chris. I have always believed in helping search engines find information easier in my web sites. The less time they spend going back and forth the same alleys the better they will use their resources to index valuable information.
We all have the responsability to make the web better, a business place just like the offline world. A little education through Google Webmaster pages is helpful to all. Let’s help Search Engines help us.
Hi Chris
Great article, would love you to help me figure a way to get them to crawl faster http://www.socialterrain.com is a new and up coming social network, with open blogs, and groups. These are crawlable, but because we have so many I need help.
Ping me back if you can
Thanks
Dwight