WebProNews

Three or so years ago, I published a study in "Current Research in Social Psychology" (http://www.uiowa.edu/~grpproc/crisp/crisp11_12.pdf) that used co-occurrence Google counts for concepts to demonstrate semantic structure in the web. Most of the material I looked at pretty clearly supported the idea that there is such structure. I did, however, run into occasional disquieting results. The problem is simply illustrated by some recently obtained follow-up data.

I looked at co-occurrences among three small sets of concepts using both Google and Yahoo: nine psychologist names (from the original study), nine animal names, and nine countries. The psychologists and the animals demonstrated clear structure using results from both search engines. Countries worked well with Yahoo, but Google gave me a very strange and essentially unusable set of results. For example, India yielded a Google count of 511,000,000, France yielded a count of 177,000,000. However, France and India submitted together had a count of 1,900,000,000. Clearly a logically impossible result. There were several similar results in the same set. I have run into anomalies like this before with Google, but they were not great enough to render the data unusable. I have not run into a similar problem with Yahoo yet. Clearly something goes radically wrong with Google on occasion (perhaps with certain classes of concepts). Note, I understand that these numbers are some kind of estimates and not exact, but here we are looking at results that appear to go beyond reasonable estimation errors.

Can you help me out with some references? I have, so far, not been able to find much that is more relevant or recent than this (now rather old) comment of yours.

Thanks,

Jack B. Arnold
Professor of Psychology, Retired
Saint Mary's College of California