They do get a little post-Thanksgiving feast of goodies thanks to a new update to the free product:

The new version of IBM OmniFind Yahoo! Edition offers the ability to easily separate content into different searchable document collections, an improved administration console that simplifies set-up and management of the search tool, and enhanced search support based on the latest open source Lucene indexing library. Other enhancements improve the performance, indexing and custom search field capabilities, as well as support for additional browsers.

The collections feature should be of interest to those who have categorized documents by certain characteristics. Assigning them to a collection allows for targeted searches within it, for faster results.

IBM also said additional custom fields can be defined through the use of collections. Field values mapped from HTML meta tags, document metadata, or data pushed through the OmniFind push API allows searchers to perform queries specifically including metadata they choose.

Installation on Windows platforms should go more smoothly as the product may be put in place as a Windows service. IBM also said they have cleaned up search accuracy by supporting better near-duplicate detection and removal from OmniFind search results.

follow me on Twitter

Jonathan Kauffman, General Manager for Enterprise Search with Microsoft, wrote at the product’s blog of its capabilities. He cited several features like relevancy tuning and security-trimmed search results as being part of the product.

It will also support the OpenSearch standard, for federating searches across products. Kauffman acknowledged federation support from EMC, Cognos, HP, Business Objects, SAS, and OpenText. Connectors for those products should arrive with the full release of Search Server Express 2008 early next year.

Though Kauffman claimed corporate passion about giving away useful tools as a motivator for making Search Server Express 2008 available for free, there’s probably a couple of other reasons that apply. Competing with the IBM/Yahoo product would be a classic Microsoft tactic.

Search Server Express 2008 also serves as a gateway product to more advanced, for-pay solutions, just as IBM Omnifind Yahoo Edition does for IBM. In Microsoft’s case, it leads to Search Server and then to Office SharePoint Server, the latter product likely ensuring plenty of Microsoft products in the enterprise accompanying it.

Delivering a capable enterprise search product for free also gives Microsoft an edge against its hated rival, Google. The dominant search company sells its Google Search Appliance in a couple of form factors for business, as well as offering the free Google Desktop for Enterprise for local PC searches.

Tag:

We like the utility of Google Desktop, but have concerns about the product’s always plugged-in nature. Despite Google’s attestations of privacy, we think other people, especially in an enterprise setting, want the utility of strong email search without the worry of information somehow ending up on a remote server not controlled by the enterprise.

The modest announcement by IBM alphaWorks of their OmniFind Personal Email Search conceals a strong product being released. They are touting it as a semantic search system for email, with several key features.

IBM said this tool matches simple keyword searches against pre-defined concepts, like addresses and phone numbers, and the relationships those concepts have. If the search needs more information from the searcher, it will interact with them to determine intent.

Personal Email Search also supports tagging through an included utility. This allows the user to create new concepts that serves to customize the search. IBM also said these tags may be shared, without compromising the privacy of one’s email.

Before I give you my opinion, remember that I work for a competing product, IBM’s OmniFind search engine. So take my views with a pillar of salt. But here goes, anyway.

For small sites with limited technical skills, it seems worthwhile. You can set up the colors of the interface to match your Web site with no programming experience, and you just need to know enough to drop a snippet of code in the right place on your Web page. There’s no server to set up, so even sites with shared hosting can play. There are also free choices, such as Rollyo, that do a good job, but with less control over the appearance.

If you have some technical skills, however, there may be better choices. Even those folks saddled with shared servers can use a free Yahoo! API to give themselves a hosted site search. It’s worth remembering the drawbacks of a hosted search service, however. You can’t ensure that all of your pages will be spidered and you can’t ensure they will be spidered quickly as they change.

If you follow the tips in our book and in the Skinflint’s Guide to Search Marketing, you can ensure that most, if not all of your pages are spidered into that search index. And you can use Sitemaps to give yourself some control over how frequently the spider visits. It’s not the same as having your own search engine, though. If you want more control, you probably need to go a step further.

If you have dedicated hosting or own your own server, you can consider IBM’s OmniFind Yahoo! Edition, a free search engine (plug, plug). You must be savvy enough to be able to administer a server, but if you can, you get a free search engine that you have total control over. And it tops out at 500,000 pages, not 5,000 (or $500 a year for up to 50,000 pages from Google).

Google’s new offering seems valuable for those with small sites and no technical skills. Site search keeps getting easier and cheaper for site owners—what’s your site search like?

Comments

Since then, IBM and Yahoo have added a few partners to sweeten the appeal of the lengthily-named IBM OmniFind Yahoo Edition. Some prominent tech names have signed on with the project.

Cognos does various business intelligence applications. The company integrated its GO! Search from Cognos 8 with OmniFind Yahoo to enable users to sift through both Cognos and unstructured document repositories.

The SEM crowd who work with Fortune Interactive and OmniFind Yahoo have a new add-on available. Called Feature Match, part of Fortune Interactive’s SEMLogic product, the add-on extends the featured links capability within OmniFind Yahoo.

“Not only will your featured links appear above the search results, but other results found by the OmniFind algorithm will be given more importance when they are found to be thematically similar to the featured links,” Fortune Interactive noted on their site.

The OmniFind Yahoo product may be downloaded at no charge. It is supported on Windows XP/2003, and RedHat and SuSE Linux Enterprise distributions.

The new release shows an international flavor, with 14 new languages available. The management interface and documentation have been translated, allowing more people globally to work with the Yahoo-branded Omnifind.

It should be noted that there are actually 16 languages in this update. There may not be a great deal of demand for Pig Latin or Klingon, however.

Brown said many of the top requests for updates from users were for the ability to further customize the user interface. Some Omnifind users wanted to embed it with other applications.

There were also requests to search metadata as well as keywords. Visitors can now construct focused searches that sift through such metadata as tags, authors, and subjects among others.

Getting this done in four months, with some 60 features added and nearly two dozen bugs fixed, is remarkably fast for IBM. Brown said IBM made a “conscious choice to be more responsive” with this Omnifind product, resulting in the swift development time.

That movie came out in 1992, just ahead of the World Wide Web, a few years ahead of decent search technology, and over a decade before IBM and Yahoo started giving away Omnifind to any site publisher who wished to implement it.

IBM Omnifind Yahoo Edition is advertised as an enterprise search solution, capable of supporting up to 500,000 documents. The companies tout it as an easy to install and manage solution, and by Phillips’ account, they succeeded.

In a phone chat with WebProNews, Anomalies.net founder Phillips described how the process did go as smoothly as advertised. UFOCrawler launched with Omnifind providing the infrastructure for searching such esoterica as UFO evidence, strange contrails in the skies, and the aforementioned cattle mutilation.

Phillips said UFOCrawler has about 140,000 URLs indexed, which took about a day to crawl to get the search ready to launch. They switched from the open source Nutch search engine, which is part of Apache’s Lucene project.

Lucene also powers OmniFind, but for Phillips the improvements really came in the administrative interface. Being an experienced Unix admin, he had no trouble navigating Nutch for configuration purposes. Other folks in the project needed better administrative tools, and Omnifind delivered them.

UFOCrawler has held up well under recent, heavy interest. It continued to function without problems even under the effects of a powerful Slashdotting, Phillips said, after the popular tech news site featured it in a post.

Mother would approve.

—

Local desktop searches and probing corporate databases will be a simpler task for IBM business customers. Reuters reported how IBM will collaborate with search engine leader Google on an initiative to combine their technologies.

“Getting these two products together makes sense for both of us,” David Girouard, general manager of Google’s enterprise business unit, said in the report. “If you want to have a good corporate search product, you have to have desktop search.”

Getting IBM’s imprimatur endorses Google to the Fortune 500 even beyond what Google has accomplished in seven years. Google’s search has become a de facto tool used worldwide, so much that the company name is now used to generally describe searching online.

IBM quashed any wild speculation by noting the arrangement with Google isn’t part of any “grand strategy” in the report. Instead, IBM appears to be embracing the web as a service even more so, and will reportedly make a bigger push into “information as a service.”

That evokes the kind of access to IBM that developers enjoy with Google, Yahoo, and even Microsoft’s MSN to an extent. Those three big Internet players have made various APIs available for outside developers to craft applications linking back to the likes of Google Maps, Yahoo Stores or MSN Search.

David Utter is a staff writer for WebProNews covering technology and business. Email him here.

However, because corporate documents do not contain a natural structure, meaning they are more unique and un-related, indexing the large amounts of business documents can be an arduous task. It’s this concept that drives Arthur Ciccolo, one of the chief developers of IBM’s Unstructured Information Management Architecture (UIMA) project.

While enterprise search may be considered a niche topic, many of the developments coming from UIMA, if applied to web-based search, could have incredible ramifications. However, this is not IBM’s goal. During a phone interview, Ciccolo stated in no uncertain terms that IBM’s goal for their search technology is the enterprise level, not web search.

To define UIMA and its function, IBM offers this:

Unstructured information represents the largest, most current and fastest growing source of information available to businesses and governments An Unstructured Information Management (UIM) application may be generally characterized as a software system that analyzes large volumes of unstructured information (text, audio, video, images, etc.) to discover, organize and deliver relevant knowledge to the client or application end-user.

In order to accomplish this task, Ciccolo and his team are putting their efforts developing different framework structures to perform text analysis, semantic comprehension, and natural language support. By doing so, IBM’s UIMA utilities can better perform the tasks of indexing and comprehending the different types of enterprise business documents.

To understand why enterprise search can be such a complicated excursion, you must first understand the different types of unstructured data that has to be indexed. With web-based documents, there is a much more narrow focus because document types are more limited (html, pdf, fla, etc.) and they usually contain links, which lends itself to easier indexing.

These off-page attributes provide the structure and makes web indexing less complicated than the unstructured environment of business documents. While web documents are more confined, enterprise documents run the gamut from word processing documents to video and sound files, and the off-page attributes that provide structure are absent.

The capabilities of UIMA, some of which are still in the developmental stages, attempt to address these concerns. For instance, the focal point of UIM is to focus on text analytics and the semantics contained within. By understanding the contents of the text being indexed, developing natural language search capabilities (“What is the formula for product X?”) is an attainable goal.

However, to understand the difficulties involved in developing a natural language search feature that works, consider this: the reason Microsoft will be releasing Longhorn without the search feature has to do with the developing a new file structure that supports natural search queries.

With UIMA, once a document is indexed, searching for it should be easier than it would if it was web indexed. Ciccolo indicated that once an item is indexed, their technology automatically generates editable meta data, which makes discovery much simpler. By integrating UIM technology with IBM’s Intranet search utility WebSphere Information Integrator OmniFind, they are able to provide an exciting era of enterprise-related search.

With regards to the current abilities of UIMA, the potential future developments are quite impressive, and if adopted, could cause huge ripples in the search technology status quo. While the list of possible developments is fairly long, there are two possible developments that could have long-reaching ramifications.

The first area of interest has to do with video search. Ciccolo and his team are developing video search methods that could revolutionize the whole concept. Normally, most video indexing is done by spidering the closed-caption text contained within the film. However, Ciccolo’s vision has to do with actually analyzing the picture to extract whatever relevant content is contained within. This data would then be indexed and have meta data generated (which would be editable using IBM’s service), making retrieval methods even better.

The other area of interest has to do with providing the ability to perform trans-lingual queries. What drives this development is the following concept:

- User A enters a query in English language
- UIMA translates query into target language
- UIMA then searches target language documents and,
- Returns search result in whatever language initiated the query. In this case, English

Arthur indicated that during the testing phases, he discovered search results were actually more relevant after the translation took place, meaning UIMA performs the translation after the query is entered. If something similar to this was adopted by search as a whole, it could and would alter the entire landscape of possibilities.

Other features and ideas of interest include the ability for administration members to write their own search algorithms, which can be implemented on top of existing framework. Speaking of admins, Ciccolo made sure to point out UIMA was developed with IT departments in mind. The technology is purposely made to be easy to install and to tweak. This makes adapting their search technology to fit your business much easier.

Another area that the team is focusing on is the medical industry. The ability to catalog and differentiate between the mountains of journals and documents over-running medical institutions is completely welcomed. Currently, IBM has an agreement with the Mayo Clinic to implement and test UIMA. If the medical field as a whole would adopt such technology, finding patient information, journals on particular illnesses, and pharmaceutical information would be much easier to accomplish. This in turn would undoubtedly improve the medical industry’s ability to treat and care for patients, as well as share information.

To understand the goal of the UIMA project, Ciccolo offers these thoughts, “IBM’s goal for UIMA is that it becomes widely accepted as a new class of middleware for analytics and that it enables the next generation of search: semantic search.”

For much more information about the project and other areas of IBM’s approach to enterprise search, please visit the following areas:

UIMA Homepage
The research proposal
About the authors
Information about OmniFind

While the subject can be tricky to navigate through, I would recommend reading the journals and documentations of Art Ciccolo and his team.

Chris Richardson is a search engine writer and editor for WebProNews. Visit WebProNews for the latest search news.