In-Depth

Searching for an Answer in the Enterprise

As the clamor from IT shops for better internal search grows louder, competition among vendors both large and small gets hotter.

• By Paul Korzeniowski
• 02/01/2008

Trent Parkhill, VP and director of IT services at the Boston-based Haley & Aldrich Inc., initiated the hunt for a new enterprise search system.

Trent Parkhill, vice president and director of IT services at Haley & Aldrich Inc., a Boston-based commercial construction consulting company, was asked a question commonly heard in many IT departments: "Why isn't our internal search as simple to use as when we look for something on the Web?"

In companies, large and small, a gaping disconnect is plainly evident between internal and external searches. While a Web user can type in a few words and regularly come back with desired information, that's not the case with enterprise searches. In enterprise search, users often enter data several times and end up so frustrated they simply stop using their companies' enterprise search systems.

There are many reasons why enterprise search systems don't work well. The first one stems from the nature of enterprise information: Enterprise search systems need to be more secure than Web searches. Also, vendors have built products that store data in Enterprise Resource Planning (ERP) systems, e-mail messages, text documents and spreadsheets. To help users find desired information, companies need a tool that can examine all of those information sources-a task that no product was originally designed to do. Compounding the problem, enterprise search usually has a much narrower focus than Web search. Additionally, products that have been built to improve enterprise search have often been expensive and difficult to deploy and maintain.

Corporations have clamored for improvements in search technologies because their employees spend valuable time fruitlessly searching for information, which results in lost revenue. In response, a regiment of vendors has emerged to try and address the problem. Autonomy Corp., Coveo Solutions Inc., Dieselpoint Inc., Endeca Technologies Inc., Exalead SA, Fast Search & Transfer (FAST), Groxis Inc., ISYS Search Software, Northern Light Group LLC, SearchInform Technologies, SearchBlox Software Inc., Siderean Software Inc., Thunderstone Software LLC and Vivisimo Inc. are just a sampling of the suppliers delivering enterprise search products.

But recently these small niche vendors have been joined by industry Goliaths Google Inc. and Microsoft. While all of these suppliers have worked diligently to make internal search as robust as external search, a silver bullet has yet to emerge. However, enterprise search has been steadily improving as these vendors take several different routes to address its traditional limitations.

Web Search vs. Enterprise Search
To deliver better enterprise search systems, software suppliers need to build products that index corporate data stored in a variety of places and then deliver that data in a secure manner to end users. Security is one function missing in many of the Web search products.

"The basic idea of a good Web search engine is to scour and index all of the information on the public Web," notes Craig VerColen, manager of public relations at Endeca. "They were never designed to interpret that information and how it maps to security models and user permissions."

Enterprise search systems need to make distinctions between delivering sensitive data, such as an employee's annual pay or Social Security number, from non-sensitive data, such as the dates when paychecks will be cut.

Another reason Internet-based search is much simpler than enterprise search is the format of the data examined. With Internet searches, information is primarily restricted to HTML Web pages. In enterprise searches, data is stored in many formats, such as word processing documents, database management systems and image processing systems. Internet search systems can also easily pinpoint information sources, which usually are individual or company Web servers. Within an enterprise, information may reside on central servers, department systems or employee machines, which can be PCs, notebooks or handheld devices.

"The processes of how users enter information for Web and enterprise searches are similar, but the results enterprise users desire are much more specific," notes Raul Valdes-Perez, CEO at Vivisimo.

"The processes of how users enter information for Web and enterprise searches are similar, but the results enterprise users desire are much more specific." Raul Valdes-Perez, CEO, Vivisimo Inc.

With Internet searches, users often have broad search goals-many times they don't know exactly what they're looking for-and are really only looking for places where they can find needed information. After they type in a word, such as "notebook," users frequently are satisfied with being brought to a comparison Web site where information about several notebooks is listed. That's not usually the case when employees search for corporate data. If they type in a key phrase, such as "Joseph Smith's address," they expect a specific piece of information to appear and are disappointed if that doesn't happen.

Consequently, companies often spend a lot of time and effort identifying where information is located and then making it available to enterprise search engines. Even after doing that, the results can be disappointing, something that Haley & Aldrich found.

A Search Story
"Our search system was returning a lot of junk," admits Haley & Aldrich's Parkhill, whose company formerly relied on search functions found in Microsoft's SQL Server. "Our employees became so frustrated that they stopped using it."

The lack of strong search features created a drag on corporate productivity. A consulting firm, the company generated many documents, which employees frequently needed to examine. Also, expertise such as prior work experiences with a potential customer was stored in the company somewhere, but often employees could not locate it.

So, in the spring of 2006, Haley & Aldrich looked for a new enterprise search system. After combing various Web sites and talking with a number of vendors, the selection boiled down to products from Coveo and Endeca. The former was chosen because its pricing model was more in line with small and midsize business; pricing for enterprise search systems tends to be high because the deployment process is complex.

The first step to installing these products is completing an information inventory. Then a company has to develop an "information repository," a central place where different data sources can be referenced; connect the repository data to an enterprise search engine system; and let users access the engine via a company's intranet. To complete these tasks, companies pay from $3 to $5 in systems integration charges for every $1 they pay in enterprise search software-licensing fees. This is a requirement that drives up costs dramatically, according to Matt Brown, a principal analyst at Forrester Research Inc.

Haley & Aldrich had 3 million documents that it needed to index, and expected that number to grow to 20 million in a few years, something Coveo Enterprise Search could handle. The system was also open, so Haley & Aldrich could tinker with it to include company-specific search items, such as searching for any prior experience with a potential customer. After completing its evaluation, Haley & Aldrich installed Coveo Enterprise Search in the summer of 2006.

Initially, indexing data was a cumbersome process, but Coveo enhanced the system so it became much simpler. Haley & Aldrich found significant differences between the Coveo system and Endeca products because suppliers have taken a number of different approaches to tackling enterprise search problems.

Search Leaders
Dallas-based Autonomy has been the traditional market leader in search.

"Autonomy has done well with large enterprises and complex searches," notes Whit Andrews, a research vice president at Gartner Inc. To make its way further down the ladder, Autonomy purchased the Ultraseek search engine primarily designed for the consumer market. It started offering companies a one-year free trial for the product, which sells for about $25,000. FAST has been using original equipment manufacturing (OEM) to deliver its search engine to software companies such as Business Objects SA, Cognos Inc. and WebTrends Inc., which have in turn incorporated it into their products. Endeca has tried to make it easy for customers to work with structured and unstructured data by building a robust rules-based engine capable of indexing many different data types. Vivisimo's Velocity is able to correlate social networking information with enterprise search.

A few of the products have gained a foothold by focusing more on helping external users -- rather than internal ones -- find information. Certain businesses find they have a Web presence but offer potential customers and partners only a limited ability to search their sites. MEDmarketplace.com Inc., which has been in business since 2002, fits into this category. The company's business centers on selling hard-to-find new and used medical products, including wheelchairs and oxygen tank holders, to businesses and consumers.

"As our business grew, we needed a way to help potential customers find desired items," notes John Michael Baratta, CTO at MEDmarketplace.com. Baratta's company, like Haley & Aldrich, formerly relied on Microsoft SQL Server queries to help customers find needed data.

At the start of 2006, the medical product marketer examined enterprise search products from vendors like Coveo, Dieselpoint, Google and ISYS Search Software. At the end of 2006, the company chose the ISYS product because it integrated well with MEDmarketplace.com's Microsoft SQL Server database-management system.

Google and Microsoft Square Off
One market development that may surprise some is that Google is a relatively new and small player in the market. The company, which has been synonymous with Web search, made a major push into enterprise search in June 2004. At the time, the company revamped its Google Search Appliance line and focused on low pricing and simplicity.

Competitors offer products that sell for $50,000 to $1 million. Pricing for the Google Search Appliance starts at $30,000; the Google Mini, a scaled-down search system geared toward departments or small companies, sells for $2,995; and Google Desktop Search for Enterprise, designed for searching individual files, is free. The company is trying to maintain the ease-of-use functions found within its Web search system. Corporations can complete product installation in a few hours compared to the days-or even weeks-typically associated with traditional enterprise search systems.

Microsoft has been right on Google's heels. It, too, has made some progress in penetrating the market, but in November 2007, the company announced Search Server 2008 Express, which will be made available as a free download. Redmond's search product was formerly part of its SharePoint collaboration system.

"Microsoft's decision to sell its Search Server separately was a bit surprising, but illustrates how important search is becoming in the enterprise," notes Gartner's Andrews.

In addition, Microsoft has moved to the forefront in the promotion of new federated search capabilities based on Creative Commons' OpenSearch standard. Several companies, including Open Text Corp., Business Objects, Cognos and EMC Corp., are developing federated search connectors to enable Microsoft's enterprise search customers to connect to their information systems.

With so many vendors trying to expand on the limitations of enterprise search in so many different ways, improvements loom on the horizon. Consequently, folks like Haley & Aldrich's Parkhill will soon be asked less about enterprise search and more about items such as, "Why does software cost so much?"