Guidelines For Managers Considering Open Source Enterprise Search

Good Information
Is Hard to Find:
Guidelines for Managers
Considering Open Source Enterprise Search
A Lucid Imagination White Paper

Abstract
Enterprise search helps employees, customers, and partners find the most relevant and
timely information, enabling them to make smart, efficient decisions about doing business
with your company. Open source has provided strong benefits in enterprise software such
as operating systems, databases, and middleware, now unleashes value in enterprise
search. Lucid Imagination brings market-leading expertise to open source enterprise
search, and can help any organization quickly design and optimize search solutions based
on Lucene and Solr.
Good Information Is Hard to Find: Considering Open Source for Enterprise Search
A Lucid Imagination White Paper • April 2009 Page 1
Table of Contents
Abstract ..................................................................................................................................................................... 1
Introduction and Overview ............................................................................................................................... 3
The Advantages of Open Source ...................................................................................................................... 5
Lower Costs ......................................................................................................................................................... 5
Pay at the Point of Value................................................................................................................................. 6
Transparent Development ............................................................................................................................ 6
Re-tool the employees, retire the software............................................................................................. 7
Lower Overall Risk ........................................................................................................................................... 7
About Lucid Imagination.................................................................................................................................... 8
Engagement Scenarios ..................................................................................................................................... 10
Considering Alternatives to Legacy Packaged Search Applications ........................................... 10
Building on In-house Lucene/Solr Expertise ...................................................................................... 12
Next Steps ............................................................................................................................................................. 13
Appendix: About Apache Lucene and Solr ............................................................................................... 14
Introduction and Overview
Raising the collective intelligence of company employees can make them smarter and more
efficient—but how do you enable them to keep up with the vast, ever-changing amount of
data your organization produces? Many operations seem to be better at creating data than
using it to operate more productively. Using search tools designed for the Web can make it
difficult to find relevant, timely corporate information, mostly because corporate data is
not much like Web data:
• Corporate data can be stored in a variety of different and unstructured formats,
including documents and database records.
• A document’s popularity is not necessarily what makes it useful to a specific search.
• Information may require controlled access, yet still be discoverable to those users
with the appropriate permissions.
Two state-of-the-art, open source search technologies—Lucene and Solr—are available for
free from the Apache Software Foundation. Lucene is a powerful search engine and library;
Solr provides a platform built on top of Lucene that makes it easy to build Lucene-based
applications.1 Rich, flexible text query tools and sophisticated ranking capabilities of
Lucene/Solr enable users to quickly find the most useful documents or records.
Either of these full-featured technologies delivers excellent performance, relevancy
ranking, and scalability. They are used today by thousands of organizations, powering
substantial and diverse search applications for AOL, CNET, Comcast Interactive Media, IBM,
Netflix, LinkedIn, MySpace, and many others. For these companies, Lucene/Solr solutions
regularly index and search hundreds of millions of documents with subsecond response
time, all without incurring any licensing fees.
These solutions excel at quickly and effectively searching large volumes of unstructured
text—documents or other records containing freeform text—and returning results based
1
Most organizations use Solr today as their search development platform. Because Lucene serves as the core of
Solr’s search capabilities, this paper refers to them as Lucene/Solr. For more information about these technologies,
see the Appendix.
on how well they match the user’s query. At most companies, this means digesting and
searching through dozens of different file formats—including documents, spreadsheets,
presentations, e-mail, and records stored in databases, to name just a few—and delivering
relevant results to authorized users. Incremental update capabilities mean that
Lucene/Solr searches can track document collections easily as they grow and change,
finding information nearly as fast as it is created.
Solr can speedily facet, or categorize, data and search results based on specific field values.
An excellent example of this function is Zappos.com, the popular shoe e-tailer, where users
can quickly refine searches based on product criteria such as price or features.
For most application development teams, building a search application is not an everyday
project. By definition, enterprise search technology processes unstructured data, which can
change frequently. Expert guidance on architectural considerations, such as index
optimization, result relevance, deployment configuration, and retrieval performance can
make a tremendous difference in deploying a successful solution. By taking advantage of
expert, experienced personnel to assist with application design, development, and
deployment, organizations can leverage the full benefit of Lucene/Solr search technologies
without the cost of licensing proprietary software.
For these reasons, Lucid Imagination provides commercial-grade support, training, and
professional consulting services that are essential to designing and installing successful
enterprise applications.
This paper is intended for business decision makers who are considering options for
powerful, flexible enterprise search solutions. It provides guidelines for understanding:
• Advantages of open source software, including ways it can lower costs and risks,
• Why Lucid Imagination’s service and support is a key ingredient in achieving successful
Lucene/Solr solutions,
• Engagement scenarios—the types of situations where Lucid Imagination can help, and
• The capabilities of Lucene/Solr, which are provided in an appendix.
The Advantages of Open Source
Open Source has changed the IT landscape. Gartner says 85 percent of polled companies
are already using open source software, calling the use of open source software
“pervasive.”2 Most organizations are now familiar with free and open source products such
as Linux, MySQL, Apache, and SugarCRM, because of the many benefits, including:
• Lower costs
• Pay at the point of value
• Transparent development
• Control and flexibility – investing in people instead of licenses
• Lower overall risk
With Lucene/Solr’s broad, successful adoption across markets and deployments, these
advantages are now available for enterprise search applications. Let’s take a closer look at
how open source pays off.
Lower Costs
While proprietary software vendors must try to recover their development costs, this is not
the case with open source software, because it does not have capital costs associated with
source code IP. The cost of talent is less, too. Community development, adherence to
standards, and lower barriers to adoption all help increase the number of developers who
become proficient in the use of a product or technology. Together, these factors combine to
reduce upward pricing pressure.
The high license fees associated with proprietary and closed source development can
discourage developers and customers from adopting a product or technology. In contrast,
open source communities help lower costs by encouraging participation and allowing
anyone to download the source code and try it out. Most open source communities release
2
http://www.theregister.co.uk/2008/11/18/gartner_open_source/
updated binaries on a periodic basis, so users can easily try the software on their own
timetables.
Many commercial solutions combine proprietary software with service and support, and
customers may believe that buying a software license is sufficient to get a search
application up and running. In most cases, however, the technology’s purchase price makes
up less than half of the implementation cost, with the balance going to services. Both open
source and proprietary software usually require a significant amount of customization,
which means some service and support costs are inevitable.
Pay at the Point of Value

Open source project code is freely available for any use. If a company can become proficient
with the code, it can make productive use the code at any phase from evaluation to
production. Only in those areas where an open source customer sees value—for support
and integration services, or for additional functionality or expertise—does money need to
be spent. There are no restrictions on when open source software can be used.
In contrast, proprietary products typically must be purchased before they can be used, or
in some cases, even evaluated. Some vendors offer evaluation or trial versions, but these
often have reduced functionality or restrictive licenses. Because the software must be
purchased before the customer can see any value from the product, return on investment is
delayed.
Transparent Development
Community-developed software enables everyone to see what is being built and which
features are included as early as possible. Developers and customers do not need to wait
for a vendor to publish a roadmap or product launch to know what is being readied for
release. As a result, prospective users can make better, faster, and more informed decisions
relating to their software infrastructure.
Compare this to proprietary software, where customers have little if any insight into
upcoming products until very late in the product life cycle. This is typically no sooner than
the software’s beta release, when it is too late to provide input on features and
functionality. This delays assessment and adoption of innovations.
Re-tool the employees, retire the software
In this tough economic climate, managers who own budgets need to review every expense
with a critical eye. Many software applications that made sense a few years back may have
out-lived their intended fit to business needs.
Any application development effort generates significant learning. The work of
development imbues it requires the expertise of in-house developers with deep knowledge
and understanding of the company, its IT infrastructure, culture, and usage requirements.
Given that software applications must keep up with an organization’s changing goals and
requirements as the needs of its market and constituents evolve, the expertise which the
technical staff develops becomes is a vital competitive asset.
This is key corollary benefit of the open source model: by retiring old software packages
and investing in staff expertise, companies combine innovative technology with their most
valuable asset – their people, establishing vital competitive advantage.
Companies who leverage savings from not purchasing software licenses to build
development talent in-house reduce the cost of addressing inevitable change. What’s more,
increasing a technical team’s ability to translate company business objectives into
technology solutions increases the likelihood that the software they build will continue to
fit that inevitable change. This is particularly true for an enterprise search solution. What’s
more, compared to closed source implementations, in-house developers can work with
open source code and supplement additional functions or expertise by relying on the
community and marketplace of readily available resources – again capturing unique
competitive advantage.
Supplementing open source development with training, consulting, and reliable support
from established industry experts reinforces a company’s competitive advantage – with the
control and flexibility needed to survive and thrive.
Lower Overall Risk

Vendors use proprietary interfaces and components to lock in customers. However, the
source code for open source software is freely available and widely supported by the
community, based on standardized, free public interfaces. If a commercial vendor goes out
of business (or is purchased by another), or tries to increase fees for a commercial product,
open source vendors may be able to step in to meet the needs of customers at market-
competitive prices.
Open source software can reduce security and operational risks, too. Widely used open
source software is essentially under constant peer review. Technical or security issues,
once exposed in the community, are readily addressed, resulting in a safer and more
reliable product.
About Lucid Imagination

The benefits of open source have unlocked tremendous value in many software categories:
Red Hat’s Enterprise Linux in operating systems, MySQL in database software, Sugar in
CRM software—all have benefited from matching the efficiencies of open source with deep,
robust commercial resources to ensure successful applications. Today, Lucid Imagination’s
capabilities and expertise brings that same approach to unlocking enterprise search with
Lucene and Solr.
Lucid Imagination’s mission is to enable customers to achieve business objectives for
optimal search performance and accuracy, with lower total cost of ownership and faster
time to market. The company’s founding team consists of many key contributors and
committers to the Lucene/Solr project, as well as other experts in enterprise search
application development. Our skills, acquired across hundreds of deployments, including
best practices and technical know-how, can enhance and optimize any phase of an open
source search implementation.
Lucid Imagination’s team has a deep understanding of indexing, which is the foundation of
any search solution; it captures all the content and location of searched documents for
quick lookup, much as a book index does. We have broad experience indexing:
• Documents of widely varying sizes and formats within a very large collection,
• Documents with diverse metadata requirements, and
• Multilingual documents.
The team is also skilled at applying business rules such as boosting documents and fields,
indexing dates, or other attributes of terms and data. Lucid Imagination has developed best
practices for indexing and metadata management, and can help establish and refine
policies to meet business and technical search requirements, such as:
• How and when to add documents to an index,
• Removing documents from an index,
• Results relevancy and document/data findability
• Undeleting documents, and
• Batch and real-time updates.
The Lucid Imagination team has extensive experience with large-scale search applications,
including engagements with:
• Large collections—more than one billion documents,
• High query volumes and large user populations,
• High document growth rates,
• Distributed indexing and searching,
• Replication and high availability, and
• Cloud environments.
In addition to fine-tuning search technology machinery, the Lucid Imagination team has
significant expertise in natural language processing, which optimizes the interaction of
compute resources with human-created content. Key considerations include:
• Developing structured methods for characterizing how well a set of results meets user
needs,
• Establishing a tradeoff between overall net gain in the quality of results across the whole
application, versus a single improvement for one query or user, and
• Improving the ability to find accurate answers by leveraging a balanced mix of content
analysis and query interpretation algorithms.
The breadth of expertise offered by Lucid is available in a variety of forms suited to a range
of different business needs and deployment requirements. This enables customers to
create even more powerful and successful search applications.
Engagement Scenarios
Virtually every company and organization uses some form of enterprise search, to help
customers, employees, and partners find the information they need. Many companies use
packaged commercial software applications; but, over time, their requirements evolve
beyond the original platform’s limitations. Also, licensing or customization costs may grow
too high, or the number and type of documents may expand beyond the original design’s
capacity. As companies evaluate the ongoing fit of their current search applications to an
ever changing market and organizational landscape, they naturally ask “Is there a faster,
cheaper, more effective way to do this?”
Today, thousands of companies and organizations—each with unique search and retrieval
requirements—answered this question with Lucene/Solr. The essential value of Lucid
Imagination and open source Lucene/Solr technology is that it provides commercial
support that adapts to specific requirements. Whether a company is evaluating
Lucene/Solr for a new implementation, considering replacement of a commercial search
product, or enhancing an existing Lucene/Solr implementation, Lucid Imagination offers
skills and resources to help at every phase of the project life cycle.
Considering Alternatives to Legacy Packaged Search Applications

Change happens quickly, but taking advantage of new opportunities can be limited by
existing applications and traditional ways of doing things. Organizations with legacy search
applications often realize that they are paying too much to align packaged enterprise
search applications with evolving business requirements. In other cases, they discover it is
too difficult to integrate existing software with new services, or it takes too long to meet
new corporate goals. With the power of Lucene/Solr, Lucid Imagination supplies the
expertise organizations need to produce successful search solution efforts, more quickly
and less expensively—now and going forward—than other solutions.
• Consulting services are highly customized and able to engage quickly to shorten
cycles and ramp times, minimize errors and design pitfalls, and improve production
results. Lucid Imagination’s consulting team consists of senior search technologists
who are intimately familiar with Lucene/Solr technologies and have extensive
experience in field-tested search solutions for diverse deployment scenarios.
Open source software is ideally suited to low-cost prototyping, because it can
reduce time to deployment and refine the user experience. For customers striving to
integrate a highly diverse base of data and documents, Lucid Imagination offers
prototyping services to assist with the process.
• Technical training can bring everyone in the IT department up to speed on best
practices and the elements of good search design—establishing a solid base of skills
before coding begins. This can greatly reduce downstream problems and reduce
overall costs. Lucid Imagination works with in-house application and system
administration teams to provide the knowledge transfer, guidance, training, and
support required to implement an enterprise search solution that fits the
organization’s specific needs.
• When dependable, predictable support is required to accompany an organization’s
efforts on a regular basis over time, Lucid Imagination’s support subscriptions
provide reliable access to domain experts during the entire application life cycle
process.
Technical Support features the latest tested versions and timely,
predictable support turnaround times.
Advanced Development Support provides expert architectural design,
development, and testing guidance for building search applications using
Lucene and Solr.
Advanced Production Support provides expert advice on configuration,
performance tuning, and optimization for applications deployed to a
production operation environment with live users and service-level
attainment regimes.
Search Health Check, included with Advanced Support, is a comprehensive
set of services that ensures applications are designed to meet recommended
best practices for search configuration, optimization, and effectiveness.
Custom Support packages are also available for unique situations.
• Lucid Imagination’s free 30-Day Get Started Program is available with downloads of
Lucidworks, our certified distributions of Lucene and Solr. The Get Started Program
complements Lucidworks with added guidance for questions on first-time
installation, configuration, and basic usage, as well as evaluation of Lucene/Solr and
included utilities. LucidWorks for Solr is the logical starting point for most
developers building search applications with Lucene/Solr technology for websites,
products, or internal organizational use, because it bundles the most recent and
stable Apache/Solr capabilities, along with other tools and utilities.
Building on In-house Lucene/Solr Expertise

Many organizations with in-house Lucene/Solr expertise have achieved considerable
sophistication in their deployments. Still, they may reach a point where it is difficult to
move the architecture or implementation past a particular design, deployment, or
optimization constraint. There can be many reasons for this, such as limitations on staff
expertise, design, or architecture. Configurations and policies may not have kept pace with
current best practices. A dependent part of the IT environment may have changed—
anything from upgraded complementary applications to new middleware, or expanded
data volume and variety.
For organizations that are ready to gain the required knowledge to move ahead, address
the current situation, and make sure that a deployment stays at peak performance, Lucid
Imagination recommends an in-depth engagement. Typically in a consultative format,
engagement begins with an in-depth assessment and review followed by best practices
design recommendations, and ends with a strategy proposal for achieving long-term,
sustainable innovation for search solutions.
Another key area where Lucid Imagination stands ready to help is in optimizing
performance—both in application response time and its utilization of hardware/software
resources. Lucid Imagination experts work with in-house teams to diagnose and improve
search application efficiencies.
As mentioned earlier, a significant benefit of open source software is its ability to provide
fast, low-cost prototyping as a means to reduce time to deployment and refine the user
experience. For customers that seek to integrate highly diverse bases of data and
documents, or accelerate evaluations of open source search solutions, Lucid Imagination
offers prototyping services.
While community support has always been a significant benefit of open source projects,
tough issues may not always be answered in timely fashion or with the discretion
necessary to prevent exposure of confidential organizational knowledge. That’s when Lucid
Imagination’s expert teams can help.
Some companies are already skilled in open source technologies in general and
Lucene/Solr in particular. For these, Lucid Imagination offers Technical Support and
Advanced Support. Technical Support can provide answers within defined response times
for users encountering problems with Lucene/Solr projects or production
implementations.
Different levels of support address most situations. For example, an e-commerce startup
may find that community forums provide suitable answers, but not always as quickly as
needed. Basic Technical Support provides Web-based and e-mail support at competitive
rates for customers that do not require same-day response or direct telephone support.
Lucid Imagination also offers various levels of Technical Support for larger or mission-
critical installations, including fast turnaround, diagnosis, and bug fixes. Finally, Enterprise
Technical Support includes Search Health Checks by Lucid Imagination domain experts to
help ensure optimal runtime effectiveness.
Next Steps
For more information on how Lucid Imagination can help employees, customers, and
partners find the information they need, please visit http://www.lucidimagination.com to
access blog posts, articles, and reviews of dozens of successful implementations. Please e-
mail specific questions to:
Support and Service: support@lucidimagination.com
Sales and Commercial: sales@lucidimagination.com
Consulting: consulting@lucidimagination.com
Or call: 1.650.353.4057
Appendix: About Apache Lucene and Solr
Apache Lucene/Solr offers an attractive alternative to proprietary search and discovery
software vendors. Lucene is a Java technology-based search library and Solr is a platform
built atop Lucene that provides application builders with a ready-to-use search platform.
Both Lucene and Solr are free and open source. They are available under the Apache
Software License, which allows users to modify or embed the technology as they see fit, and
to keep, sell, and/or redistribute any resulting product.
Solr is the logical starting point for most developers building search applications with
Lucene/Solr technology for websites, products, or internal organizational use. Most users
building Lucene-based search applications will find it is quicker to start with Solr, since it
contains many of the capabilities needed to turn a core search capability into a full-fledged
search application.
The full-featured core Lucene search engine library offers:
• Speed: Sub-second performance for most queries.

• Relevancy ranking: Out-of-the-box rankings are as good or better than the best
commercial competitors.
• Complete query capabilities: Keyword, Boolean and +/- queries, proximity operators,
wildcards, fielded searching, term/field/document weights, find-similar, spell checking,
multilingual search, and much more.
• Full results processing: Sorting by relevancy, date or any field, dynamic summaries, hit
highlighting, and more.
• Portability: Runs on any platform supporting Java and indexes are portable across
platforms. Indexes built on Linux can be copied to a Microsoft Windows machine where
they can be searched. Lucene and Solr are written entirely in Java; .NET and other
versions are also available.
• Scalability: There are production applications in the hundreds of millions that can search
billions of documents/records.
• Low-overhead indexes and rapid incremental indexing.
The Solr platform adds the following capabilities:
• Web services: Solr places Lucene over HTTP, allowing programs written in any
language to invoke Lucene.
• Faceting: The dynamic clustering of items or search results into categories enables users
to drill into search results (or even skip searching entirely) by any value in any field, as
seen on popular e-commerce sites such as Amazon or Zappos.
• XML-based schema: Manages indexed fields and their characteristics.
• Admin tools: Configuration, data loading, index replication, statistics, logging and cache
management, and more.
• Scalable: Distributed architecture enables large-scale distributed search.
• Configurable: Fixed/paid result list placement.

Guidelines For Managers Considering Open Source Enterprise Search

Загружено:

Сведения о документе

Оригинальное название

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

Guidelines For Managers Considering Open Source Enterprise Search

Загружено:

Авторское право:

Доступные форматы

Good Information

A Lucid Imagination White Paper

Pay at the Point of Value

Lower Overall Risk

About Lucid Imagination

Considering Alternatives to Legacy Packaged Search Applications

Building on In-house Lucene/Solr Expertise

The full-featured core Lucene search engine library offers:

• Speed: Sub-second performance for most queries.

Вам также может понравиться