MarkLogic Server Overview:

Infrastructure Software for Building

Information Applications
Table of Contents
1 | Introduction
2 | XML is the Future
2 | The Challenges of Information
5 | MarkLogic Server: Addressing the Challenges of Information
6 | An Enterprise-Ready Database Management System for XML
8 | Accessing Information: Search and Query
13 | A Complete Platform for Building Information Applications
15| Conclusion

Abstract
Rapidly changing conditions are forcing organizations to re-think how they use information
to meet their objectives. Whether battling in the market place or on the battlefield, the
need for flexibility and agility with information has never been greater. Organizations are
looking to integrate and enrich information to create additional value for users. User ex-
pectations are changing too, as they demand Web 2.0 and Enterprise 2.0 style applications
that provide modern search capabilities, as well as an ability to interact with information
through tagging and user generated comments. And various distribution channels present
new challenges for information providers in exposing their information through rich user in-
terfaces or through syndicated services like RSS and Atom feeds, allowing users to explore
and access information in their own context.

Choosing the right technology at the core of their application architecture is critical for
any organization to provide them with the agility they need to meet these goals and rapidly
respond to unforeseen changes. XML servers such as MarkLogic Server provide that agility
by providing a single unified platform for storing, manipulating and delivering XML and
building innovative information applications.

This paper provides a technical overview of MarkLogic Server, the industry’s leading XML
server, and also discusses some of the challenges facing organizations today for storing,
repurposing, and dynamically delivering information.
MarkLogic Server Overview:
Infrastructure Software for Building Information
Applications

Introduction
The amount of information produced by private and public organizations contin-
ues to increase. For government in particular, more publicly available, private,
and general information is being created and processed than anyone predicted.
Along with the information explosion, XML is gaining momentum as the paradigm
for storing, managing, and delivering information. The increasing adoption of
XML is driven by a combination of factors including:

• Organizational desire to maximize the value of existing information

• Changing user expectations

• Technical factors such as the need to reuse information at any level of detail

Organizational objectives and rapidly changing environmental factors increas-

ingly require improvements in how information is reused and delivered. Many
organizations struggle to reuse information across different deliverables and
channels. For example, subject matter experts easily create documents with
common authoring tools, but lack the ability to use them again in books, journals,
articles, on-line in customer facing applications, training materials, doctrine, and
technical manuals. Organizations want to create information once and reuse it in
as many channels as possible to add additional value.

On the user side, there are new expectations when it comes to accessing, inter-
acting with, and collaborating around information. These expectations impact
how organizations exploit information. The proliferation of Web 2.0-style inter-
faces lets users interact directly with information, and this heavy reliance on
internet search has set a high bar in terms of easy and immediate access to the
most up-to-date information.

2 | MarkLogic whitepaper

For many organizations, XML will be even
more ubiquitous due to the simple fact that
productivity tools such as Microsoft Office
and OpenOffice have adopted XML as their
base format.
3 | MarkLogic whitepaper
XML has proven to be the right technol-
ogy for these issues. Because of its ability
to represent the many forms information
comes in—ordered, hierarchical, textual,
and irregular—there are new and emerg-
ing XML-based standards for representing
information in different industries. In the
financial sector, industry standards include
Financial products Markup Language
(FpML); for general business and financial
reporting, eXtensible Business Reporting
Language (XBRL); for Department of De-
fense resource discovery, Defense Discov-
ery Metadata Specification (DDMS); and in
publishing, DocBook. There are also XML
standards for representing and integrating
technical concepts, such as SensorML for
describing sensor systems, and geography
markup language (GML) for modeling and
exchanging geographic information. And
XML is already widely used as a messag-
ing/enterprise information integration (EII)
format across industries and for document
markup in publishing and government.
Industry standards are based on XML be-
cause it improves an organization’s ability to
achieve objectives via fast, intuitive access,
dynamic reuse, and flexible, personalized
information delivery.
XML is the Future
With the growing number of XML-based
industry standards, more tools are available
for authoring, manipulating, and enriching
XML. This has increased the adoption of
XML and spurred the creation of more tools
for XML creation and processing. For many
organizations, XML information will be even
more ubiquitous due to the simple fact that
productivity tools, such as Microsoft Office
and OpenOffice, have adopted XML as their
base format. This means office workers are
now XML authors and create huge amounts
of XML documents with no additional
training. Vendors, such as Adobe and Quark
with broadly deployed document creation-
focused products, have started to increase
the XML orientation in their offerings.
Whether you realize it or not, XML already
exists in your organization.
Platforms designed for XML help people
unlock the value of information. The prob-
lem is existing tools and technologies have
not met all the challenges of information
management and have forced organizations
to make trade-offs. MarkLogic Server, the
industry’s leading XML server, was designed
and optimized specifically to address the
complex nature of XML. It allows enter-
prises and government organizations to
transform operations by maximizing the
value of their information assets. In the fed-
eral government, a wide range of agencies
have used MarkLogic Server to change the
way they manage and distribute informa-
tion through innovative, new knowledge and
information management systems which
dynamically deliver information to users.
The Challenges of Information
What is information? For most organiza-
tions, it is the combination of discrete
data types—such as numbers, dates, and
names—with semi-structured and unstruc-
tured text. Information is often recognized
as books, journals, articles, reports, and
technical documentation. It also includes
popular text sources such as email, web
pages, general message traffic, and user-
generated comments. In this context,
information is semi-structured: unlike data,
it is irregular in length and structure, and
largely textual in nature. XML is the ideal
model for storing, managing, and delivering
all information.
Information often creates challenges for IT
organizations because they typically treat
it like relational data. Among the biggest
challenges are:
• Structures— information comes in many
forms, formats and structures
• Size— organizations generate an enor-
mous amount of information
• Semantics— relevant results depend on
understanding the meaning of information
The Challenge of Structure
One of the fundamental challenges of infor-
mation is the nearly infinite variations of
structure it can have. Structure can be high-
ly variable and dynamic. We see this just by
looking at a few different information types:
articles can have a different structure from
journals, which can have a different struc-
ture from books; email has some structure
(e.g., standard fields like to, from, date); and
HTML web pages mix structure with display.
Consider the text in a book which can have
headings, chapters, and paragraphs that
form the first levels of structure, then add
tables and captions, extracted entities (peo-
ple, places, things) and even user-tagging
and comments. What appears at first glance
to be regularly structured quickly becomes
irregularly structured information.
When storing, manipulating, or updating
information, organizations consider using
1
schemas for a range of reasons including
catching errors and making queries faster
and more accurate. In doing so, they often
have to make a choice between maintaining
multiple structures and attempting to en-
force a single structure on their information.
Enforcing such a schema on a large scale is
not always possible. Books and journals may
have been authored with different tools, and
information may come through corporate
acquisition or organizational merging. Yet
an organization may want to exploit all
of this information together to build new
products or repurpose information for new
uses. Additionally, changing business condi-
tions inevitably force the creation of new
structures over time.
The following simple examples in Figure 1
illustrate different ways to model informa-
tion with the text-oriented DocBook schema
(www.docbook.org). Inside a paragraph you
can have any number of tables, emphasis,
sub-titles, etc., in any order. This schema al-
lows information to be arranged in different
ways.
Both examples above are valid structures in
DocBook, but they have significant differ-
ences in terms of which elements exist and
the hierarchy they represent.

<book xmlns=”http://docbook.org/ns/docbook”> <book xmlns=”http://docbook.org/ns/docbook”>

<title>...</title> <title>...</title>
<chapter> <subtitle>...</subtitle>
<title/> <bookinfo>
<para> <author>
<phrase>...</phrase> <firstname>...</firstname>
<emphasis> <surname>...</surname>
<quote>...</quote> </author>
</emphasis> <author>
</para> <firstname>...</firstname>
</chapter> <surname>...</surname>
<chapter> </author>
<title/> </bookinfo>
<para/> <chapter>
</chapter> <title>...</title>
</book> <para> ...
<phrase>...</phrase> ...
<table>
<caption>...</caption>
<tbody>

<tr>
<th/>
<td/>
<td/>
</tr>
</tbody>
</table>
<graphic/>
</para>
</chapter>
</book>

Figure 1. Different valid structures with the same schema.

1 www.wikipedia.org [Schema] A schema is a way to define the structure, content and, to some extent, the

semantics of XML documents.

4 | MarkLogic whitepaper
 Understanding the structure and represen-
tation of information allows you to build
more powerful solutions because you have
more control over access and retrieval.
You can ask questions of your information
that combine the knowledge of structure
and the text. It is possible to search for the
existence of a particular word in a part of
the structure like a title, footnote, or cap-
tion. For example, when searching medical
journals a researcher might ask “find the
word lymphoma when it is used in a caption”
and then retrieve that caption along with its
associated table or image.
When accessing information using search
techniques, it is important that the full-text
search capabilities are more than just “XML
aware” meaning it knows the type of docu-
Whether your organization is big or small, you
ment is XML or that some fields might need
want to know that the solution you choose for
processing. As a comparison, search engines
loading, storing and accessing information can
do not typically handle XML well—at best
scale to provide the same level of capabili-
they may recognize (in order to skip during
ties in 5 years as it does for relatively small
indexing) or completely ignore XML tags, or
amounts of information today.
index a fixed number of pre-defined fields.
This limitation makes search engines a poor
choice for exploiting XML, especially since
updates to the indexing structure — known
as schema migration — require complete
re-indexing of the documents, increasing
the administrative burden and reducing user
flexibility.
The Challenge of Size
It has been estimated that 80% of an
organization’s total information is unstruc-
tured or semi-structured. Only 20% is struc-
tured and suitable for storage in a relational
database. The volume of unstructured or
semi-structured information presents many
challenges beyond the obvious storage and
computing requirements. Whether your or-
ganization is big or small, you want to know
the solution you choose for loading, storing,
and accessing information can scale to
provide the same level of capabilities in five
years as it does for relatively small amounts
today. For some, scalability is about the
number of users, simultaneous queries or
transactions, and query speed. For others, it
is about the sheer amount of raw informa-
tion the system must store and manage. And
for many, it is all of the above.
5 | MarkLogic whitepaper
As information grows, the ability to ef-
ficiently respond to complex queries
becomes more important. Complex queries
might have tens or even hundreds of terms.
For example, a researcher might need to
query a large amount of scientific and pat-
ent data with extensive queries for patent
related research. By providing a fast re-
sponse, an XML server lets researchers give
better value to customers because they can
be more responsive and do more detailed
research. Architecturally, this means a solu-
tion must be able to distribute processing
and storage efficiently in order to provide
the necessary responsiveness.
.
The Challenge of Semantics
Another significant challenge related to
information is that of semantics, or the
understanding of meaning within informa-
tion. Understanding the semantics lets
an organization more accurately search,
access, interpret, and manipulate text. They
can use the semantic knowledge to access
and deliver the right information to users.
At the most basic level, understanding
meaning requires words to be broken down
to their base forms, a process called stem-
ming, in order to ensure the right informa-
tion — and only the right information — is
found. Consider the words “mouse” and
“mice” — a search for “mouse” should find
all instances of both words. Additionally,
a thesaurus can be used to find synonyms,
or a broader or narrower form of a word or
phrase. In the example above, a thesaurus
would enable the search for “mouse” to be
expanded to find the word “rodent” or nar-
rowed to “field mouse.” Similarly, taxono-
mies — the classification of information
based on a set of criteria — can be applied
to organize information.
What about semantic ambiguity? Should
a search for “George Washington” return
results for the street, the university, the
person, and the bridge? Increasingly, entity
extraction engines are being used to disam-
biguate George Washington as the person
and president from the bridge and from the
university through XML tags. Solutions that
manage and deliver XML must leverage this
additional semantic information to deliver
the correct information to users.
Organizations will face the challenges of
structure, size, and semantics to varying de-
grees when trying to maximize the value of
their information. A system that can over-
come these challenges must be architected
in a way that allows for superior flexibility
and scalability.
MarkLogic Server: Addressing the
Challenges of Information
MarkLogic Server meets today’s informa-
tion challenges of structure, size, and
semantics because it was specifically
designed from the ground up for XML. In
MarkLogic Server, XML is the native format
for storing and delivering information.
MarkLogic Server is a platform for build-
ing information applications that address a
wide range of solutions, including content
integration, metadata catalogs, information
analysis, information reuse, and dynamic
delivery.
Optimized for XML and XQuery
When MarkLogic Server was first conceived,
two fundamental assumptions were made
regarding the direction of technology.
The first was the emergence of XML as a
format for representing semi-structured
information. The second was the flexibility
of XQuery to easily access and manipulate
XML. Both of these assumptions have
proven to be correct, and today they repre-
sent two of the fundamental strengths of
MarkLogic Server. By focusing on these two
industry-standard technologies, MarkLogic
engineers can make design decisions that
optimize performance for processing XML.
This is at the core of MarkLogic Server’s sig-
nificant long-term, architectural advantages
over traditional technologies.
XML Top-to-Bottom
XML is the format used for storing and
delivering information to other applications,
to the web, and everywhere in between.
MarkLogic Server is the industry’s leading
XML server and is an extremely effi-
2 W3C XQuery Test Suite Result Summary - http://www.w3.org/XML/Query/test-suite/XQTSReportSimple.html
cient system for processing XML. Within
MarkLogic Server, information is main-
tained in a highly compressed, parsed XML
form to improve throughput and processing
speeds. It decompresses and serializes
information before it is delivered to an ap-
plication or the web.
XQuery for Information Access and
Manipulation
XQuery was designed as a query language
for XML, just as SQL is a query language for
relational databases. Today, MarkLogic has
the industry’s most extensive implementa-
tion of XQuery 1.0, the W3C standard for
XML query, and achieves 99.9% conform-
2
ance to the specification. XQuery is a
powerful language for accessing informa-
tion and building applications that contain
all of the constructs needed to provide
large-scale, highly sophisticated solutions.
MarkLogic Server also includes extensions
for full-text search, information update,
and programming. All of this means you can
write high-performance applications in a
high-level, declarative language.
The MarkLogic full-text search extensions
to XQuery allow organizations to compose
full-text search and structured XML search
to provide users with one-step answers to
questions. MarkLogic Server, with these
XML full-text extensions, provides signifi-
cant advantage over other full-text search
solutions, such as enterprise search prod-
ucts that simply provide users with pages
of links to documents, web pages, and other
information.
6 | MarkLogic whitepaper
 Another extension of the XQuery standard
enables fine-grained updates to XML in-
formation. XQuery was originally designed
3
for accessing XML through an XPath-like
selection of information. The architects
of MarkLogic Server recognized that a
database management system (DBMS) for
XML needed to have the ability to update
information, resulting in the development of
information-update extensions. MarkLogic
Server’s XQuery implementation includes
extensions for inserting, updating, and
deleting parts of documents.
A third set of extensions focus on error and
exception handling. Programs often perform
many operations that have the potential
for generating exception conditions that
under normal circumstances would cause an
application to fail. An example is a divide by
zero error. One easy way to gracefully han-
dle this error is through try/catch exception
handling. MarkLogic Server extends XQuery
with try/catch support to capture errors as
part of the normal application functionality.
Most programming languages have this ca-
pability but the XQuery specification does
not include such support, so this MarkLogic
Server extension enables the development
of robust applications.
By selecting XML and XQuery as the stand-
ard for storing and interacting with informa-
tion, MarkLogic has created a platform ideal
for building information applications that
address the challenges of structure, size,
and semantics.
3 XML Path Language - http://www.w3.org/TR/xpath
7 | MarkLogic whitepaper
An Enterprise-Ready Database
Management System for XML
Designed specifically to handle XML
information, MarkLogic Server has the
capabilities of an enterprise-class DBMS. It
provides storage, query, read consistency,
update, failover, backup, and restore on a
highly scalable and flexible architecture. It
also provides administrators with easy-to-
use tools and interfaces to load and manage
information, and interfaces for managing
and monitoring a deployment. It easily in-
tegrates with systems within a data center
and provides key functionalities previously
missing from enterprise architectures that
must cope with the challenges of informa-
tion.
Fast, Efficient Information Loading and
Processing
Loading information efficiently lets an
organization keep up with the rapid informa-
tion creation, including high-speed activi-
ties such as capturing and storing message
traffic. With MarkLogic Server, you can
create pipelines for processing information
that automates loading using the Content
Processing Framework (CPF). This program-
mable feature allows you to perform various
operations on-the-fly, such as conversion,
information transformation, XML classifi-
cation, entity enrichment, and more. CPF
uses triggers that cause user-specified
operations to be performed. CPF is an ef-
ficient pipeline that can be parallelized to
maximize information loading operations
for deployments with extreme throughput
requirements.
Transactional Storage of Information with
No Compromises
MarkLogic Server implements a transac-
tional system that adheres to the ACID
4
model to guarantee returning the correct
answer. MarkLogic Server also supports
non-blocking read operations which pro-
mote efficiency by ensuring users will never
have to wait on the read/write operations
of others.
The transactional model and ACID compli-
ance give MarkLogic Server an advantage
over search engines and XML-extended
relational database management systems
(RDBMS) in returning correct search results.
Search engines, which are read optimized,
typically use a web-crawler design where
the state of the information reflects the
most recent crawl. Similarly, RDBMSs often
use an asynchronous or lazy index update
model in XML implementations and/or in
full-text implementations. This results in
an index that is often out-of-synch with
the database. In both cases, search engines
and RDBMSs have to sacrifice result ac-
curacy and currency to achieve acceptable
performance.
MarkLogic Server does not force organi-
zations to make these trade-offs. When
information is loaded, it is immediately
made available to query users with assured
transactional integrity. This means organi-
zations can be confident users will get the
correct results.
System of Record
An important difference between
MarkLogic Server and enterprise search
engines is that its XML repository makes it
a system of record for information because
it stores the actual information. Search
engines can only provide pointers to the
original information because their indexes
contain only keywords. This is an important
distinction. When you need to create ap-
plications that modify or alter information,
4 Atomicity, Consistency, Isolation, Durability. These are four primary requirements for transactional data-
bases, ensuring that only complete, valid transactions are written to the database, and preventing any reads
from returning intermediate data from a transaction. See http://en.wikipedia.org/wiki/ACID
or create new information from existing
information, you can store these changes,
updates, and new information directly in
MarkLogic Server.
High Availability
Another key component of any enterprise
DBMS is high availability, which assures
users continuous access to information.
MarkLogic Server ensures high availability
through several mechanisms, including jour-
naling, clustering, and automatic failover.
The journaling system used in MarkLogic
Server ensures data integrity during up-
date operations. A clustered architecture
spreads the storage and query processing
load across multiple servers and improves
system responsiveness in times of heavy
use. Automatic failover protects in the
event of a node failure, which will automati-
cally switch traffic between nodes based on
their status. MarkLogic Server gracefully
handles node failures, so users have access
to the information they need, when they
need it.
Shared-Nothing Architecture
It can be difficult to predict future infor-
mation requirements, how much informa-
tion you will need to store and access, the
number of users that may need to access in-
formation, or the complexity of the queries
users may want to run. One way to protect
your deployment is to use a system where
you can easily add commodity hardware to
scale across more information and/or more
users. This type of architecture is called a
shared-nothing architecture and MarkLogic
Server was designed for deployment using
this method. The goal of this architecture is
to provide scalability by easily adding nodes
to handle increased inbound query requests
and increased information volumes. An important difference between
MarkLogic Server and enterprise search
In a clustered deployment, MarkLogic
engines is the fact that its XML repository
Server can be set up on separate hosts
makes it a system of record for information,
(nodes) and tuned to process inbound query
because it contains actual information.
requests along with information storage and
retrieval. Each of these nodes runs Mark-
Logic Server independently of other hosts.
Nodes tuned for storage and retrieval man-
8 | MarkLogic whitepaper
 age components of the database (Figure 2).
Nodes tuned to handle inbound query traffic
evaluate queries and transform, assemble,
and deliver results back to the requesting
application. Each node can communicate
with every other node in a cluster. Increasing
storage capacity is a simple matter of add-
ing additional nodes configured to store and
retrieve information.
Figure 2. MarkLogic Server shared-nothing architecture.
The process required to add user query
capacity or support increased query com-
plexity is a simple matter of adding nodes to
evaluate inbound requests. Handling larger
information volumes in a deployment is a
matter of adding servers configured for in-
formation storage and retrieval (as opposed
to query focused). MarkLogic currently has
customers with multi-hundred terabyte
databases in production.
Backup and Restore
Information is often an organization’s most
valuable asset, so the ability to protect
and backup that information is crucial.
MarkLogic Server provides a facility for
consistent database backup that allows
administrators to consider their own unique
requirements when determining how much
or how little they want to backup or restore.
They have the flexibility to backup selected
components of the database or the entire
database, giving them the control needed to
9 | MarkLogic whitepaper
meet the specific needs for each deploy-
ment.
Secure Web-Based Administration
Administering a DBMS should be easy
and straight forward. MarkLogic Server
features an easy-to-use, web-based ad-
ministration interface. Administrators can
complete a wide range of tasks, including
changing configuration, managing data-
bases, and managing users. Administrators
can also perform important system-level
tasks such as security management (using
MarkLogic Server’s built-in security compo-
nents), backup and restore operations, and
performance tuning. Finally, the interface
lets administrators perform a complete
status check on their resources.
Efficient Use of Storage
As a result of the rapid increase in storage
densities, conventional wisdom holds that
“storage is cheap.” Unfortunately, while
comparatively inexpensive, it isn’t free. For
databases sized in the hundreds of tera-
bytes, cost can become an issue. MarkLogic
Server addresses this by efficiently using
disk space through extremely effective
and efficient compressed storage. An ad-
ditional benefit of storing information in a
compressed format is that it reduces disk
and network traffic, resulting in enhanced
application performance.
Accessing Information: Search and
Query
MarkLogic Server provides both search
and query capabilities. MarkLogic Server
provides all the standard functionality you’d
expect in a full-text search engine, plus
database-style queries against informa-
tion. At the heart of the search and query
capability is a patented indexing technique
that results in fast queries. So applications
can dynamically transform text, sort results,
reformat for a specific device, and trans-
form messages from one format to another.
Unified Indexing of Text and Structure
Accessing semi-structured information is
easily optimized when you can combine the
existence of the text with the knowledge
of the structure. However, access to text relational databases require three or more
has typically been done through an inverted different indexes to access information.
5
index , supplemented by a second index of These models have serious disadvantages,
the information’s structure in an attempt including additional administration costs,
to provide search in limited portions of a synchronization issues, performance, as
document structure. A better method is to well as forcing administrators to anticipate
create an index that includes the complete user queries. Instead, MarkLogic Server
knowledge of the words and the structure indexes all text and XML structure at load
and inline markup. The result is a system time, and can be further updated without
where all information is accessible, includ- downtime.
ing its structure. MarkLogic refers to this
The universal index includes text, structure,
type of index as a universal index.
and values and can be used by an application
The universal index includes full text, values, to deliver rich user experiences and effec-
and XML structure. By combining what are tively target information delivery.
typically separate indexes into a single
Search Capabilities
universal index, MarkLogic Server delivers
Search is one of the primary ways users
extremely powerful ways to access informa-
expect to locate the information they need.
tion.
Full-featured, highly relevant search is a
One of the advantages of the universal core capability of MarkLogic Server. This
index is you can load any XML as-is. This includes:
allows for immediate access regardless of
• Standard search functionality including
schema or structure. The universal index in
word, phrase, Boolean, wildcard, proximity,
MarkLogic Server uses a patented approach
stemmed, spell checking, and thesaurus
to indexing structure that allows informa-
assisted search
tion in any format to be quickly loaded and
made immediately available to users. Infor- • Relevance measure and relevance ranking
mation from different sources, with differ- based on term frequency and document
ent schemas, can be used without needing frequency, tunable according organiza-
to be normalized, overcoming traditional, tional needs
structure-related challenges.
• Advanced language processing including
Another advantage of the universal index tokenization, collation, and diacritic sup-
is it does not require up-front decisions port
and pre-processing. Search engines
• Faceted navigation and other rich analysis
typically index text with a fixed number of
and exploration user interfaces
pre-determined fields. That enables them
to answer certain types of questions, but if
other types of questions need to be asked
in the future, the schema migration effort
requires a complete, costly re-indexing of
documents. RDBMSs are more restrictive
since they were not designed for ad hoc user
queries, and only provide answers for ques-
tions you know in advance. XML-extended

5 An inverted index is an index data structure which stores a mapping from content elements, such as words or

numbers, to their locations in a document or a set of documents, in this case allowing full text search. – http://
en.wikipedia.org/wiki/Inverted_index
10 | MarkLogic whitepaper
 Combining the powerful full-text search
capabilities in MarkLogic Server with an
inherent understanding of the elements and
attributes of XML exposes the semantics
of information. The following example of
enriched XML demonstrates some of the
complexity involved with the semantics of
information.
When enriched using a text mining tool the
following snippet of text:
“General Motors (NYSE: GM-news) expects
its sales in central and eastern Europe
to jump to 505,000 units this year from
350,000 in 2006, GM Europe chief Jonathan
Browning…”
Transforms into the XML in Figure 3:
<entity><company Domain=“Automotive” value=“general motors”>General Motors</company></entity>
(NYSE:GM - news) expects its sales in central and eastern
<entity><location> <continent value=“europe”>Europe</continent></location></entity>
to jump to 505,000 units
<entity><time><relative_time value=“this year”>this year</relative_time></time></entity>
from 350,000
<entity><time><exact_time Year=“2006” value=“in 2006”>in 2006</exact_time></time></entity>
, GM
<relationship><board_and_management_changes value=“board_function”>
<entity><location><continent role=“Context” value=“europe”>Europe</continent> </location>
</entity>
<entity><function FunctionCategory=“Board” role=“Has Function” value=“chief”>chief</function>
</entity>
<entity><person Family_Name=“Browning” First_Name=“Jonathan” Gender=“Male” role=“Person”
value=“jonathan browning”>Jonathan
Browning</person></entity></board_and_management_changes></relationship>
Figure 3. Enriched XML.
The markup in this example identifies the
organization (General Motors), the locations
(Europe), time elements, role of the person
mentioned, and more. The entire markup
can be captured and stored with the original
text and is called “inline” markup. With appli-
cations powered by MarkLogic Server, users
can search this information and leverage the
markup to get more refined, exact answers
to their questions.
For example, a user could find the names
of executives in GM’s European operations
and have the name “Jonathan Browning”
returned with his title. They could also re-
quire or exclude results containing the year
2006. These capabilities make it possible
11 | MarkLogic whitepaper
to accurately search across information and
leverage the semantics and the meaning of
the text.
Dynamic Navigation
Solutions built using MarkLogic Server
provide a wide range of navigation options
for exploring information. Navigation op-
tions like faceted navigation are possible
because of the universal index and the
fast query processing that enable dynamic
updates of navigation options as the user
clicks through the information. Common
navigation techniques built using MarkLogic
Server include facets, tag clouds, heat
maps, node edge graphs, temporal explora-
tion, geospatial navigation, and traditional
pie and bar charts. Some organizations use
MarkLogic Server to provide users with
multiple navigation techniques so they can
select one that best meets their needs and
preferences.
Combined XML and Full-Text Search
MarkLogic Server goes beyond full-text
search to provide structured XML search.
It can search information within XML ele-
ments, attributes, and values, as well as
combinations of each. This means a user’s
search will examine document structure
and metadata and words. You can restrict
search to specific parts and return indi-
vidual elements that are on a specific path,
such as “return /book/chapter//title”. Users
can also search for specific element values,
so the path-specific example above could
be narrowed to find all elements whose
values equal “The Red Book” on the path /
book/chapter/title. Or users can search for
all elements called <title> with occurrences
of the word “dog” in the path /book/chap-
ter/. The combination of XML search and
full-text search lets organizations answer
questions about information not possible in
other systems, and return not just a pointer
to a document, but the part(s) of the docu-
ment requested. This combined capability is
known as “fine-grained search and retrieval”
of information. It empowers organizations
to dynamically repurpose information at a document as a relevant hit for the phrase
granular level, creating on-the-fly, custom query: “retract the landing gear.” In Example
personalized documents and web pages to B, the XML tags again indicate different
meet specific user’s needs. contexts for the word “retract,” and the rest
of the search phrase, “the landing gear,” is
A Deep Understanding of Structure
in a footnote. Again, a search engine that
Even if an information application can index
ignores XML tags would incorrectly return
all of the structure like MarkLogic Server
this as a relevant hit. Examples C and D
does, it takes more than that to correctly
should be returned as relevant hits, but a
process information; you also have to know
system that does not recognize and cor-
the difference between different kinds of
rectly process XML tags will miss them, as
tags. Some tags are hierarchical, indicat-
the intervening markup breaks the string.
ing parent/child relationship, and some
Example C also illustrates what is known
are inline, e.g., indicating display style.
as “phrase through.” In this case, the XML
MarkLogic Server easily distinguishes
tags <b> and </b> should be ignored for the
between hierarchical and inline markup and
purpose of relevance since they refer only
provides correct results when processing
to bolding the associated text. Example
information that includes these tags.
D demonstrates what is known as “phrase
Let’s look at a simple example of how XML around.” The XML tags <footnote> and
structure interacts with search. In Figure 4 </footnote> must be recognized and ignored
below, examples A, B, C, and D all show inline for the purpose of relevance and proximity.
markup of text. They represent the case of
MarkLogic Server’s deep understanding
pilots in an airplane looking for instructions
of XML allows it to easily handle informa-
on how to retract landing gear in a document
tion in the examples above as well as more
using search. This type of inline markup of
complex XML such as enriched information
information is sometimes referred to as
that is becoming more common in real-world
mixed content and presents serious chal-
implementations. Mixed content creates
lenges to search engines and XML-extended
special challenges because markup, such as
relational databases. Even in situations
style information like bolding or references,
where the XML is relatively simple, search
is embedded within the text. MarkLogic
engines and XML-extended relational data-
Server has been built for arbitrarily complex
bases can return the wrong answer.
XML and is the right technology to unlock
In Example A, the document describes the the value of your information.
operational steps and the XML tags indicate
that “retract” is in a different step from “the
landing gear.” A search engine that ignores
XML tags would incorrectly return this

Example A Example B Example C Example D

<step number=“3”>
Wait for the automatic pilot
control to retract.
</step> <step number=“4”>
The landing gear should now
be ready to deploy.
</step>
</warning>
<warning>
While waiting for the flaps to
fully retract
<footnote> The landing gear
need to be stowed or else the
flaps will remain deployed
< /footnote>, ensure that the
automatic pilot control is at
hand.
<p>
It is vital that you retract
<b>the landing gear</b>, but
leave the flaps fully deployed.
</p>
<step number = “5”>
Partially retract
<footnote>It is not recom-
mended to fully retract at
this point.
</footnote> the landing gear.
</step>

Figure 4. Search engines and XML extended relational databases mishandle XML (search phrase “retract the landing gear” shown in bold font for emphasis).

12 | MarkLogic whitepaper
 Query Optimization
In database query performance optimiza-
tion, an important technique is to evaluate
the most restrictive predicates first in
order to minimize the data sets on which
the database operates. MarkLogic Server
does this with the universal index, where
predicates are automatically evaluated so
restrictive terms constrain less restric-
tive terms. By using an extremely efficient
algorithm for merging term lists, MarkLogic
Server is able to deliver maximum applica-
tion performance.
Instant Search
MarkLogic Server supports instant search,
which ensures that information loaded into
MarkLogic Server can be searched im-
mediately. It eliminates the latency that is
typically found in applications that rely on
separate systems to handle storage and
search. In those applications, the search
index is often not up-to-date, as the lag re-
duces the overhead cost of continual index
updates. Shorter lags require more comput-
ing resources, so applications developers
must make a trade-off by only periodically
updating the search indexes to reduce
resource requirements.
MarkLogic Server’s unified architecture en-
sures that storage and search synchroniza-
tion is done automatically, immediately, and
MarkLogic Server’s enterprise-class database capabilities, extensive full-text
and XML search, and a robust XQuery development environment provide a
complete platform for building information applications.
13 | MarkLogic whitepaper
efficiently, allowing end users to search for
information as soon as it is loaded. Users
are assured that when they run searches,
they are accessing all information available
at the time of the search. Any end user who
needs immediate access to time-sensitive
information, such as financial market news
or intelligence reports, can search over the
latest information without waiting for the
technology to catch up.
Real-time Alerting
Real-time alerting, also known as profiling,
filtering, and proactive search, is the search
technology for users who need to receive
information as soon as it is available. The
value of information is often related to
its timeliness, so whether the information
consists of intelligence reports, financial
market data, or even late-breaking busi-
ness news, users are demanding real-time
delivery so they can respond immediately.
Users can set up an alert such as, “send me
an email whenever there’s news about oil
and gas within 100 miles of San Francisco”,
and will not need to run the search over and
over.
Real-time alerting takes MarkLogic Server’s
extensive query capabilities and rapidly ap-
plies them to an incoming feed of informa-
tion. This type of monitoring allows users
to set up alerts to take specific actions on
information that matches their topics of
interest. Millions of alerts can be created to
enable a variety of immediate actions on the
matching information, including delivery,
categorization, enrichment, and transforma-
tion. Since the alerts are optimized for this
type of search, they are far more efficient
than mechanisms in systems that periodi-
cally run standard queries.
Geospatial Search
The use of geospatial information continues
to grow as more users demand the geo-
graphical relevance of information. Many
organizations already have large collections
of geospatial information, and those that
do not can leverage enrichment tools to add
geospatial data to entities such as coun-
tries, buildings, and points of interest. And
with the growth of location-aware mobile
devices, geospatial information appears to
become even more relevant to our everyday
lives. One consumer-oriented example of
geospatial search is, “show me all restau-
rants within 5 miles of where I am that have
server filet mignon”.
MarkLogic Server enables organizations
to identify geographical relevance in
their information with geospatial search
capabilities combined with full-text search.
MarkLogic Server supports a comprehen-
sive set of query criteria including point
search, radius search, latitude-longitude
box search, and polygon search in which
thousands of boundary points can be speci-
fied for a polygon. Users can also specify
any number of locations per document or
record, allowing flexibility in the formatting
of information.
MarkLogic Server has built-in support for
various geospatial types such as Geography
Markup Language (GML), Keyhole Markup
Language (KML), and GeoRSS. MarkLogic’s
high-performance engine allows fast query-
ing over billions of records.
Visualization is an important part of any
geospatial interface, and MarkLogic Server
can integrate with any geographic-aware
product such as ESRI, Google Earth, Google
Maps, Yahoo! Maps, and Microsoft Live
Search Maps. Heat maps are also supported
so users can see a distribution of data over
a geographic area and instantly refine those
results.
A Complete Platform for Building
Information Applications
MarkLogic Server’s enterprise-class data-
base capabilities, extensive full-text and
XML search, and robust XQuery develop-
ment environment provide a complete
platform for building information applica-
tions. Because MarkLogic Server is XML
top-to-bottom, application developers can
more quickly develop better applications.
They can build end-to-end applications in
simple infrastructures, or they can integrate
with existing applications in Java or .NET, or
integrate with a Service-Oriented Architec-
Figure 5. XML servers are comprehensive
ture (including SOAP or ReST interfaces). platforms for application development.
They create more efficient and reliable ap-
plications that incorporate built-in business
logic, expose powerful search capabilities,
and ensure users get the most up-to-date
and correct results.
A Robust Application Development
Platform
The MarkLogic Server architecture
provides a complete platform for develop-
ing information applications. In addition
to its extensive search capabilities and
enterprise-class DBMS, MarkLogic Server
includes a web server for serving XHTML
to web browsers. Developers can create
complete applications and define func-
tional libraries for reuse. Organizations can
create centralized repositories of informa-
tion within MarkLogic Server and rapidly
develop multiple applications efficiently
reusing and repurposing information in
multiple applications.
The completeness of MarkLogic Server’s
XML and XQuery implementations minimize
the need to use other languages such as
Java to complete the data processing. This
is important because there is a performance
penalty that comes with converting XML
to objects for Java processing. This XML
conversion, often referred to as the X/O
impedance mismatch problem, is a com-
mon problem if you need to program in Java
against XML.
Applications that deliver information to
a browser in particular are more efficient
when information remains in XML and does
14 | MarkLogic whitepaper
 not have to be transformed into other for-
mats or storage models. MarkLogic Server
outputs XML directly to eXtensible Hyper-
text Markup Language (XHTML), which can
be rendered by web browsers without the
need for expensive transformations and
allows for easy modification of applica-
tion user interfaces. In addition to XHTML,
MarkLogic Server can render final output
in any format required (such as Microsoft
Word or PDF) using native capabilities or
third-party conversion tools. Furthermore,
MarkLogic Server includes a full, embedded
web server, allowing it to serve HTML or
XHTML directly to the browser and avoid
potential bottlenecks and latency issues.
MarkLogic Server also avoids the need
for a thick “middle layer” of information
manipulation code. Instead of getting URLs
from a search engine, fetching documents,
parsing them, and navigating the Docu-
ment Object Model (DOM) to find required
elements, MarkLogic Server lets users
simply request the elements they need in
XQuery and returns them directly. Addition-
ally, developers are able to keep business
logic close to the information making them
more efficient to write and run. You can, of
course, still build a middle layer with Java or
.NET through standard APIs.
Developer-Friendly Environment
As developers create more and more
complex applications using XQuery, it is
important their platforms provide tools and
interfaces for optimizing and debugging
applications. MarkLogic Server provides a
developer-friendly environment including
interfaces that profile XQuery programs
to reveal call trees and execution times for
each XQuery function executed. This ena-
bles developers to optimize code and iden-
With applications built with MarkLogic Server, businesses and
government organizations are taking the use and reuse of their
XML information to new levels with innovative information ap-
plications.
15 | MarkLogic whitepaper
tify potential bottlenecks. MarkLogic also
has integrations with XQuery debugging
environments such as Eclipse and oXygen.
These help developers more efficiently find
code errors.
Ease of Integration with Standard Inter-
faces
Since MarkLogic Server conforms to XML
standards and operates natively in XML for
input, storage, and delivery, it can eas-
ily integrate with any third-party product
that supports XML. It is commonly used
as a “hub” in a system architecture linking
different applications which speak XML,
enabling them to work together. It does this
easily because XML message traffic can be
consumed as-is and can speak all the differ-
ent flavors of XML spoken by the various
systems. This speeds and eases integration
of disparate applications in large system
architectures. Additionally, MarkLogic has
prebuilt interfaces to JAVA and .NET to al-
low for integration into those development
frameworks.
Rapid Application Development
MarkLogic Server supports rapid applica-
tion development with Application Services,
a set of high-level APIs and interactive
tools. Application Services consists of three
components. The Search API provides a
high-level library that developers can use to
quickly incorporate core search functional-
ity, such as faceted navigation, dynamic
result snippets, and automatic term sugges-
tions in their own applications. MarkLogic
Library Services API is a way for developers
to add document management functionality
such as versioning and check-in/check-out
to their information applications. Finally,
MarkLogic Application Builder is a browser-
based tool designed to rapidly prototype
applications through agile development
cycles, and ultimately to build baseline ap-
plications that can be further customized.
All components of Application Services
leverage best practices for performance
and scalability , freeing application devel-
opers to focus on their applications’ unique
features, not the underlying search and
management infrastructure.
Toolkits and Connectors
MarkLogic Server’s set of toolkits and
connectors enable rapid integration with
existing IT infrastructures. MarkLogic
Connector for SharePoint enables informa-
tion in SharePoint to be integrated into
MarkLogic Server for additional processing
and delivery. The Office toolkits MarkLogic
Toolkit for Word, MarkLogic Toolkit for
PowerPoint, and MarkLogic Toolkit for Excel
— enable Office files to be easily reused in
custom documents with a search-and-insert
interface built into the Office interfaces.

This set of integration tools promote com-

ponent content reuse, document tracking,
auditing and reporting, and extensive de-
livery options. Custom dynamic publishing
infrastructures can be built with MarkLogic
Server that leverage collections of existing
Office files and require minimal training to
increase end-user productivity through easy
information reuse.

Conclusion
MarkLogic Server overcomes the structure,
size, and semantic challenges related to in-
formation through its superior architecture
and XML capabilities. The universal index,
integrated full-text and XML search, and
high scalability helps organizations maxi-
mize the value of information. MarkLogic
Server’s complete application framework
provides the industry’s leading XQuery
implementation, which means organiza-
tions can confidently rely on it to meet their
needs today and grow with them as their
information and business needs evolve.

16 | MarkLogic whitepaper
 MarkLogic Corporation
www.marklogic.com
sales@marklogic.com
Headquarters
999 Skyway Road, Suite 200
San Carlos, CA 94070
+1 650 655 2300
New York, NY
5 Penn Plaza
23rd Floor
New York, NY 10001
+1 646 378 2104
Washington, D.C.
1600 Tysons Boulevard
8th Floor
McLean, VA 22102
+1 703 245 8590
United Kingdom
3000 Hillswood Drive
Hillswood Business Park
Chertsey, Surrey, KT16 0RS
United Kingdom
+44 (0) 1932 796 400

Version 2 © Copyright 2010 MarkLogic Corporation. MarkLogic is a registered trademark and MarkLogic Server is a trademark of MarkLogic Corporation, all
January 2010 rights reserved. All other product names mentioned herein are the property of their respective owners.