Вы находитесь на странице: 1из 31

Application, Database, Document, eBusiness, and Legacy System Archiving With iWay Service Manager and MongoDB

iWay Software Scott Hathaway Kam Wong

Agenda
iWay Archive Overview and Use Cases iWay Archive Technical Discussion

Copyright 2009, Information Builders. Slide 2

iWay Archive

 What is iWay Archive


 An extension of iWay Service Manager Integration Server  Leverage iWay process-oriented integration and data    
federation capabilities Long term data retention Large repository, large index (Big Data) Search and retrieve capabilities (High performance) Business use examples  Satisfy regulatory requirement  e-Discovery (e.g. research, forensic)  Business analytics

Copyright 2009, Information Builders. Slide 3

iWay Archive Solving Business Needs


Examples of Business Requirements:
Regulations / Reqrs Federal Record Retention Requirement FDA 21 CFR Part 11 HIPAA (Healthcare) Sarbanes-Oxley (public companies) SEC 17a-4 (Financial services) Research Analytics Example Data Patient health records Clinical trials and FDA approval Pediatric medical records Audit Account records Corporate documentation Life science Financial / Legal Retention 75 years (after last episode of care) 35 years 21 years 7 years 6 years Life of the enterprise Long-term Long-term

Copyright 2009, Information Builders. Slide 4

iWay Archive Types of Data


Can handles all types of data, for example:
 Electronic Documents     
Word, Excel, EDI, HL7, XML, Applications  ERPs, CRMs, SAP, SFDC, Database Data IMS, DB2, Oracle, Sybase, SQL Server, MUMPS, Electronic Files VSAM, Unix, Logs, Email Outlook, Lotus Notes Others Multimedia files, Paper, Blueprints, Forms, Claims,

iWay pre-built interface components can be used to connect to the different types of data.

iWay Archive Archiving Needs


Examples of Archiving Requirements:
Archive Requirements Policy Based Logical selection of DB records/transactions to be archived Store very large amounts of data in archive Keep data for a very long periods of time Become independent from Applications/DBMS/Systems future proof Protect authenticity of data regulation and compliance Access archived data when needed / as needed Quickly search huge numbers of archived documents Discard data after retention period regulation and compliance

Copyright 2009, Information Builders. Slide 6

iWay Archive Example (#1) Business Use Case

 Store 75 years worth of patient data  Diverse Sources


 XML  MUMPS  Oracle  HL7  Support archive, query and integration scenarios  XML to remain unchanged and exist outside the data store  Ability to query documents  Ability to retrieve original XML or part of XML using XQuery  Ability to integrate XML archived data in federated services with operational sources (e.g. MUMPS, HL7, Oracle)
Copyright 2010, Information Builders. Slide 7

iWay Archive Example (#1) Business Requirements

 Highly scalable high performance document 

management database Easily integrates into the iWay Federation architecture  Multi-threaded parallel processing  Distributed processing  Just another data source along with, e.g., Oracle and MUMPS databases  Leverage iWays Integration Tools (iIT) that include process orchestration, data mapping/transformation, security and data aggregation capabilities. Implementation and vendor neutral archived data (e.g. XML) stored in the operating systems native file system
Copyright 2007, Information Builders. Slide 8

iWay Archive Example (#2) Business Use Case

 Aggregate various data types from multiple locations  Real-time location systems, personnel tracking, equipment

tracking, patient tracking, (via RFID)  Other Sources  DBMS, Oracle, Maximo  Support archive, query and integration scenarios  Ability to aggregate data at the facilities, regional and national levels  Ability to query data and perform business analytics across the enterprise  Ability to integrate archived data in federated services with operational sources
Copyright 2010, Information Builders. Slide 9

iWay Archive Technical Discussion

Copyright 2009, Information Builders. Slide 10

Overview
Highly configurable iSM application that can be customized to specific needs. Load Channel Reads XML documents and loads them into the document repository. Query Channel Handles query request and response against the document repository. Test Channel Simple visual interface displaying functionality and usage of the Query API.
Copyright 2009, Information Builders. Slide 11

Technology Involved
iSM iWay Service Manager nHTTP Listener mongoDB Agent mongoDB - http://www.mongodb.org/ JSON - Java Script Object Notation XQuery - XML query language

Copyright 2009, Information Builders. Slide 12

mongoDB

 Humongous  Scalable, high-performance, document-oriented database.  JSON-style documents.  Mirror capable.  Auto-Sharding (clustering), horizontal scaling, automatic
failover, zero single point failure.  MapReduce support for complex processing. Work is distributed among the cluster.  GridFS support.  A distributed file system.  Commercial support (OEM by iWay)

Copyright 2009, Information Builders. Slide 13

XQuery

 A query and functional programming language for XML


documents.  Is to XML documents what SQL is to databases.  FLWOR expressions.  FOR, LET, WHERE, ORDER BY, RETURN  Example: for $x in /FEDREG/CNTNTS/AGCY where $x/EAR=Agricultural order by $x ascending return $x  Supports syntax for constructing new documents.

Copyright 2009, Information Builders. Slide 14

JSON JavaScript Object Notation


The new data-interchange language of the web. www.json.org

Copyright 2009, Information Builders. Slide 15

Base Loading Architecture


iWay Service Manager Listener XML to JSON Flow Store JSON Store XML

GridFS

mongoDB

Binary Storage

Copyright 2009, Information Builders. Slide 16

Base Query Architecture


iWay Service Manager Listener HTTP Query DB Flow (Optional) Get XML

Requester

GridFS

mongoDB

Binary Storage
Copyright 2009, Information Builders. Slide 17

Loading Modification
External Storage iWay Service Manager Listener XML to JSON Flow Store JSON Store XML

mongoDB

File System

Copyright 2009, Information Builders. Slide 18

Loading Modification
SAP Loading Architecture iWay Service Manager Flow RFC Server SAP System IDOC to XML Store JSON XML to JSON Store XML Store IDOC

GridFS

mongoDB

Binary Storage
Copyright 2009, Information Builders. Slide 19

Loading Modification
CDC Loading Architecture

iWay Service Manager Flow CDC Listener RDBMS XML to JSON Store JSON Store XML

GridFS

mongoDB

Binary Storage

Copyright 2009, Information Builders. Slide 20

Loading Modification
Salesforce Loading Architecture

iWay Service Manager Flow SOAP XML to Listener JSON Salesforce System
GridFS

Store JSON

Store XML

mongoDB

Binary Storage

Copyright 2009, Information Builders. Slide 21

Loading Modification
FTP Loading Architecture

iWay Service Manager Flow FTP Server File System


GridFS

XML to JSON

Store JSON

Store XML

mongoDB

Binary Storage

Copyright 2009, Information Builders. Slide 22

Query Modification
Web Service SOAP Query Architecture

iWay Service Manager Flow Listener Web Service Client SOAP Query DB (Optional) Get XML/ IDOC

GridFS

mongoDB

Binary Storage
Copyright 2009, Information Builders. Slide 23

The Test Client

Note: The archive is designed to be called from other flows or programs.

 A simple AJAX based human interface for querying the iWay


Archive.  Provides examples of the HTTP query interface provided by the base iWay XML Archive.  Installed with the base implementation of the iWay XML Archive.

Copyright 2009, Information Builders. Slide 24

Simple Example
Loaded this simple XML Doc:

Copyright 2009, Information Builders. Slide 25

Displaying the Document


XML Link:

JSON Link:

Copyright 2009, Information Builders. Slide 26

Basic Query
Return all documents who have the name attribute of the <a> element equal to bob.

Copyright 2009, Information Builders. Slide 27

Advanced Queries
Query handler is a wrapper around the mongoDB query language. Support for: And Or Regular Expressions Ranges

Copyright 2009, Information Builders. Slide 28

Basic XQUERY
Return only the <b> element from the document.

Formatted Result:

Copyright 2009, Information Builders. Slide 29

Availability

 6.1.1 at the end of July  Installed using the new iWay Integration Application (iIA)
feature.

Copyright 2009, Information Builders. Slide 30

Questions?

 Email: Scott_Hathaway@ibi.com

Copyright 2009, Information Builders. Slide 31