DataFrameworkTooling SEARCH InstallationGuide

Data Framework
Searchable Database (SOLR)

Installation Guide
Data Framework Searchable Database – Installation Guide
Table of Contents
Table of Contents ............................................................................................................................... 2
History ............................................................................................................................................... 4
Installation Guide ............................................................................................................................... 5
Overview ................................................................................................................................................................... 5
Searchable Database Architecture............................................................................................................................ 6
Functional Architecture ......................................................................................................................................... 6
Technical Architecture........................................................................................................................................... 7
End to End Architecture ........................................................................................................................................ 8
System Requirement................................................................................................................................................. 9
Runtime Configuration............................................................................................................................................10
ETL Configuration ....................................................................................................................................................15
Configuring Data Framework for the Extract phase on JBoss .............................................................................15
Configuring Data Framework for Transform and Load .......................................................................................22
Artefacts Generation...............................................................................................................................................25
Generation of Artefacts.......................................................................................................................................25
Solr Server Configuration ........................................................................................................................................32
Solr 7.5 download................................................................................................................................................32
Set JAVA_HOME ..................................................................................................................................................32
How to start Solr Server ......................................................................................................................................32
Solr server setup..................................................................................................................................................32
How to Stop Solr Server ......................................................................................................................................35
UXPB Configuration.................................................................................................................................................36
SMS Configuration ..................................................................................................................................................37
Deployment.............................................................................................................................................................40
Deployment Packs ...............................................................................................................................................40
IF Package Installer ..............................................................................................................................................40
DataFrameworkTooling-SEARCH Distribution pack ............................................................................................44
Deployment of Extract ........................................................................................................................................44
Deployment of TL ................................................................................................................................................45
Initial Load Process .................................................................................................................................................46
APPLICATION - INITIAL LOAD PROCESS ...............................................................................................................46
Configuring the IF.INITIAL.LOAD.SERVICE ...........................................................................................................46
2
Verification After Data Load during ILP PROCESS ...............................................................................................51

Data Search .............................................................................................................................................................53
Display Content in the drop down ......................................................................................................................53
Format of the search result shown in the drop down box in UXPB ....................................................................53
Sample screen .....................................................................................................................................................53
Search is made on Plain text as well as Encrypted Text ......................................................................................54
Encryption and Masking ......................................................................................................................................55
How to add new Local Reference & Modify fl list? .................................................................................................58
Other Configuration ................................................................................................................................................63
OtherBookAccess ................................................................................................................................................63
Row Size...............................................................................................................................................................64
AES256 Encryption Algorithm .............................................................................................................................66
APPENDIX ................................................................................................................................................................67
How to generate password in encryped form?...................................................................................................67
3
History
Author Version Release Date Comments
Sivaprakash C 1.0 24-Jun-19 Initial Draft

Sivaprakash C 1.1 15-Jul-19 Modified Searchable Database Architecture, ETL Configuration,
Initial Load Process, Data Search and Other configuration
Jothilakshmi G 1.2 17-Jul-19 Updated the document with the latest screenshot for all the
sections
4
Installation Guide
Overview
The searchable data is data that can be used with a Google type text search whereby hit lists of results
can be returned and used to refine either the search or link to other queries or data. An example could
be searchable customer data, which includes relevant ‘text’ data about the customer, such as name,
age, address, sector, industry, relationships, etc.
Searchable database provides google-style search capability in the UXP browser and the search
database is getting updated in near real time through ETL process. The data has secured with T24 SMS.
The default configuration is only for Customer application.
It can be extensible with Integration framework and Data Framework design time.
Searchable database provides-
 Ability to search any data from T24 applications , namely Customer irrespective of fields in T24
Source being Single value or Multi Value
 Ability to support multiple languages
 Ability to add extra attributes – using joins, enrichments to search a data set based on client
needs
 Ability to support additional data using client specific local reference fields
 Fully integrated to T24 Security Management System for searchable data
 Ability to join Account number and ALT.ACCT.ID, ALT.ACCT.TYPE from
CUSTOMER.ACCOUNT and ACCOUNT application via IF exits
 Ability to launch a 360 Single Customer View Screen from the dropdown box provided in UXP
browser Screen
 Ability to provide configurable data below search box window
 Ability to show masked data – e.g. show ‘4 last characters’ for accounts
 Ability to support data encryption for selective data (for commerce ACCOUNT.NUMBER,
ALT.ACCT.ID and TAX.ID to be encrypted)
 When entering a search string this should search both encrypted and unencrypted data in SOLR
and return the merged results
5
Searchable Database Architecture
Functional Architecture
 Core Banking System emits the data via Integration framework and push the data in to SOLR DB
via DataFramework ETL processing
 The data push to SOLR DB is Near Real Time with Event driven Solution
 The solution is completely followed by Metadata Management
 For any data search the search to be initiated from UXP Browser and the request comes to
SOLR DB via interaction Framework followed by DataFramework Processor Layer to
format/encrypt/etc. the request message before sending to SOLR DB
 The response will again formatting/decrypt done by DFP layer and send back to IRIS followed by
UXP Layer.
6
Technical Architecture
 The data is emitted through Integration Framework using Business events
 The emitted data is consumed by ETL using IF API to pull the data from Interface table
 After polling the data, the ETL process will do the necessary parsing, transformation and load the
data in to Searchable database (SOLR DB)
 During the ETL Process the data has to be stage in ETL Staging along with maintain the data in
to Metadata Schema.
7
End to End Architecture
Data Storage
 Data emitted from IF driven business events
 DF consume the IF Event via ETL Processing
 Consumed data will be transformed via ETL
 Load the data in to Solr Database
 Any encryption field will be stored in encrypted format
Data Search
 Search text request will send by UXPB to IRIS
IRIS Layer
 Determine Cores
 Invoke SMI API to identify the filter
 Send request to DFP with filter criteria
DFP
 Read property file to get ‘fl’ list
 Apply encryption
 Send request to Solr database
 Response will be decrypted and formatted based on definition and send back to IRIS
IRIS
 Receive response DFP and send back to UXPB
UXPB
 User click manually on dropdown data, browser automatically launches SCV Enquiry
8
System Requirement
The following are the pre-requisites for installing this package.
 Ensure the Windows Server 2017 Standard Edition operating system is installed.
 T24 Release based on Client Release
 JDK 1.7 and above

 MS SQL Server 2016
 JBoss 7.0 EAP
 Set the corresponding environment variables for the third party software. For example, set JAVA_HOME to
the Java install path
 Latest TAFJ/TAFC Pack with latest SP in line with T24 Release
 Searchable database Release Pack in line with T24 Release (DataFrameworkTooling-SEARCH-xx.x.x.zip)
from Distribution
 IF Packager.zip
 UXPB pack from Distribution (covers both IRIS 1.0 and Browser IRIS)
 Design Studio pack from Distribution
 The below list of updates are expected to be installed
S.NO Component Name Updates
1 EB_Sms Latest Update
2 AC_AccountOpening Latest Update
3 AC_AccountClosure Latest Update
4 AC_AccountingEventsService Latest Update
9
Runtime Configuration
Configuring TAFJ
Procedure
1. Make the following changes to the TAFJ properties file.

a. Change property tafj.home to point to the TAFJ folder.
b. Change property temn.tafj.directory.precompile to point to the folder
containing the precompiled .jar files.
c. Update the database setup section with the transactional and reporting database details.
temn.tafj.jdbc.url =
jdbc:sqlserver:://localohost:1493:databaseName="T24":integratedSecurity=false
temn.tafj.jdbc.driver = com.microsoft.sqlserver.jdbc.SQLServerDriver
temn.tafj.jdbc.username = T24
temn.tafj.jdbc.password = T24
temn.tafj.jdbc.ro.url = jdbc:sqlserver:://localohost:1493:databaseName=DIM
temn.tafj.jdbc.ro.driver = com.microsoft.sqlserver.jdbc.SQLServerDriver
temn.tafj.jdbc.ro.username = DIM
temn.tafj.jdbc.ro.password = DIM
d. In the TAFJ locking mechanism section, set temn.tafj.locking.mode to

MSQL.
2. Copy the SQL database driver from TAFJ_HOME\dbdrivers to TAFJ_HOME\ext.
10
Configuring JBoss
Copying the MSSQL database driver files
Procedure
1. Under JBOSS_HOME\modules\com\, create a new folder structure

microsoft\sqlserver\main.
2. Copy sqljdbc4.jar from TAFJ_HOME\dbdrivers\sqljdbc_4.0\enu to
JBOSS_HOME\modules\com\microsoft\sqlserver\main.
3. Copy module.xml from
TAFJ_HOME\appserver\jboss\jboss6eap\modules\com\microsoft\sqlserver
to JBOSS_HOME\modules\com\oracle\sqlserver\main.
4. In the module.xml file, in the resources tag, insert <resource-root
path="sqljdbc4.jar"/> and <resource-root path="sqljdbc4.jar"/>.
Preparing the main directory for T24
Procedure
1. Under JBOSS_HOME\modules\com\ create the following directories.

 JBOSS_HOME;\modules\com\temenos\t24\main.
 JBOSS_HOME\modules\com\temenos\t24\main.
2. Copy all the precompiled .jars to JBOSS_HOME\modules\com\temenos\t24\main.
3. Copy the T24 module.xml file from
TAFJ_HOME\appserver\jboss\jboss7eap\modules\com\temenos\t24 to
JBOSS_HOME\modules\com\temenos\t24\main folder.
4. Edit module.xml and update the resources tag with the names of the .jars in the main folder
under t24 (refer the TAFJ manual TAFJ-AS JBossInstall v6.4 EAP).
5. In module.xml update the dependencies section with <module
name="com.temenos.tafj" /> as illustrated below.
<dependencies>
<module name-"com.temenos.tafj">
</dependencies>
11
Preparing the main directory for TAFJ
Procedure
1. From TAFJ_HOME copy ext & lib folders to

JBOSS_HOME\modules\com\temenos\tafj\main.
2. Copy the TAFJ module.xml file from
TAFJ_HOME\appserver\jboss\jboss7eap\modules\com\temenos\tafj to
JBOSS_HOME\modules\com\temenos\tafj\main.
Do not change the resources section in this file.
3. In module.xml update the dependencies section with <module

name="com.microsoft.sqlserver"/>
Preparing the standalone-sql.xml file
Procedure
1. Copy standalone-full.xml from TAFJ_HOME\appserver\jboss\jboss6eap\ to

JBOSS_HOME\standalone\configuration\.
2. Rename this file to standalone-DFR.xml.
3. Add a <system properties> tag to this file.
4. Add or update section t24DS datasource. Set the T24DS datasource connections to point to
the T24 transactional database. For the datasource, set the jta attribute value to false.
12
5. Add or update the t24LockingDS datasource. Set the t24LockingDS datasource connections to
point to the T24 Transactional database. For the datasource, set the jta attribute value to false.
6. Add or update the t24Axis2DS datasource. Set the t24Axis2DS datasource connections to point
to the T24 Transactional database. For the datasource, set the jta attribute value to false.
7. Add or update the t24RODS datasource. Set the t24RODS datasource connections to point to the
T24 read only database. For the datasource, set the jta attribute value to false.
13
8. Add a relevant driver to connect to the databases. For example:
9. Save the file.
Preparing the war and ear files
Procedure
Copy the files listed below to JBOSS_HOME\standalone\deployments.
File name File to use

axis2.war Use the correct axis2.war file for your version of Reporting Runtime.
TAFJJEE_EAR.ear Obtain this file from TAFJ_HOME\appserver.
14
ETL Configuration
Prerequisites
Before start the ETL configuration make sure the below pre-requisites are in place.
 Create ETL database
 Create CONFIG and MDATA schema to holds the ETL configuration and Metadata information for
searchable model
 Create User creation in ETL database
In ETL configuration, follow the below steps in a given sequencing order
Configuring Data Framework for the Extract phase on JBoss

Number of actions to be performed to configure Data Framework for the Extract phase on JBoss.
Prerequisites
Before configuring the Extract Phase on JBoss, ensure the following are in place.
 The IF Event table has promoted columns and is indexed.

 Scripts have been run to create the table, view and stored procedure for Extract Jobs creation.
 The ETL tables have been created and their SQLs have been run.
 A copy of t24-dataframework-etl-extract-ear-xx.x.x.ear is available.
Creating and indexing promoted columns
Execute the statements described below. In each case, change the IF table name as per your target
environment. If your target environment has more than one company, then you will need to amend and
run the script for each relevant company.
Procedure
1. Run EVENT_TYPE:
CREATE OR REPLACE FUNCTION FN_IF_EVENTS_C2 (XMLRECORD SYS.XMLTYPE) RETURN VARCHAR2 DETERMINISTIC AS RTNVAL
VARCHAR2(100); BEGIN SELECT XMLCast(XMLQuery('/row/c2/text()' PASSING XMLRECORD returning content) as
VARCHAR2(100)) into rtnval FROM DUAL; return rtnval; END;
ALTER TABLE FBNK_IF_EVENTS_INTE000
ADD EVENT_TYPE VARCHAR2(100) GENERATED ALWAYS AS (CAST(FN_IF_EVENTS_C2(XMLRECORD) AS VARCHAR2(100
15
2. Run CREATION_TIME:
CREATE OR REPLACE FUNCTION FN_IF_CREATIONTIME_C11 (XMLRECORD SYS.XMLTYPE) RETURN VARCHAR2 DETERMINISTIC AS
RTNVAL VARCHAR2(100); BEGIN SELECT XMLCast(XMLQuery('/row/c11/text()' PASSING XMLRECORD returning content)
as VARCHAR2(100)) into rtnval FROM DUAL; return rtnval; END;
ADD CREATION_TIME VARCHAR2(100) GENERATED ALWAYS AS (CAST(FN_IF_CREATIONTIME_C11(XMLRECORD) AS

VARCHAR2(100)));
3. Run CREATION OF COMPOSITE INDEX:

CREATE INDEX IDX_IF_EVENTS_INTER_C2_C11 ON FBNK_IF_EVENTS_INTE000(CREATION_TIME,EVENT_TYPE);
4. Run DELIVERY_STATUS:
CREATE OR REPLACE FUNCTION FN_IF_DELSTATUS_C6 (XMLRECORD SYS.XMLTYPE) RETURN VARCHAR2 DETERMINISTIC AS
RTNVAL VARCHAR2(100); BEGIN SELECT XMLCast(XMLQuery('/row/c6/text()' PASSING XMLRECORD returning content)
as VARCHAR2(100)) into rtnval FROM DUAL; return rtnval; END;
ADD DELIVERY_STATUS VARCHAR2(100) GENERATED ALWAYS AS (CAST(FN_IF_DELSTATUS_C6(XMLRECORD) AS

VARCHAR2(100)));
CREATE INDEX IDX_IF_DELSTATUS_INTER_C6 ON FBNK_IF_EVENTS_INTE000(DELIVERY_STATUS,1);
5. Run DROP & RECREATE VIEW:

DROP VIEW V_FBNK_IF_EVENTS_INTE000;
CREATE OR REPLACE VIEW V_FBNK_IF_EVENTS_INTE000 AS
SELECT a.RECID, a.XMLRECORD "THE_RECORD"
,a.RECID "EVENT_ID"
,extractValue(a.XMLRECORD,'/row/c1[position()=1]') "EVENT_DATA"
,EVENT_TYPE "EVENT_TYPE"
,extractValue(a.XMLRECORD,'/row/c3[position()=1]') "EVENT_TIMESTAMP"
,extractValue(a.XMLRECORD,'/row/c4[position()=1]') "OLD_EVENT_XML"
,extractValue(a.XMLRECORD,'/row/c5[position()=1]') "LOCAL_TRANSFORM"
,DELIVERY_STATUS "DELIVERY_STATUS"
,extractValue(a.XMLRECORD,'/row/c7[position()=1]') "STATUS_REASON"
,extractValue(a.XMLRECORD,'/row/c8[position()=1]') "EVENT_XML_NAME"
,extract(a.XMLRECORD,'/row/c8') "EVENT_XML_NAME_8"
,extractValue(a.XMLRECORD,'/row/c9[position()=1]') "OLD_FIELD_VALUE"
,extract(a.XMLRECORD,'/row/c9') "OLD_FIELD_VALUE_9"
,extractValue(a.XMLRECORD,'/row/c10[position()=1]') "FIELD_VALUE"
,extract(a.XMLRECORD,'/row/c10') "FIELD_VALUE_10"
,CREATION_TIME "CREATION_TIME"
,extractValue(a.XMLRECORD,'/row/c12[position()=1]') "CREATION_DATE"
,extractValue(a.XMLRECORD,'/row/c13[position()=1]') "RESERVED21"
,extractValue(a.XMLRECORD,'/row/c15[position()=1]') "COMMON_XML_NAME"
,extract(a.XMLRECORD,'/row/c15') "COMMON_XML_NAME_15"
,extractValue(a.XMLRECORD,'/row/c16[position()=1]') "COMMON_XML_VAL"
16
,extract(a.XMLRECORD,'/row/c16') "COMMON_XML_VAL_16"
,extractValue(a.XMLRECORD,'/row/c17[position()=1]') "SOURCE"
,extractValue(a.XMLRECORD,'/row/c24[position()=1]') "RESERVER32"
,extractValue(a.XMLRECORD,'/row/c32[position()=1]') "OVERRIDE"
,extract(a.XMLRECORD,'/row/c32') "OVERRIDE_32"
,extractValue(a.XMLRECORD,'/row/c33[position()=1]') "RECORD_STATUS"
,extractValue(a.XMLRECORD,'/row/c34[position()=1]') "CURR_NO"
,extractValue(a.XMLRECORD,'/row/c35[position()=1]') "INPUTTER"
,extract(a.XMLRECORD,'/row/c35') "INPUTTER_35"
,extractValue(a.XMLRECORD,'/row/c36[position()=1]') "DATE_TIME"
,extract(a.XMLRECORD,'/row/c36') "DATE_TIME_36"
,extractValue(a.XMLRECORD,'/row/c37[position()=1]') "AUTHORISER"
,extractValue(a.XMLRECORD,'/row/c38[position()=1]') "CO_CODE"
,extractValue(a.XMLRECORD,'/row/c39[position()=1]') "DEPT_CODE"
,extractValue(a.XMLRECORD,'/row/c40[position()=1]') "AUDITOR_CODE"
,extractValue(a.XMLRECORD,'/row/c41[position()=1]') "AUDIT_DATE_TIME"
FROM
"FBNK_IF_EVENTS_INTE000" a;
Running a script to create tables, views & stored procedures for Extract job
creation
Execute the statement described below. In each case, change the IF table name as per your target
environment. If your target environment has more than one company, then you will need to amend and
run the script for each relevant company.
17
Procedure
Locate the script database_schema/search/sql/server/5_etl_create_proc.sql in the DF distribution pack
and run it.
--This file is used to create SP for creating jobs in Dispatcher. Following steps is done here
--1. Create table as df_extract_job_list for holding job_id, start_time, end_time, row_count
--2. Create a view df_extract_job_view and promoted column for IF_EVENT_INTERFACE TABLE (applicable for all
companies)
--3. Create the stored procedure sp_df_extract_job_list for which the inputs are eventCount, ifFromTime,
ifToTime and optional rowCount (0 or 1)
--4. Update the T24 Database name and IF EVENT INTERFACE TABLE names mentioned in view before running the
scripts.
USE <T24 database name>

GO
--DROP TABLE df_extract_job_list
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
SET ANSI_PADDING ON
GO
CREATE TABLE [dbo].[df_extract_job_list](

[job_id] [varchar](50) NOT NULL,
[start_time] [varchar](50) NOT NULL,
[end_time] [varchar](50) NOT NULL
) ON [PRIMARY]
GO
SET ANSI_PADDING OFF

GO
--Create a VIEW df_extract_job_view for IF EVENT Table to extract Creation_time and event_type for all DF
related events
CREATE VIEW df_extract_job_view (creation_time, event_type)
AS
SELECT CREATION_TIME AS creation_time, EVENT_TYPE AS event_type FROM FBNK_IF_EVENTS_INTE000
--SELECT creation_time, event_type FROM (
--SELECT creation_time, event_type FROM FAE1_IF_E000 UNION ALL
--SELECT creation_time, event_type FROM FBNK_IF_E000 UNION ALL
--SELECT creation_time, event_type FROM FCN1_IF_E000 UNION ALL
--SELECT creation_time, event_type FROM FGB2_IF_E000 UNION ALL
--SELECT creation_time, event_type FROM FSG1_IF_E000 UNION ALL
--SELECT creation_time, event_type FROM FIN1_IF_E000 UNION ALL
--SELECT creation_time, event_type FROM FUS1_IF_E000 UNION ALL
--SELECT creation_time, event_type FROM FHK2_IF_E000 UNION ALL
--SELECT creation_time, event_type FROM FDE1_IF_E000 UNION ALL
--SELECT creation_time, event_type FROM FBE1_IF_E000 UNION ALL
--SELECT creation_time, event_type FROM FGB1_IF_E000 UNION ALL
--SELECT creation_time, event_type FROM FLU1_IF_E000 UNION ALL
--SELECT creation_time, event_type FROM FHK1_IF_E000 )
WHERE EVENT_TYPE IN (
'FinancialReporting-Language','Searchable-SrchCustomer','SearchableService-SrchCustAccount');
/****** Object: StoredProcedure [dbo].[sp_df_extract_job_list_demo] Script Date: 10/11/2017 7:32:51 PM

******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
CREATE PROC [dbo].[sp_df_extract_job_list]

@p_event_count VARCHAR(100), --Event count to fetch each job
@p_from_date VARCHAR(100), --From time for the time range
@p_to_date VARCHAR(100), --To time for the time range
18
@p_get_count VARCHAR(100) --Default parameter to define 0/1, 0-not to get count for each time range, 1-get
count for each time range
AS
/*
-----------------------------------------------------------------------------------
Name : [df_extract_job_list].Proc.Sql
Purpose : To get the final resultant set Job_Id, start_creation_time, end_creation_time Aanf the
no.of eventids
Execution Example : EXEC [dbo].[sp_df_extract_job_list] 100 ,'1507117308.809' ,'1507117910.217' ,1

-----------------------------------------------------------------------------------
* Change History *
DATE Alias Release Description
*/
BEGIN TRY
SET NOCOUNT ON
/* declare variables */
DECLARE @v_sql VARCHAR(8000)
DECLARE @v_rowcount VARCHAR(50)
DECLARE @tempTable TABLE (EVENTCOUNT int)

INSERT INTO @tempTable
SELECT @p_event_count
DECLARE @TABLECOUNT VARCHAR(50)

SET @TABLECOUNT = ( SELECT COUNT(1) FROM df_extract_job_list )
IF @TABLECOUNT > 0
TRUNCATE TABLE df_extract_job_list
IF OBJECT_ID('TEMPDB..#EVENTCOUNT') IS NOT NULL

DROP TABLE TEMPDB..#EVENTCOUNT
CREATE TABLE TEMPDB..#EVENTCOUNT ( CNT INT NOT NULL)

INSERT INTO #EVENTCOUNT
SELECT @p_event_count
SET @p_from_date= CHAR(39) + @p_from_date +CHAR(39)
SET @v_sql = 'INSERT INTO df_extract_job_list

SELECT job_id, start_creation_time, end_creation_time
FROM
(SELECT job_id, creation_time start_creation_time, COALESCE(LEAD(creation_time,1)
OVER (ORDER BY creation_time ASC),'
+ @p_to_date + ')end_creation_time FROM
(SELECT creation_time, (rn/trc.CNT)+1 job_id FROM
(SELECT rn, COALESCE(LAG(creation_time,1) OVER (ORDER BY creation_time
ASC),'
+ @p_from_date + ')creation_time FROM
(SELECT creation_time, ROW_NUMBER() OVER (ORDER BY
creation_time ASC)-1 rn
FROM df_extract_job_view WHERE creation_time >= '
+ @p_from_date + ' AND creation_time <= ' + @p_to_date + '
) x
) x CROSS JOIN #EVENTCOUNT trc WHERE rn%trc.CNT = 0
)A
) x
WHERE start_creation_time != end_creation_time';
EXEC (@v_sql);
IF @p_get_count > 0
SET @v_rowcount =( SELECT COUNT(*) r

FROM df_extract_job_view
WHERE creation_time > @p_from_date
AND creation_time <= @p_to_date )
ELSE SET @v_rowcount = 0
END TRY
BEGIN CATCH
DECLARE
@ErrorMessage nvarchar(400)
,@ErrorSeverity int
19
,@ErrorState int
,@ErrorNumber int
,@ErrorProcedure nvarchar(256)
SELECT
@ErrorMessage = ERROR_MESSAGE()
,@ErrorSeverity = ERROR_SEVERITY()
,@ErrorState = ERROR_STATE()
,@ErrorNumber = ERROR_NUMBER()
,@ErrorProcedure = ERROR_PROCEDURE()
RAISERROR (@ErrorMessage,@ErrorSeverity,@ErrorState);
END CATCH
GO
Configuring ETL tables in the reporting database
Procedure
1. Configuration parameters used by Extract and TL must be loaded into the reporting database
using insert queries.
2. To do this, find the queries from location database_schema/search/sql/server in the DF distribution
pack and run the scripts in sequence
3. Store the metadata password in encrypted form.
4. To generate password in encryped form, follow the below steps (Refer Appendix)
a) From DataFramworkTooling-SEARCH-19.x.x, unzip t24-dataframework-mapper-19.0.3 utility.

b) Run the batch file run.bat
c) Choose option1
d) Enter the password.
e) Find the encrypted password as the output.
5. Run the script against TMN_DF_CONF_TL to set the encrypted password.
UPDATE TMN_DF_CONF_TL SET PARAM_VAL='QDNeC2zbTfguaGhot7R2qg==' WHERE

PARAM_NAME='tmn.df.tl.metadata.db.password';
Copying Eson output files
Eson output files in JBoss must be copied to a defined location so they can be referenced by ETL.
Procedure
1. In the JBoss standalone .xml define parameter tmn.df.data.mapper.output.path.

2. Copy the Eson output .xml files to this path.
20
Configuring and deploying the Data Framework Extract
Procedure
1. Create a standalone-extract folder for Extract.

2. Copy the existing JBoss standalone folder to this new folder standalone-extract. This folder will
contain the configuration and deployment files for the Extract.
3. In the configuration folder rename standalone.xml to standalone-extract.xml. Do not
change the other files.
4. Add or update the etlDS datasource. Ensure the etlDS connections point to the reporting
database.
5. Copy file t24-dataframework-etl-extract-ear-XX.X.X.ear from the DF distribution

pack to the deployment folder.
6. Create a new .bat file for the JBoss Extract start-up.

7. Copy launch_Jboss.sh to a new standalone file launch_Jboss_extract.sh. This file
must point to --server-config=standalone-extract.xml (the new XML file created
for the Extract).
21
8. Add the following parameters to launch_Jboss_extract.sh.

-Ddf.etl.extract.app.name1=xyz Any meaningful name for the Extract
application. If multiple instances of
Extract servers are running, then the
names have to be unique for each.
-jboss.server.base.dir=../standalone-extract Standalone folder used by the Extract

application process
-Djboss.server.log.dir=../standalone-extract/log Logs folder to be created and used by

the Extract application process
-Dtmn.df.data.mapper.output.path= Folder path to data mapper output files.

C:\Temenos\dfmapperfiles This is place where the Eson output xmls
are placed.
Configuring Data Framework for Transform and Load

A number of steps to be performed to configure Data Framework for the Transform and Load phases
Procedure
1. Define a property file to specify metadata schema name
2. Create a standalone-tl folder for TL (Transform and Load).

3. Copy the existing JBoss standalone folder to this new folder standalone-tl. This folder will contain
the configuration and deployment files for TL.
4. In the configuration folder rename standalone.xml to standalone-tl.xml. Do not change the
other files.
5. Add or update the etlDS datasource. Ensure the etlDS connections point to the reporting
database.
22
6. Add or update the jdbc/metaDatasource. Ensure the metaDS connections point to the reporting
database
7. Add three new tags, along with the existing <bean-instance-pools> definitions, in the pools
section in the file.
<strict-max-pool name="df-tl-transform-new-strict-max-pool" max-pool-size="10"
instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/>
<strict-max-pool name="df-tl-transform-abort-strict-max-pool" max-pool-size="5"

instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/>
<strict-max-pool name="df-tl-load-strict-max-pool" max-pool-size="10" instance-

acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/>
23
8. Remove all the files inside the deployment folder. Copy file t24-dataframework-etl-TL-
ear-XX.X.X.ear from the DF distribution pack to the deployment folder.
9. Create a new .bat file for the JBoss TL start-up.

10. Copy launch_Jboss.sh to a new standalone file launch_Jboss_tl.sh. This file must
point to --server-config=standalone-tl.xml (the new XML file created for the TL).
11. Add the following configurations to launch_Jboss_tl.sh.

-Dtmn.df.tl.app.name=abc Any unique node name.
-DJboss.server.base.dir=../standalone-tl A meaningful name for the TL

application. If you are running
multiple TL servers, the names have to
be unique for each.
-DJboss.server.log.dir=../standalone-tl/log A log folder to be created and used by

the TL application process.
-Dtmn.df.data.mapper.output.path=C:\Temenos\df_mapper_files The path to the eson output files.

This is where you saved the xmls
generated by the esons.
-Dtmn.df.metadata.schema.config= The property file path where metadata

C:\Temenos\dfmapperfiles\schema.properties schema name is defined
24
Artefacts Generation
A set of DDL and XML artefacts must be generated from ESON Models.
Generation of Artefacts
Procedure
1. Make sure the t24-DF_DfMappingService-tafj-jws.aar (part of temp

release UD\Extension folder) file is in the services folder of the axis2.war.
2. Deploy axis2.war to the application server

3. Deploy Browser war and Authenticator war to JBoss standalone deployment folder
25
4. Make sure you can access the T24 Browser.
5. Make sure that DfMappingService and wsdl are accessible on link

http://localhost:9089/axis2/services/DfMappingServiceWS?wsdl.
6. Change the server name and port name to access the DfMappingService in the application server
where it is deployed.
7. Copy commons-dbcp jar from EDS folder from \\EDS-T24-2019.x.x\t24-binaries\commons-

dbcp\commons-dbcp\1.2.2 to plugins\com.odcgroup.dataframework.generation_XXXX.X.X.jar\lib
folder
26
8. Launch EDS.
9. Unzip DataFrameworkTooling-Model-SEARCH-<Release_Version>.zip folder which has df-
search-models and df-search-models-gen folders as a zip folder.
10. Import of df-search-models and df-search-models-gen as a maven project into EDS.
11. Df-search-models project holds the eson files

12. Df-search-models-gen folder holds the generated artefacts from DesignStudio
13. Below is the structure of df-search-models
27
14. DataMappingConfig.eson is the configuration file.

This holds the configuration parameters such as T24Service provider, metadata database configuration and other
Searchable product configurations.
 First set of configuration defines the connection details to T24 server
 Second set of configuration defines the metadata database connection details and schema name
 Third set of configuration defines the Solr version and Solr artefacts generation folder (DO NOT CHANGE
IT)
 validateModel config is for validating the eson files
 searchMetadaOtherBookAccess config is to define whether the search data should be retrieved either
from User login company or including other books of same lead company. Refer Other Configuration
Section for details.
28
17. How to generate Artefacts

 To generate the code, right click on DataFramework.eson and select Design Studio -> ‘Generate Code’. It
performs 2 actions.
 Generation of Solr artefacts
 Load Solr Metadata into Metadata tables
 Right click on df-search-models, select properties and ensure that T24DataFramework cartridge is
enabled.
29
 Generate Solr Artefacts by right click on DataFramework.eson  Design Studio  Generate Code
 The artefacts will be generated in df-search-models-gen under src/generated/dataframework
2. 18. Metadata Load from DS

Generation of Solr artefacts loads the “DB” product schema to metadata database.
Execute the below query against DIM database and ensure that “DB” product schema are loaded into the
database.
30
31
Solr Server Configuration

To configure Solr Sever you need to follow the below steps.
Solr 7.5 download

Download Solr zip file from https://archive.apache.org/dist/lucene/solr/7.5.0
Set JAVA_HOME
Set JAVA_HOME in which the library jars required for AES256 encryption algorithm is being set. Refer Other
Configuration section.
How to start Solr Server
Solr server setup

This involves three steps.
1. Placement of generated artefacts in Solr server
2. Uploading of IRIS xml to IRIS core
3. Placement of DFR jars in Solr library files
Placement of generated artefacts in Solr server
Unzip the SEARCH_SOLR_SCHEMA_CREATE_7.5 folder
Place generated artefacts in Solr server
32
Uploading of IRIS xml to IRIS core

 Steps to upload IRIS xml to IRIS core
 Generated IRIS xml
 Content of IRIS.xml
 Solr should be started before uploading IRIS document to IRIS core
 API’s in post.jar is used to upload the document to core
 Go to the folder where post.jar is present in Solr server
33
 Use the command,

java -Dc=IRIS -jar post.jar C:\solr-7.5.0\server\solr\IRIS\Doc\IRIS.xml
to upload the document to IRIS core where –Dc is the Core name
 Verify in Solr admin console if the document is uploaded succesfully to IRIS core.
Placement of DFR jars in Solr library files

 From the distribution pack,
34
Copy the jar files t24-dataframework-solrext-XXX and t24-dataframework-solr-encrypt-XXXX in Solr server

in the path specified.
How to Stop Solr Server
35
UXPB Configuration
In UXPB Browser, SEARCH GLASS will be displayed to perform SEARCH operations. To get the search glass in UXP
Browser, the property ENABLE_SOLR should be enabled in setenv.bat.
36
SMS Configuration
User profiles have to be defined with SMS restriction to enable the search result with filter conditions.
Case 1: Define a USER profile with a filter condition SECTOR = 1000
Login as TESTUSER1 and SEARCH.
Search result will bring only the records with SECTOR = 1001 i.e Individual
37
Case 2: Define a USER profile with a filter condition SECTOR NE 1001
Search result will bring only the records with SECTOR NE 1001 i.e Individual
38
39
Deployment
Deployment Packs
Below are the set of distribution packs of Searchable Database.
 Design Studio 19.7
 IRIS1.0 19.0.4
 UXP Browser 19.0.5
 Browser-Iris 19.0.4
 UXP Generated R19.5
 SearchableModel_IFPackager_19.0.3
 DataFrameworkTooling-SEARCH-19.0.3.zip
 t24-DF_DfMappingService-tafj-jws.aar
IF Package Installer
A number of steps to be performed for Package Installation.
Procedure
1. Modify the property in TAFj properties
temn.tafj.runtime.ud.exception.list=F.DL.DATA;DW.EXPORT;&HOLD&;&COMO&;package;REL.R19_IF_Searc
hable_1.IF
2. Package Folder Creation:
3. Copy the jar file into Package Folder
40
4. Set T24_HOME=C:\Sharanya\R19\Temenos\TAFJ\UD\
5. Run T24PackageInstaller
a) In case if faced any issues like below screenshot, Verify JAVA_HOME & PATH is set correctly.
b) If you face any issues like below, set T24_HOME
c) Successful Execution
6. Check the record 201904_IF_Searchable_1 created in T24.MODEL.PACKAGES for DATA.RELEASED field is

set to ‘N’ initially
41
7. If Deployment not occurred successfully in MSSQL.
SELECT F.T24.MODEL.PACKAGES WITH DATA.RELEASED EQ "N" BY RESTD.DATE

Error Message: Execution of user code in the .NET Framework is disabled. Enable "clr enabled" configuration
option.
Solution:
sp_configure 'clr enable', 1
go
RECONFIGURE WITH OVERRIDE
Go
Since DATA record is released it is marked as Y. In previous section, DATA record is not released and hence it was
marked as N.
42
8. Check the IF.EXIT.PONTS and IF.INTEGRATION.FLOW.CATALOG to ensure that events are published
successfully.
43
DataFrameworkTooling-SEARCH Distribution pack

It holds the list of deployment ears, library jars, zip files and database scripts for setting up the Searchable
database from DFR end.
Deployment of Extract
Procedure
1. Start the Extract process by invoking the launch_Jboss_extract.sh file.
When the process is running, you should see the .ear file being deployed. You should also see
the JBoss status, as illustrated below.
2. Check the deployment folder. You should see the .ear files.
44
Deployment of TL
1. Prerequisite: Solr server should be up and running
2. From the DF Distribution Pack, copy the TL ear and place it in the standalone-TL/deployments folder.
3. Start the TL process by invoking the launch_Jboss_tl.sh file.
When the process is running, you should see the .ear file being deployed. You should also see
the JBoss status, as illustrated below.
45
Initial Load Process
The IF.INITIAL.LOAD SERVICE retrieves records from T24 application and generates as many events and loads
them in to the T24 event table called FXXX.IF.EVENTS.INTERFACE.TABLE, where XXX denotes Company Mnemonic.
APPLICATION - INITIAL LOAD PROCESS

DataFrameWork Model Applications- Initial Load Processing
SEARCH CUSTOMER
Configuring the IF.INITIAL.LOAD.SERVICE

Follow the below sequence of steps to configure & run the IF.INITIAL.LOAD.SERVICE in the target environment.
1. Edit BATCH>BNK/IF.INITIAL.LOAD.SERVICE to add the required application names in field Data for which
the initial load events are to be generated.
2. Start TSA.SERVICE ->TSM
46
3. Start TSA.SERVICE -> BNK/IF.INITIAL.LOAD.SERVICE
4. Increase the number of agents TSA.WORKLOAD.PROFILE -> BNK/IF.INITIAL.LOAD.SERVICE
47
5. Run TSM - tRun START.TSM –DEBUG

For TAFJ, from {TAFJ_HOME}/bin
tRun START.TSM –DEBUG
For TAFC, on jsh

START.TSM –DEBUG
6. Run TSA
tRun tSA {agent number}
For TAFC, on jsh

tSA {agent number}
48
7. Edit BATCH>CUSTOMER.ACCOUNT.TAKEON to add the required TAKEON name in field Job Name for
which the initial load events are to be generated.
8. Start TSA.SERVICE ->TSM
49
9. Start TSA.SERVICE -> CUSTOMER.ACCOUNT.TAKEON
10. Run TSM

tRun START.TSM –DEBUG
For TAFC, on jsh
50
START.TSM –DEBUG
11. Run TSA

tRun tSA {agent number}
For TAFC, on jsh

tSA {agent number}
Verification After Data Load during ILP PROCESS
EVENT_TYPE & COUNT should match between IF Interfaces table under T24 & ETL Tables under ETL
database\schema.
NOTE: IF Interfaces table might have events generated from T24 application which does not belong to DF
Application, and those events should be ignored while checking the results.
Execute the below query against T24 database to check the event count for ILP.
SELECT EVENT_TYPE, COUNT(*) FROM V_FBNK_IF_EVENTS_INTE000
GROUP BY EVENT_TYPE
ORDER BY EVENT_TYPE;
Example:
Execute the below query against ETL database to check the event count for ILP.
SELECT EVENT_TYPE, COUNT(*) FROM TMN_DF_EVENT
WHERE EVENT_ID IN(SELECT EVENT_ID FROM TMN_DF_EVENT_LOAD_STATUS WHERE ERROR_TYPE_ID IS
NULL)
GROUP BY EVENT_TYPE
ORDER BY EVENT_TYPE;
Example:
51
To make sure that the Initial Load service executed successfully, check the Count for the above two queries they
should match. If the count does not match then execute the following query to check if any records went in to
error while loading, the result of the query shows you the failed records with the reason for the failure.
Execute the below query against ETL database.

SELECT * FROM TMN_DF_EVENT_LOAD_STATUS WHERE ERROR_TYPE_ID>0;
52
Data Search
Display Content in the drop down
Format of the search result shown in the drop down box in UXPB
First line – Short.name Name.1 Name.2
2nd line – Street, Address, Town.Country, Post.code
3rd line – Email Address
4th line – phone number
5th line – Account number. This is ALT.ACCT.ID account reference part last 4 characters (e.g. for
DDA123456, ATL342342FAAFAS would be show as 3456 and AFAS).
6th line – TaxId
Sample screen
53
CUSTOMER 360 degree view with drill down
Search is made on Plain text as well as Encrypted Text

In Searchable database, a search is made on both the given text as well as the encrypted search text.
Here, for the given search text “444”, a search is made on q=444 OR encrypted value of 444 which brings he
mapping documents from Searchable database.
54
Encryption and Masking

Search on AltAcctId
55
Search on masked AltAcctId
Search on TaxId
56
Search on masked TaxId
57
How to add new Local Reference & Modify fl list?

Searchable database provides an ability to support additional data using client specific local reference fields.
A number of steps to be performed for adding a LocalRef field in the filter list.
Procedure
1. Add a new local ref field in the LOCAL.TABLE application
LOCAL.TABLE I LOCALFIELD and LOCAL.TABLE A LOCALFIELD
2. Add the created local ref field to the customer application by means of the LOCAL.REF.TABLE application
LOCAL.REF.TABLE I CUSTOMER and then authorize the record using the command LOCAL.REF.TABLE A CUSTOMER
3. Create a Searchable Project and import the existing Events & Flows
4. Include the Local Fields to the project.
5. Then do a generate code for the updated Project.
58
6. On Click Next
7. Then Click Finish
59
8. Add the New Field in Flows
9. Do Publish Event & Ensure the events published successfully.
60
10. Now add the attributes and attribute mapping in the Customer.eson
11. Then do a Generate Code on DataFramework.eson and verify the new Solr cores & its related artefacts
are generated and metadata loaded successfully.
12. Stop the Solr Server if Running
61
13. Copy the SearchCustomer-T24Field.properties to %SOLR_HOME%\server\solr\GB0010001\Customer

14. Go to the path %SOLR_HOME%\server\solr\GB0010001\Customer\conf
15. Delete the Existing managed Schema & Copy the t24-df-fields. Rename the .bak file to schema.xml file
16. Start the Solr Server & verify if any validation errors in Core.
17. Restart the Appserver if running. ( For refresh of Metadata)
18. Input a transaction for the Local Field and verify the data via Solr Search
62
Other Configuration
OtherBookAccess
SOLR Core will be created by Company as per T24 Structure. For example, CUSTOMER is of type ‘CUS’, customer
data will be available in GB0010001 as T24 source data will be in BNK
Based on settings in property searchMetadataOtherBookAccess data will be retrieved either from User login
company or including other books of same lead company
searchMetadataOtherBookAccess is enabled
This is the outcome of T24Company-SolrCore.properties.
63
searchMetadataOtherBookAccess is disabled
This is the outcome of T24Company-SolrCore.properties
Row Size
Number of records to be displayed in the drop down box in UXPB is controlled by the rows attribute in solrConfig
file.
64
Default Solr Configuration parameters are defined in -Dsolr.default.confdir=C:\solr-

7.5.0\server\solr\configsets\_default\conf and the default Row size is 10.
This default conf value can be overriden at each CORE level. Rows count is configured as 50 in
solr/GB0010001/Customer/conf/solrConfig.xml.
Rows count is configured as 10.
65
AES256 Encryption Algorithm

On a default JDK installation, AES is limited to 128 bit key size. In order to perform 256-bit AES encryption, hence
Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files need to be downloaded and
installed.
Here are the steps.
1. Go to Oracle’s website and search for ‘Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy
Files’.
2. Depending upon the Java version installed, download the zip file and extract it on the drive.
3. From the extracted folder, copy local_policy.jar and US_export_policy.jar files.
4. Go to the_java_installation_directory/jre/lib/security and paste the copied files. These files will already be
there, just need to copy and replace.
5. Refresh the project
66
APPENDIX
How to generate password in encryped form?
1. From DataFramworkTooling-SEARCH-19.x.x, unzip t24-dataframework-mapper-19.0.3 utility.
2. Run the batch file run.bat
3. Choose option1
4. Enter the password.
5. Find the encrypted password as the output.
67

DataFrameworkTooling SEARCH InstallationGuide

Загружено:

Сведения о документе

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

DataFrameworkTooling SEARCH InstallationGuide

Загружено:

Авторское право:

Доступные форматы

Data Framework

Searchable Database (SOLR)

Verification After Data Load during ILP PROCESS ...............................................................................................51

Sivaprakash C 1.0 24-Jun-19 Initial Draft

Searchable database provides-

Searchable Database Architecture

 The solution is completely followed by Metadata Management

 The data is emitted through Integration Framework using Business events

End to End Architecture

 T24 Release based on Client Release

 JDK 1.7 and above

S.NO Component Name Updates

1 EB_Sms Latest Update

2 AC_AccountOpening Latest Update

3 AC_AccountClosure Latest Update

4 AC_AccountingEventsService Latest Update

1. Make the following changes to the TAFJ properties file.

d. In the TAFJ locking mechanism section, set temn.tafj.locking.mode to

Copying the MSSQL database driver files

1. Under JBOSS_HOME\modules\com\, create a new folder structure

Preparing the main directory for T24

1. Under JBOSS_HOME\modules\com\ create the following directories.

Preparing the main directory for TAFJ

1. From TAFJ_HOME copy ext & lib folders to

Do not change the resources section in this file.

3. In module.xml update the dependencies section with <module

Preparing the standalone-sql.xml file

1. Copy standalone-full.xml from TAFJ_HOME\appserver\jboss\jboss6eap\ to

8. Add a relevant driver to connect to the databases. For example:

9. Save the file.

Preparing the war and ear files

Copy the files listed below to JBOSS_HOME\standalone\deployments.

File name File to use

In ETL configuration, follow the below steps in a given sequencing order

Configuring Data Framework for the Extract phase on JBoss

 The IF Event table has promoted columns and is indexed.

Creating and indexing promoted columns

ALTER TABLE FBNK_IF_EVENTS_INTE000

ADD EVENT_TYPE VARCHAR2(100) GENERATED ALWAYS AS (CAST(FN_IF_EVENTS_C2(XMLRECORD) AS VARCHAR2(100

ALTER TABLE FBNK_IF_EVENTS_INTE000

ADD CREATION_TIME VARCHAR2(100) GENERATED ALWAYS AS (CAST(FN_IF_CREATIONTIME_C11(XMLRECORD) AS

3. Run CREATION OF COMPOSITE INDEX:

ALTER TABLE FBNK_IF_EVENTS_INTE000

ADD DELIVERY_STATUS VARCHAR2(100) GENERATED ALWAYS AS (CAST(FN_IF_DELSTATUS_C6(XMLRECORD) AS

CREATE INDEX IDX_IF_DELSTATUS_INTER_C6 ON FBNK_IF_EVENTS_INTE000(DELIVERY_STATUS,1);

5. Run DROP & RECREATE VIEW:

CREATE OR REPLACE VIEW V_FBNK_IF_EVENTS_INTE000 AS

SELECT a.RECID, a.XMLRECORD "THE_RECORD"

USE <T24 database name>

--DROP TABLE df_extract_job_list

CREATE TABLE [dbo].[df_extract_job_list](

SET ANSI_PADDING OFF

/****** Object: StoredProcedure [dbo].[sp_df_extract_job_list_demo] Script Date: 10/11/2017 7:32:51 PM

CREATE PROC [dbo].[sp_df_extract_job_list]

Execution Example : EXEC [dbo].[sp_df_extract_job_list] 100 ,'1507117308.809' ,'1507117910.217' ,1

DECLARE @tempTable TABLE (EVENTCOUNT int)

DECLARE @TABLECOUNT VARCHAR(50)

TRUNCATE TABLE df_extract_job_list

IF OBJECT_ID('TEMPDB..#EVENTCOUNT') IS NOT NULL

CREATE TABLE TEMPDB..#EVENTCOUNT ( CNT INT NOT NULL)

SET @p_from_date= CHAR(39) + @p_from_date +CHAR(39)

SET @v_sql = 'INSERT INTO df_extract_job_list

SET @v_rowcount =( SELECT COUNT(*) r

ELSE SET @v_rowcount = 0