Вы находитесь на странице: 1из 24
Ultimatix - SAP HANA Implementation TCS Confidential Technical Review Report 06-Jan-2017 to 5-Feb-2017 By -

Ultimatix - SAP HANA Implementation

TCS Confidential

Technical Review Report 06-Jan-2017 to 5-Feb-2017

By - EntSol SAP CoE

TCS Confidential

Ultimatix SAP HANA Implementation

Technical Review

Vers. No.

Date

Author

Status

Comment

0.1

30.Jan.2017

Mayank Mishra

Draft

Initial version of Tech Review Report

0.2

9.Feb.2017

Dhunjishaw Sunavala

Draft

Comments from sizing are included

1.0

10.Feb.2017

Mayank Mishra

Final

Report Finalization after CoE Internal review

Ultimatix SAP HANA Implementation

Technical Review

Table of Content

1.

Purpose of the document

5

2.

Review Focus Areas (Requested by Ultimatix Team)

5

3.

Review

Pre-requisites

5

4.

Review

Activities

5

5.

Current State of Infrastructure

6

6.

Review Findings & Recommendations

9

6.1.

Absence of real requirements

9

6.2.

Incorrect understanding of operational reporting needs

9

6.3.

Review Team: Inadequate staffing and conflict of interest

9

6.4.

High frequency of provisional approval

10

6.5.

Inadequate handing of reported unloads/expensive statements

10

6.6.

Violation of development’s own checklist

10

6.7.

Heavy use of materialization & data inconsistency

11

6.8.

Understanding of source system data and functional requirements

11

6.9.

Very poor documentation

11

6.10.

HANA Model Improvements

12

6.10.1.

Union node pruning on Query view

12

6.10.2.

Materialized fact tables for current/archive cost

12

6.10.3.

Access Logic: Union node pruning

13

6.10.4.

Expensive source transaction tables joins in cost model

13

6.10.5.

Improvements in

CV_FACT_FINANCE_COST_REDESIGN_OPT

14

6.10.6.

Passing Filter Criteria from Business Object to HANA

16

6.10.7.

Common dimensions

bottleneck

16

6.10.8.

Column Length and Data Types

16

6.10.9.

Unnecessary calculated columns

17

6.10.10.

Redundant Nodes in Data Models

18

6.10.11.

Parallelization in Data models

18

6.10.12.

Analytical Privileges Implementation

18

6.10.13.

Cost Procedure optimization

18

6.10.14.

Miscellaneous

19

6.11.

Housekeeping

20

6.12.

Incremental Report Option

21

6.13.

Data Archiving

21

7.

Recommendation Prioritization & Impact Matrix

22

8.

Revised Sizing

23

9.

Conclusion

24

LIST OF ABBREVIATIONS TCS Confidential

3

Impact Matrix 22 8. Revised Sizing 23 9. Conclusion 24 LIST OF ABBREVIATIONS TCS Confidential 3

Ultimatix SAP HANA Implementation

Technical Review

This table provides the meanings for acronyms, abbreviations, and terms as used in this document. These definitions establish the common terminology to be understood and used by TCS in this document.

Abbreviation

Expanded Form

AP

Analytical Privileges

BODS

Business Object Data Services

BO

Business Object

CoE

Center of Excellence

GL

General Ledger

HANA

High Performance Analytic Appliance

PO

Purchase Order

PoC

Proof of Concept

SOP

Standard Operating Procedure

SRS

Sybase Replication Server

Ultimatix SAP HANA Implementation

Technical Review

1. Purpose of the document

This document highlights findings of technical review conducted for Ultimatix HANA implementation by SAP CoE. It explains the current state of infrastructure, issues and their resolutions along with implementation prioritization of recommendations.

2. Review Focus Areas (Requested by Ultimatix Team)

Review Cost report Model, Design, Code etc to identify improvement opportunities to reduce memory consumption. Current memory consumption based on optimized code from Ultimatix team is as below.

o

Stored procedure - which is executed every hourly basis. -- 80 GB

o

Execution of Company level report

- 50 GB

Based on memory consumption of cost report post improvement opportunities, work with Ultimatix team to arrive at sizing requirement for HANA to be used for enterprise level reporting for finance reports.

Review existing code and Design of cost program (and 1-2 more programs if required) to suggest improvement opportunities to improve the quality of code considering maintainability, extensibility, modularity etc.

3. Review Pre-requisites

Current (deployed) HANA Landscape Architecture details to be made available. (A very primitive diagram

was shared with us and later details of infrastructure were provided)

Details of HANA Sizing when instance was set-up. (Not provided date as it does not exist)

Details Loads/replications in HANA with classification of Full/Incremental. Loads schedule details and how they are aligned with non-critical hours. (Not provided till date as it does not exist)

Alternative landscape where review comments can be incorporate by Ultimatix development team and tested to see the benefits. (Not provided till date)

HANA Admin user with only Read access to everything specially _SYS/_SYS_STATISTICS schemas. (Not

provided till date due to security reasons)

Dedicated team members from Ultimatix HANA Admin, Development and functional areas working

with us in the review process. (Members from development, review and functional team were made available as per request)

Details (Overall Experience, Experience in HANA, Core skill set of HANA) of HANA Development Team.

(Not provided till date)

4. Review Activities

Carry out a fresh sizing exercise considering current data growth.

Technical review of Cost report Model on Source structure, Data types, Data granularity, master data, Loads, Joins, Projections, filters, input variables, security etc.

Incorporate review finding into existing model/loads estimate the improvement achieved.

Extrapolate the value for other models and calculate cumulative improvements If done for all models.

Formulate checklist, guidelines, and best practices for the project and educate team on same.

Ultimatix SAP HANA Implementation

Technical Review

5. Current State of Infrastructure

We classify current system as unstable, prone to frequent slowness, with extremely high CPU utilization and memory consumption.

Infrastructure Details (Extracted on 30 th Jan):

Parameter

Value

SYSTEM ID

PRD (Single node)

Hardware Manufacturer

HP

CPU (Cores/Threads)

80/160

Operating System

SUSE Linux Enterprise Server 11.2

Kernel Version

3.0.34-0.7-default

Product

SAP-HANA

Version

1.00.097.01.1436865239

Product Limit (Memory)

~900 GB

Usage (Memory)

~860 GB

Limit (Memory) ~900 GB Usage (Memory) ~860 GB HANA contains total 406 GB of application data

HANA contains total 406 GB of application data from source systems and materialized locally. Application data around 290-330 GB found to be loaded in memory whenever we extracted statistics during review. Application data which came from various sources is about 202 GB and surprisingly same amount of data (204GB) is duplicated/materialized in HANA after performing database joins and then storing it separately. We can conclude that actual application data in 900 GB HANA server is 202 GB. This 202 GB data would come down to 150 GB after we perform long pending housekeeping exercise.

Ultimatix SAP HANA Implementation

Technical Review

CPU utilization and memory consumption is exceptionally HIGH. During month-end HANA is unstable because no resources are left for processing. CPUs and Memory peaks are very often in production.

CPUs and Memory peaks are very often in production. There are 2 lacs+ unloads happened in

There are 2 lacs+ unloads happened in production during Jan-2017 month i.e. 6000-7000 unloads every day. Number of unloads happening are exceptionally high and it brings instability and affect performance of system negatively. It is clear that FACT_FINANCE_COST_ARCHIVE (size 34GB) under COSTING schema is the real problem. We have detailed the solution to address this in section 6.10.1/2.

schema is the real problem. We have detailed the solution to address this in section 6.10.1/2.

Ultimatix SAP HANA Implementation

Technical Review

We have 5000+ expensive statements reported in Jan, 2017 month. In below representation we have highlighted the procedures which are repeat offenders. Team has to carry out analysis from SQL quries appearing frequently on expensive statements.

SQL quries appearing frequently on expensive statements. There are many expensive statements with costly operations

There are many expensive statements with costly operations such as rtrim(), ltrim(), substring(), concat(). Ultimatix team has not yet provided any explanation on why these operations are performed.

e.g. SELECT Table

Table

4.\"VENDOR_NAME\",

to_char(Table

Table

4.\"SOURCE\",

Table

4.\"SUB_SOURCE\",

4.\"PO_NUMBER\",

4.\"CAL_MONTH_START_DT\",'Mon-YY'),

concat(substring(Table

4.\"FSC_QTR_NAME\",8,2),substring(Table

4.\"FSC_QTR_NAME\",4,2)),

Ultimatix SAP HANA Implementation

Technical Review

6. Review Findings & Recommendations

6.1. Absence of real requirements

Current HANA and business object implementation is being used for generating huge excel files with lowest granularity business data (e.g. GL line items) and with 100+ additional attributes. During month end, huge excel files approximately of size 800MB with all GL line items gets generated every hour. It results into high overhead, poor automation level and inefficient reporting. High-end platforms like HANA and business objects are being used for generating huge data sheets only. These huge files are plain data dumps which are used by respective business teams for further analysis in tools like Microsoft power pivot or excel. This very nature of development is present in all other modules and it can neither be classified as operational nor analytical reporting. While discussion with finance team, we got to know the first activity done on cost data dump is to detect any outlier (e.g. abnormal behavior in expense posting) and this is the real requirement.

Recommendation: During the audit it was observed that providing these data sheets is the major part of the requirement and development done in HANA Ultimatix development. 80%-90% of user actions both operational and analytical are done outside system after getting this data dump. We strongly recommend to initiate a separate thread to get real requirement from business into the system and reduce data sheets creation/ exchange.

Implementation Plan

Priority

Ownership

Long Term

HIGH

Business/BA Team

6.2. Incorrect understanding of operational reporting needs

It was observed that both functional and technical teams have a misunderstanding about operational reporting. Current data sheets are considered as operational reporting by the project. Operational reports does have a low granularity data but with an objective or a question being answered. E.g. reporting all high value (e.g. above 1 million USD) expense entries. Unlike the current requirement of downloading complete month GL line item data into an excel file.

Recommendation: Team should work on refining their reporting requirements for both operational and analytical purposes. It would result into breaking existing cost sheets into multiple operational reports.

Implementation Plan

Priority

Ownership

Long Term

HIGH

Business and Development Team

6.3. Review Team: Inadequate staffing and conflict of interest

Current review team only has one associate (Rajashree earlier Shirish) to do a detailed review of all deliverables from 30+ associates (as informed by Prakash) from development team. Review team is understaffed (strength of development team is yet to be shared with us. We requested this information at the start of our review but not yet provided). We pointed out conflict of interest in review team setup as it reports to Sanjay Hanche who is in-charge of development as well.

Recommendation: Review team’s hierarchy must not overlap with development team at any cost. Otherwise it would be a faulty & biased setup. First level of code review should be done with in the development team preferably by senior developer after that it should be sent to review team. After implementing this if best

Ultimatix SAP HANA Implementation

Technical Review

practices are not followed then review team should escalate the issues. Also management should look at adequately staffing review team (Current ratio 1::30 as informed).

Implementation Plan

Priority

Ownership

Immediate

HIGH

Project Management

6.4. High frequency of provisional approval

It was observed that 80%-90% of changes are done by taking provisional approval. These are the cases where proper clearances from review are not obtained to deploy code in production. This results into pushing sub- optimal code into production environment.

Recommendation: Development should be deployed into production only after obtaining clearance from review team.

Implementation Plan

Priority

Ownership

Immediate

Medium

Project Management

6.5. Inadequate handing of reported unloads/expensive statements

Server administration team has a weekly process of generating statistics of expensive statements and unloads happening in the system and sharing it with development team. As stated in section 5, these figures are exceptionally high. During the audit, team could not show an actionable plan for handling these alarming situations. We analyzed expensive statements of last four months and unloads of Jan-2017 month, we did not see a declining trend. There are many repeat offenders in expensive statements statistics. Administration is already facilitating this activity but development team is not actively fixing them.

Recommendation: This is to be dealt with utmost importance. Once admin team shares the statistics, development team must analyze them and share the root cause analysis with admin team. Development team must commit to dead line to fix these issues. These items must be to be followed up and included in reporting to higher management. Management must setup a framework to address these issues.

Implementation Plan

Priority

Ownership

Immediate

HIGH

Development & Admin Team

6.6. Violation of development’s own checklist

During the review we looked at project’s existing development checklist. HANA models we reviewed had violations of their own checklist (Sections 6.10.9/6.10.10/6.10.13).

Recommendation: While creating a model, there should not be any exception in following the best practices. Provisional approval must not be given to such cases and development team must follow their own checklist. It should be responsibility of development team to ensure adherence to best practices. Owner of the checklist should enhance it based on recommendations in section 6.10.

Implementation Plan

Priority

Ownership

Immediate

Medium

Development Team

Ultimatix SAP HANA Implementation

Technical Review

6.7. Heavy use of materialization & data inconsistency

As explained in section 5, 50% of data present in HANA is materialized data. Application data coming from source is only 202 GB out of total 405 GB. It is against the basic rule “No materialization” of HANA. It was observed that source table’s materialization is being followed as standard development practice in Ultimatix implementation across modules.

Materialization of transaction table with master data also leads to inconsistency. It could happen that master data has gone through changes over a period. If we view materialize data for longer period we would see multiple rows appearing for the same master data. This in turn would lead to inconsistency and wrong reporting.

Recommendation: In conjunction with section 6.10.1, team should do away with materialization. After we have a clear picture on real user requirement, approach should be to utilize HANA’s ability to perform complex join on the fly. This could only happen when requirements are not about extracting month long GL line item data with 100+ columns. A separate exercise of data consistency checks to be carried out in production to see the harm caused by materialization.

Implementation Plan

Priority

Ownership

Long Term

HIGH

Development Team

6.8. Understanding of source system data and functional requirements

For better and optimized report models, it is required for development team to have good understanding of functionality. During audit, we observed the lack of functional understanding in development team.

Recommendation: We suggest developers to be included in requirement discussion with business users. Involving developers at early stage would help in developing optimized model. Developer should also not hesitate to do data profiling at the start of the development to understand functionality better not when performance issues are reported in production.

Implementation Plan

Priority

Ownership

Long Term

High

Development Team

6.9. Very poor documentation

Project in general has a very poor documentation. Project does not even have a proper landscape architecture document. We were provided a very primitive document on architecture. Rest all documents such as data loads, component traceability were never provided.

Recommendation: A thorough compliance audit is to be conducted to check the missing documentation. We see a lot of non-compliance in the project on documentation front.

Implementation Plan

Priority

Ownership

Medium Term

Medium

Development Team

Ultimatix SAP HANA Implementation

Technical Review

6.10. HANA Model Improvements

6.10.1. Union node pruning on Query view

During review it was observed that calculation view used by Business Objects reports does union to current and archive cost data. We were told that most often users query running or previous month/quarter/year data. But current model scans archive data even for Jan, 2017 report. Existing model is done in such a way that flow will always go to Archive data irrespective of the date range input by user. For example: Report of Jan, 2017 will scan complete archive data but nothing would be returned as it is older than 2014. This faulty processing leads to FACT_FINANCE_COST_ARCHIVE (size 34 GB) table always being needed in memory.

Recommendation: It is recommended to implement union node pruning on top query view. Business Object would use the user input From/To Date and determine whether current/Archive/Current+Archive data is asked. This is could be done by putting a local logic in BO or making a call to HANA and determining the data selection range as current/Archive/Current+Archive (First option is preferred). By doing so we would be able to bypass scanning of huge archive data. This would result into significant benefits. Objective of doing this to introduce a where clause on data based on “Current” and “Archive” values.

Note: A small PoC was done to demonstrate the concept and benefits of union node pruning. Union node pruning is to be applied across module, wherever current and archive data are linked via union node. After performing if archive data is found in memory then team needs to investigate case on case basis and fix the all cases who are responsible for loading archive into memory.

cases who are responsible for loading archive into memory. Implementation Plan Priority Ownership Immediate

Implementation Plan

Priority

Ownership

Immediate

Very High

Development Team

6.10.2. Materialized fact tables for current/archive cost

Materialized table FACT_FINANCE_COST has data since 2014. Looking at the user query pattern, it was informed during review that most often users would select running or previous month/quarter data.

Schema Name

Table Name

Table Type

Record Count

Table Size(GB)

COSTING

FACT_FINANCE_COST_ARCHIVE

COLUMN

282,827,654

34

Ultimatix SAP HANA Implementation

Technical Review

COSTING

FACT_FINANCE_COST

COLUMN

197,857,457

32

Recommendation: Our recommendation would be reduced load on FACT_FINANCE_COST by moving 2014 and 2015 data into Archive. This approximately would cut down the size of FAC_FINANCE_COST to 50%. It would result into compounding improvements in all queries which are using FACT_FINANCE_COST Table. Development should also consider the possibility of archiving the data of FACT_FINANCE_COST_ARCHIVE table. Project team should observe queries on archive data and see a pattern on when user would query data beyond 2011. If not needed then data beyond 2011 should be archived.

Implementation Plan

Priority

Ownership

Immediate

Very High

Development Team

6.10.3. Access Logic: Union node pruning

In existing development, current and archive cost model nodes include resource intensive access logic. Access logic have 16 different sub nodes depending on access type of the user. Among all nodes, company node has maximum resource requirement. For a user with “company” access current access logic alone consumes approximately 169 GBs. During the review it was told that there are few users who have company level access.

But for restricted user, company node is executed and no rows are returned vice versa. Which could have been completely avoided by using union node pruning.

have been completely avoided by using union node pruning. Recommendation : It is recommended to implement

Recommendation: It is recommended to implement union node pruning on access logic. To do this business object should determine user type as “Company” or “Restricted”. This could be done by creating a small view in HANA and exposing it to BO to return the user type. This fetched value of user type should be passed to HANA Access logic as a where clause.

A long term plan should be to do pruning on level of access. There should be a separate query to determine the

level of access at user level and that should be passed to HANA as a where clause and pruning would happen on

level column in union. For example if user has level 10 and 11 access then remaining 14 nodes would be completely skipped.

Implementation Plan

Priority

Ownership

Immediate

Very High

Development Team

6.10.4. Expensive source transaction tables joins in cost model

It was understood during the review that GL_JE_LINES is driving table for cost model and every incremental run

has limited number of rows getting picked which are joined with huge transaction tables or views. If there has

Ultimatix SAP HANA Implementation

Technical Review

been some recent accounting movement then corresponding invoices/PO would have also gone through some update. It would be more efficient to join GJ_JE with relevant records of huge transaction tables rather than all.

Recommendation: After a brief discussion with Prakash, suggestion came to check last_update_date of tables such as invoices distribution, invoices, purchase orders etc. If there a recent GL movement then Last_update_date field should have been accordingly updated. It was suggested to be on safer side let’s consider last 3 months invoices while joining these tables with GL_JE. This would significantly reduce data set we consider in HANA Model for upward calculations.

Tables for Analysis: AP_INVOICE_DISTRIBUTIONS_ALL, AP_INVOICES_ALL, PO_HEADERS_ALL, PO_DISTRIBUTIONS_ALL, AR_CASH_RECEIPTS_ALL, RA_CUSTOMER_TRX_ALL etc for calculation view CV_FACT_FINANCE_COST_REDESIGN_OPT. Please consider all transaction tables with last_update_date for analysis which are not mentioned here in all HANA Models. Analysis should not be limited to CV where big transactions tables are use. It should be done for attribute as well as analytical views across modules.

for attribute as well as analytical views across modules. Implementation Plan Priority Ownership Short Term

Implementation Plan

Priority

Ownership

Short Term

High

Development & Functional Team

6.10.5. Improvements in CV_FACT_FINANCE_COST_REDESIGN_OPT

While reviewing HANA model “CV_FACT_FINANCE_COST_REDESIGN_OPT” it was observed that there is lot of scope for improvement in this CV. Any performance improvement in this view would have compounding effect as this is executed every two hours during normal days and every hour during month-end.

Ultimatix SAP HANA Implementation

Technical Review

Ultimatix – SAP HANA Implementation Technical Review Recommendation :  This particular model has a poor

Recommendation:

This particular model has a poor level of parallelization. This view has sequential steps and HANA performs best when modeling is done considering its ability of parallelization.

In addition to above points, Team should look at the scope of making a new join optimized information view, where all joins are performed as a star join and included in this model.

We should look at the possibility of logical grouping of join tables e.g. all invoices related joins are done at one node.

A Source table is being joined at multiple steps in the same model to include different columns. Our recommendation is to change model to join a source table at single place for all needs. E.g. HRM MR Organization table joined twice.

Removal of Redundant projection nodes.

New information views are to be created with needed columns in parent view instead of using bulky project common dimension views.

Implementation Plan

Priority

Ownership

Immediate

Very High

Development Team

Ultimatix SAP HANA Implementation

Technical Review

6.10.6. Passing Filter Criteria from Business Object to HANA

In current development, there are 3 mandatory filters and 20 optional filters enforced by Business object detailed cost report. But the benefits are not being passed to HANA. This is a massive bottleneck and should be removed on priority. E.g. consider a case where a user has access to multiple geographies. He/She specifies single geography value on the Business object front-end and runs the report. In this case HANA would process data for all the geographies even after user mentions a specific geo value. Complete data set would be returned to BO server and filtering would happen in BO. This is completely unnecessary. In existing development, these filter values are not being passed as where clause/variables to HANA to take advantage of it even though user diligently specifies the filters.

Recommendation: It is must to pass Filter values to HANA to take advantage of it. In fact, there should be more mandatory filters on the BO Front-end. We could look at a possibility of segregating reports into two category Cost Report for central team which has access to all the data with less mandatory filters. For rest all user we can create a copy of same report with more mandatory filters and less columns.

Implementation Plan

Priority

Ownership

Immediate

Very High

Development Team

6.10.7. Common dimensions bottleneck

Current implementation has lot of bulky calculation views created on common dimensions. Some of views are created with 100+ columns added as output and they end up consuming huge memory. This approach is okay with business warehouse system but goes against the thumb rule “Take what you need” in HANA. Consuming these bulky views at all places has a very negative impact on infrastructure.

Recommendation: You may continue to have these views for the cases where project data extract is being generated with all the attributes. But for the reporting and materialization views, which internally uses these common dimension views, a new set of dimension views are to be created with only columns which are required. This would significantly reduce memory consumption. This may slightly increase number of views but it is worth adding news views in the interest of infrastructure health.

E.g. consider a scenario where one of the cost model view requires 2 attributes from project. A new view in project module is to be created only using source tables whose attributes are needed in cost view. If only 2 attributes, location from project_master and last_update_date from project_history are needed then the new view would have only these two tables being joined. Not all tables of project module. This is applicable to all reporting and materialization information views using bulky common dimension views.

Implementation Plan

Priority

Ownership

Long Term

High

Development Team

6.10.8. Column Length and Data Types

It was found that materialized fact tables columns were created with higher sizes. These columns data types were out of sync with source systems and were unnecessarily created with additional size. No proper reasoning was given to do this in the model. Examples:

AL_CURRENCY_CODE is created as data type NVARCHAR (50).

Source system have GL_JE_HEADERS is create as a VARCHAR (300) and it comes NVARCHAR (500) in FACT_FINANCE_COST. There is no point in making a column NAVARCHAR where source table has it as

Ultimatix SAP HANA Implementation

Technical Review

VARCHAR. Doing this would consume 1 additional byte per character to store in unicode or multilingual.

Given are the wrong practices followed in the case of FACT_FINANCE_COST materialized table:

FACT_FINANCE_COST Table Column Name

FACT_FINANCE_COST Column Data Type & Size

PERIOD_NAME

NVARCHAR

100

WON

NVARCHAR

50

INVOICE_CURRENCY_CODE

NVARCHAR

50

AL_CURRENCY_CODE

NVARCHAR

50

INV_LINE_DESCRIPTION

NVARCHAR

300

INV_HDR_DESCRIPTION

NVARCHAR

300

TO_WHOM_INV_RAISED_EMP_NO

NVARCHAR

100

VENDOR_SITE

NVARCHAR

500

VENDOR_NAME

NVARCHAR

500

PO_NUMBER

NVARCHAR

500

PROJECT_TYPE_FINAL

NVARCHAR

100

JE_NAME

NVARCHAR

500

CLIENT_COUNTRY

NVARCHAR

150

OPERATING_UNIT

VARCHAR

100

THIRTEEN_CODE_COBINATION_ID

VARCHAR

250

FINAL_IOU

NVARCHAR

500

FINAL_SUB_IOU

NVARCHAR

500

Recommendation: We recommend to review all materialized fact tables and bring them to sync with source system. For any new development this should be included as a part of check list.

Implementation Plan

Priority

Ownership

Long term

Medium

Development Team

6.10.9. Unnecessary calculated columns

There are cases in the cost model where calculated columns were added to do amount fields up-casting but during review, team could not find and justify the need to doing so. This is done by adding a node and multiple calculated columns in cost data model. These up-casted calculated columns are later mapped to columns which are lower in size.

Recommendation: Please remove resource consuming redundant/unnecessary steps in existing model where need of having it not justified. This is to be validated in all models across modules.

Implementation Plan

Priority

Ownership

Short Term

Medium

Development Team

Ultimatix SAP HANA Implementation

Technical Review

6.10.10. Redundant Nodes in Data Models

During the audit, it was found that cost data models had redundant aggregation and projection nodes. There were cases where aggregation node was present but with no aggregation taking place. These redundant nodes are overheads in the HANA Models and they negatively impact CPU utilization and Memory Consumption.

Recommendation: All such cases have to be removed from data models across modules.

Implementation Plan

Priority

Ownership

Medium Term

Medium

Development Team

6.10.11. Parallelization in Data models

It is really important for HANA to have high a degree of parallelization being enforced by Data Model. Existing calculation view CV_FACT_FINANCE_COST_REDESIGN_OPT can be considered as bad way of designing a calculation views. Mentioned view has 25+ levels in graphical representation with absolutely no scope for HANA to parallelize execution of nodes.

Recommendation: It is highly recommended to look at all existing resource intensive views and try enforcing parallelization while modeling. This may result into creating new model all together but worth the pain.

Implementation Plan

Priority

Ownership

Long Term

High

Development Team

6.10.12. Analytical Privileges Implementation

Ultimatix HANA implementation does not use HANA Analytical privileges (AP) to control access to data. There is

a single user created in HANA for all reporting purpose. There are massive user tables creates in HANA to

implement custom access logic rather than using SAP recommended approach. Current custom access logic is memory/CPU intensive and slows down processing in HANA and in turn result into poor response time. For a single corporate finance user, access logic alone (without txn. data) consumes approximately 170GB of memory

if date selection is for a complete month. Current approach puts a restriction on auditing HANA, we would never

be able to HANA auditing feature to troubleshoot if we have only one reporting user in HANA.

Recommendation: Team should migrate to SAP recommended analytical privileges. A through exercise to be carried out to derive a security matrix for the project and implemented as per SAP Guidelines. Any custom implementation of access control would obviously be slower than the SAP recommended way.

Note: We did a small PoC in our lab to show the benefits of using analytical privileges.

Implementation Plan

Priority

Ownership

Long Term

Medium

Development Team

6.10.13. Cost Procedure optimization

Current cost procedure does not do exception handling and considers last thirty days data for analysis. Cost procedure runs every 2/1 hour in production during regular days/month end.

Ultimatix SAP HANA Implementation

Technical Review

Recommendation: Exception handling is to be included in procedure and current processing of doing a look up of last 30 days should be optimized to current day. Project team should do an analysis on deciding an optimal value. Logically speaking, it would be around 2 hours. A suggestion came from Prakash to do analysis on replacing current delete and insert with update/upsert statements.

Implementation Plan

Priority

Ownership

Immediate Term

Medium

Development Team

6.10.14.

Miscellaneous

Data Profiling: We recommend team to carry out data profiling specially for scenarios such as Tables and their columns which are materialized OR Columns which are used in joins OR later used in some calculation. It was observed during the audit that few columns who were used in calculations/joins had all null values. E.g. ATTRIBUTE4 and ATTRIBUTE8 of GL_JE_LINES has all NULL value for all the records.

Implementation Plan

Priority

Ownership

Immediate

High

Development Team

JOIN Columns: Special care is to be taken for columns which are part of join conditions. It was observed that columns reference 5, 7 and 8 in GL_IMPORT_REFERENCES table had many null values and they were in inner join. This scenario could be optimized by putting a filter at lower nodes and not considering null values for joins. This would reduce the data set considered for join.

Implementation Plan

Priority

Ownership

Immediate

High

Development Team

Cardinality Assignment: It was observed during the audit that cardinality assignments were not accurate. Cardinality assigned for all the join should be reviewed and redone wherever is not assigned accurately. Development team can take help from business team to find out the nature and cardinality of joins.

Implementation Plan

Priority

Ownership

Immediate

High

Development Team

Optimize join: Optimize option for all left joins should be set true. Presently it is not set right.

Implementation Plan

Priority

Ownership

Immediate

High

Development Team

JOIN Relevance: During the audit we found few joins which were not needed. Team should take help from business and determine the joins which are not require and should remove them. This should be carried for all modules. Examples: From one table max values of contract version is selected and another table is joined to get the contract ID for max version. Where in contract ID field is available in the same table where we store versioning of contract. Same was true with case of multiple end-client join analyzed during review.

Implementation Plan

Priority

Ownership

Immediate

High

Development Team

Ultimatix SAP HANA Implementation

Technical Review

6.11.

Housekeeping

We observed that admin team has sent out multiple communications in the past to development team for confirmation but there was no response/action from development team.

Business Critical and Real time Reporting: It is recommended to use HANA for business-critical and real- time reporting. It was observed during audit that that card access details which are 22.5 GB on disk and 16 GB loaded in memory. Team informed that SRC_CARDACC is planned to be taken out from HANA to Hadoop. This point is mentioned in report to record this as an action item.

SCHEMA_NAME

TABLE_NAME

RECORD_COUNT

SIZE on Disk (GB)

SIZE on Memory (GB)

SRC_CARDACC

FACT_CARD_ACCESS

211

975 120

12.26

12.21

COMMDIM

FACT_CARD_ACCESS_ANALYSIS

121

233 618

4.06

3.77

SRC_CARDACC

FACT_CARD_ACCESS_JUL

94 436 083

3.03

0.00

BILLING

FACT_CARD_FIRST_LAST_SWIPES

8 838 795

0.08

0.06

SRC_CARDACC

FACT_CARD_ACCESS_OCT

104

308 453

2.78

0.00

SRC_CARDACC

FACT_CARD_ACCESS_TEMP

 

0

0.26

0.00

COMMDIM

CARD_ACCESS_VALID_SOURCES

 

0

0.00

0.00

SRC_SURVEY

SURVEY_DATA

 

4180

.001

.001

It was found that schema SRS_TEST of size 48.5 GB was created in memory for some testing purpose. Team must plan to remove this schema once the need of this schema is over.

Implementation Plan

Priority

Ownership

Medium Term

High

Development Team

Redundant Tables: A separate exercise is to be conducted to remove redundant tables from production. A clean up activity considering all schemas/tables must be planned and tracked to optimize storage needs. Below are the list of tables we highlighted during the review but have not received any reply till date. Below list contains probable redundant tables consuming 26GB of storage.

SCHEMA_NAME

TABLE_NAME

RECORD_COUNT

SIZE on Disk(GB)

SRC_CARDACC

FACT_CARD_ACCESS_TEMP

0

0.2552

SRC_IPMS

T_WONLIST_TESTING

258012

0.0451

COMMDIM

DIM_TIME_BKP_TEST

10593

0.0018

SRC_CARDACC

FACT_CARD_ACCESS_TEMP

0

0.2552

SRC_CARDACC

FACT_CARD_ACCESS_JUL

94436083

3.0313

SRC_CARDACC

FACT_CARD_ACCESS_OCT

104308453

2.7766

SRC_CRM

S_CONTACT_HIST1

705730

0.3138

SRC_ETCS

GJL_TST_1YR

47926072

2.6930

SRC_ETCS

GL_BALANCES_2010_2012

189576335

2.0709

SRC_ETCS

GL_BALANCES_2002_2009

92281601

1.1924

SRC_ETCS

GL_JE_LINES_2010_2012

103846036

6.5474

SRC_ETCS

GL_JE_LINES_2002_2009

54313722

5.3333

SRC_HRMS

HRM_TX_ASSIGNMENT_BI_T_TEMP

21739213

1.085

Ultimatix SAP HANA Implementation

Technical Review

COMMDIM

DIM_PROJECT_TEMPR

455271

0.301

SRC_CRM

S_OPTY_TEMP

217212

0.042

SRC_CRM

S_OPTY_X_TEMP

217209

0.023

SRC_CRM

S_EMP_PER_TEMP

131982

0.028

SRC_CRM

S_ORG_EXT_TEMP

43531

0.008

SRC_FIN

TCS_AP_AMORTIZE_GL_TEMP

190173

0.015

Implementation Plan

Priority

Ownership

Medium Term

High

Development Team

6.12. Incremental Report Option

During one of our call with finance team, it was suggested by Prakash to stop extracting complete month information in excel file every hour. He suggested to download complete file at the start of month end process then keep downloading incremental till resource intensive month end process ends. We agree with this suggestion from Prakash and it could massively reduce resource consumption during month end on HANA system.

Implementation Plan

Priority

Ownership

Medium Term

High

Development Team

6.13. Data Archiving

There is no data purging or archiving process in place currently. Currently large amount of data is uploaded into main memory. For reporting purpose, all the data in the database (warm data) might not be required to be loaded into main memory (hot data).

Recommendation: It is recommended to archive out the data (cold data) which is not required for operational reporting. Data may be carved out using the SAP Data Archiving tool or data may be exported out and deleted from the system. The archived data may be stored on the storage system or in a 3rd party tool. Date archived through SAP tools can be viewed at a later point in time.

Another option to store cold data is to export out the data into a Near Line Storage (NLS) system. SAP Sybase database may be used as the NLS storage.

Implementation Plan

Priority

Ownership

Long Term

Medium

Development Team

Ultimatix SAP HANA Implementation

Technical Review

7. Recommendation Prioritization & Impact Matrix

Action Plan: Immediate Action Items: 10

Heading

Description

Priority

Ownership

6.10.1

Union node pruning on Query view

Very High

Development Team

6.10.2

Materialized fact tables for current/archive cost

Very High

Development Team

6.10.3

Access Logic: Union node pruning

Very High

Development Team

6.10.5

Improvements in CV_FACT_FINANCE_COST_REDESIGN_OPT

Very High

Development Team

6.10.6

Passing Filter Criteria from Business Object to HANA

Very High

Development Team

6.10.14

Miscellaneous

High

Development Team

6.3

Review Team: Inadequate staffing and conflict of interest

High

Project Management

6.5

Inadequate handing of reported unloads/expensive statements

High

Development & Admin Team

6.4

High frequency of provisional approval

Medium

Project Management

6.6

Violation of development’s own checklist

Medium

Development Team

6.10.13

Cost Procedure Optimization

Medium

Development Team

Action Plan: Short Term Action Items: 3

Heading

Description

Priority

Ownership

6.10.4

 

High

Development &

Expensive source transaction tables joins in cost model

Functional Team

6.12

Incremental Report Option

High

Development Team

6.10.9

Unnecessary calculated columns

Medium

Development Team

Action Plan: Medium Term Action Items: 3

Heading

Description

Priority

Ownership

6.11

Housekeeping

High

Development Team

6.9

Very poor documentation

Medium

Development Team

6.10.10

Redundant Nodes in Data Models

Medium

Development Team

Action Plan: Long Term Action Items: 3

Heading

Description

Priority

Ownership

6.1

Absence of real requirements

High

Business & BA Team

6.2

Incorrect understanding of operational reporting needs

High

Business/Dev. Team

6.7

Heavy use of materialization & data inconsistency

High

Development Team

6.8

Understanding of source data & functional requirements

High

Development Team

6.10.7

Common dimensions bottleneck

High

Development Team

6.10.11

Parallelization in Data models

High

Development Team

6.10.8

Column Length and Data Types

Medium

Development Team

6.10.12

Analytical Privileges Implementation

Medium

Dev. & Admin Team

6.13

Data Archiving

Medium

Admin & Dev Team

Ultimatix SAP HANA Implementation

Technical Review

8. Revised Sizing

The production HANA database is installed on a HP SAP In-Memory Appliance configured with 1 TB memory and 160 CPU. The HANA 1.0 database is running on SLES 11.2. The size of the HANA database currently is around 500 GB on disk (system + application data) with a growth rate of approximately 110 GB per year.

Recommendation: It needs to be pointed out that the sizing will need to be revisited after the suggestions given in this report above are implemented namely, optimization of the expensive SQL statements, redesigning of the materialized views and rescheduling the housekeeping jobs. Current state does not represent the right picture of infrastructure to do accurate sizing.

Considering the size of the existing database the memory, current estimate for sizing of the HANA appliance looks adequate. Keeping in mind the data growth rate and considering that no data reduction exercise is carried out, the HANA Appliance will need to be scaled up after a year and half to two years.

Ultimatix SAP HANA Implementation

Technical Review

9. Conclusion

This report takes into consideration all aspects of the implementation, ranging from development to team staffing. While we have identified a few risks from a business and project management standpoint, the most of the risks we found were in the suboptimal practices employed by the development team. A sizeable number of our risks currently faced by the system could have been easily mitigated by performing basic housekeeping tasks and expensive statement analysis.

Considering the size of the existing database the memory, current estimate for sizing of the HANA appliance looks adequate. Sizing will need to be revisited after the suggestions given in review report are implemented. Current state does not represent the right picture of infrastructure to do more accurate sizing

On the whole, we see a major scope for improvement in the current landscape. There also needs to be a collaborative effort between management, business developers and technical developers to identify better and more relevant requirements in order to deliver optimal output from this HANA system. Depending on the criticality, we have also categorized our recommendation into four broad categories: Immediate, Short Term, Medium Term and Long Term.

HANA has been designed with operational as well as analytical reporting in mind, and to use it to generate data sheets which are subsequently used for analysis is not an optimal usage of such advance platform. We strongly recommend that the system itself be used for operational as well as analytical reporting not generating data sheets.

----End of the Report----