Академический Документы
Профессиональный Документы
Культура Документы
Edwards EnterpriseOne
BSSV Cluster Project Result Analysis
Disclaimer
The following is intended to outline our general product direction. It is intended for information purposes only,
and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or
functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing
of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
Page i
JD. Edwards EnterpriseOne BSSV Cluster Project Result Analysis
Page 1
JD. Edwards EnterpriseOne BSSV Cluster Project Result Analysis
Executive Overview
The JD Edwards EnterpriseOne interoperability solution, namely Business Services (BSSV) is a
Java-based web service that works with JD Edwards EnterpriseOne software to service
EnterpriseOne consumer and provider external requests.
The results of the BSSV Clustering project presented in this document are to characterize three
EnterpriseOne business services, namely:
The BSSV WebLogic 11g cluster resides on two separate machines pointing to a common
EnterpriseOne/Database architecture to provide a high availability and load balancing system.
Testing of the BSSV services included load and stress scenarios in a load balanced and high
availability WebLogic cluster configuration. Although the main benefit of the testing was the
validation of the architecture, transaction load rates and specific configuration changes are
documented to help provide information for BSSV scaling for customers. The main conclusions
drawn from the testing of the BSSV Services on a multi-server Clustered WebLogic server
architecture are:
Page 2
JD. Edwards EnterpriseOne BSSV Cluster Project Result Analysis
Project Goals
Test BSSV services in a WebLogic Cluster in a load balanced and high availability architecture;
evaluate the performance characteristics under unit, load and stressed conditions. This goal was
achieved in two phases
Performance – Phase II
The second stage was to introduce load and create stress scenarios to characterization the
performance of BSSV services and tuning of BSSV services in a clustered environment. Some
tuning was also made to optimize the WebLogic server and Web Services to achieve good
performance and to remove any limitation to the BSSV process flow that might limit performance.
Load was generated using SOAP requests from an HP LoadRunner controller to allow for
increasing load to the BSSV WebLogic cluster.
The project analyzed performance and characterized the BSSV web service process flows through
the EnterpriseOne application on the BSSV WebLogic Cluster by measuring the key metrics
provided by the LoadRunner controller, database, and operating system. The figures below depict
the architectures and process flows used for this project.
Page 3
JD. Edwards EnterpriseOne BSSV Cluster Project Result Analysis
Page 4
JD. Edwards EnterpriseOne BSSV Cluster Project Result Analysis
Page 5
JD. Edwards EnterpriseOne BSSV Cluster Project Result Analysis
Test Design
BSSV Load and Stress Testing
The BSSV web services load and stress tests are driven by the Load Generation HP Software
LoadRunner that generates SOAP messages to the BSSV WebLogic Server. The testing consisted
of unit, load and stress testing in a load balanced and high availability configuration.
1. Unit Testing
Each Soap Request run in DEBUG mode
2. Load Balance Load Testing
100 user/200 user/300 user of each Soap Request Type, single line order entries
3. High Availability Load and Stress Testing
100 user/200 user/300 user of each Soap Request Type, single line order entries
300/600/900 user mix load (Address Book, SOE, POE), sing line order entries
300 user SOE and POE by varying number COK kernels
50/100/150 user SOE and POE – 10 line orders
50/100 user SOE and POE – 50 line orders
4. High Availability Reliability Testing
300 users of each Soap Request Type run for 8 hours, single line order entries
Negative Testing
Failing nodes in Load Balanced and High Available configurations
Bad Data
BSSV Cluster Project Metrics
The following metrics were collected to measure and characterize the performance of the BSSV
test scenarios:
1. LoadRunner
Average Response Time of Soap Request
Transaction Rate
Soap Message Request Size
2. EnterpriseOne Operating System Metrics (E1 Logic Server, DB Server)
CPU, Memory, and Disk metrics
3. BSSV WebLogic Cluster Metrics
CPU, Memory, and Disk metrics
4. WebLogic Java Metrics for each managed server instance (node)
Page 6
JD. Edwards EnterpriseOne BSSV Cluster Project Result Analysis
5. SQL Validation
LoadRunner and the Generation of SOAP Requests
For all of the tests described in this document, HP LoadRunner was used to allow multiple users to
generate SOAP requests that were consumed by the WebLogic BSSV cluster architecture and then
processes by the EnterpriseOne services in a provider scenario.
As part of the presentation of the results, a number of graphs will be presented depicting the metrics
that were collected, these include
1. Transaction Rate
The transaction rate is the total number of SOAP messages that were submitted to the
WebLogic BSSV cluster during the 1 hour testing cycle. The relative rates will increase as
the number of users increased. Furthermore, the transaction rate is somewhat arbitrary and
can be varied by changing the rate of submission from the LoadRunner controller. A
variable rate of submission of a SOAP request by the LoadRunner controller of 30-45
seconds was used as a standard during all testing of BSSV SOAP messages to the
WebLogic BSSV cluster.
2. Response Time
The response time is measured as the average acknowledgement time of the SOAP request
submission from the LoadRunner controller to the WebLogic BSSV cluster.
Database Memory
There are two main variables when considering memory consumption on the
database server
Variable 1: The Oracle database configurations for memory, SGA, and PGA. The
memory from this variable is allocated at database startup.
Variable 2: In all tests, dedicated connections were defined. Memory consumed by
database is dependent on the number of users of the system. For BSSV an average
of 1.5 connections were established to the Oracle database per user.
For best results, database memory consumption was kept under 90% of total
available RAM memory to avoid operating system paging conditions.
EnterpriseOne Memory
The primary consumer of memory for the EnterpriseOne software is that of the
kernel process Call Object Kernels (COK) or KDEF6 as defined by the JDE.INI
configuration file.
Page 7
JD. Edwards EnterpriseOne BSSV Cluster Project Result Analysis
There were two primary guidelines in setting the EnterpriseOne configuration of call
object kernels.
The first guideline is that 1 Call Object Kernel was defined for every 10 BSSV users
The second guideline is that total memory consumption for the test would not
exceed 90% of total memory of the server to avoid operating system paging
conditions
Although the average operating system metrics are reported for each scenario definition, it
was found that each metric fluctuated constantly during the test. Each of the components of
the architecture has an inherent ability to self tune. For example, the Oracle database will
start to cache certain tables in memory if it finds that they are used frequently. The
operating system will try to use swap and resident memory for processes that are very active
as is the case with the EnterpriseOne call object kernel processes. Finally, the WebLogic
server cluster will try to balance SOAP message requests among the defined managed
server instances available. All of the test scenarios that were initiating in this project were
for a 1 hour duration. Metrics were collected and reported in this document when the
system was at steady-state.
Page 8
JD. Edwards EnterpriseOne BSSV Cluster Project Result Analysis
Test Results
WebLogic Cluster–Load Balancing and High Availability Testing
The packet size variable plays a key role in the performance of the BSSV web services in a
WebLogic clustered environment. Figure 4 below lists the packet sizes of all of the SOAP requests
that were used in the testing process. The effects of packet size on performance and tuning will be
explained in the discussion of the test scenarios used in this project.
Packet Size
150000
123737
Packet Size (Bytes)
100000
55773
50000 27104
6509 5561 12303
2606
0
The transaction rates can be increased by either increasing the number of users submitting SOAP
requests or by increasing the rate of SOAP submission through LoadRunner by decreasing the
amount of time between user SOAP requests
Performance Tip:
The size of the SOAP message will have an impact to the amount of operating system consumed
resources. Larger packet SOAP message size will have an impact to metrics such as JAVA heap,
Oracle Database memory and EnterpriseOne memory components.
Page 9
JD. Edwards EnterpriseOne BSSV Cluster Project Result Analysis
WebLogic Cluster–Load Balancing and High Availability Testing – Low Packet Size
Figure 5 below shows the results of the testing of increasing interactive user load using BSSV web
services in a WebLogic Cluster load balanced and high availability architecture. A low SOAP
message packet size (ranging from 2606 to 6509 bytes) was used in this testing.
The below diagram shows that all of the BSSV web services scale well in the load balanced (LB)
and high available (HA) configurations for the WebLogic BSSV cluster.
The main observations that can be stressed with the following diagram
1. SOE and POE submissions was initiated with a single line item order, thus scaling was
performed in a low SOAP request size configuration
2. Response times for BSSV web services were better in a high availability environment
where two more WebLogic managed server JVM instances are available to handle requests
Page 10
JD. Edwards EnterpriseOne BSSV Cluster Project Result Analysis
Figure 6 below shows the transaction rates that were achieved with the varying number of users.
The rate increased from a base of an average of 15,000 transactions per hour to a rate of
approximately 41,000 transactions per hour.
50000
Rate (trans/hr)
Testing Tip:
Transaction rates were variable and are controlled by the pacing of SOAP request submissions
through LoadRunner. The data shown thus far is with a set pacing of a SOAP message submission
per user every 30-45 seconds.
Page 11
JD. Edwards EnterpriseOne BSSV Cluster Project Result Analysis
Figure 7, Figure 8, and Figure 9 below are the operating system metrics obtained for the various
scaling tests.
Address Book
Operating System Metrics
60
50
Percentage
40
EnterpriseOne CPU
30
20 Database CPU
30 EnterpriseOne CPU
20 Database CPU
10 BSSV WLS CPU
0 BSSV JAVA (Used)
100 200 300
BSSV Users
Figure 8: Operating System Metrics for Sales Order Entry (Load Balanced)
Page 12
JD. Edwards EnterpriseOne BSSV Cluster Project Result Analysis
25
EnterpriseOne CPU
20
15 Database CPU
10
BSSV WLS CPU
5
0 BSSV JAVA (Used)
100 200 300
BSSV Users
Figure 9: Operating System Metrics for Purchase Order Entry (Load Balanced)
The main observations that can be stressed with the above operating system statistics diagrams
1. Sales order and purchase order entry have greater requirements of EnterpriseOne Logic and
Oracle Database. The EnterpriseOne feature of Advanced Pricing was enabled for this
testing to create a more realistic representation of a customer use case.
2. EnterpriseOne Memory consumed: 2.3GB-2.9GB
3. Oracle Database Memory consumed: 6.2GB-6.6GB
4. BSSV WebLogic Server Memory consumed: 3.1GB-3.8GB per server
The remainder of the testing scenarios are performed in a high availability scenario since it made
no apparent difference in LoadRunner response time whether the test was run in a load balanced
configuration or a high availability configuration. The decision to proceed in this fashion was also
influenced by the fact that more load could be handled with the larger number of managed
instances available in a high available configuration.
Page 13
JD. Edwards EnterpriseOne BSSV Cluster Project Result Analysis
Figure 10 below shows the transaction rates achieved in a mixed load testing scenario.
Transaction Rate
Mixed Load Testing
200000
Rate (trans/hr)
150000
100000
50000
0
300 600 900
BSSV Users
Figure 11 below shows the operating system metrics from this exercise.
Mixed Load
Operating System Metrics
70
60
50
Percentage
40 EnterpriseOne CPU
30
Database CPU
20
10 BSSV WLS CPU
0 BSSV Java (Used)
300 600 900
BSSV Users
Page 14
JD. Edwards EnterpriseOne BSSV Cluster Project Result Analysis
The main observations that can be stressed with the above diagrams
1. The largest transactional load of above 150,000 transactions per hour was achieved during
this test with 900 users (300 of each of the three BSSV web services)
2. The LoadRunner pacing was set to achieve these numbers and maintain a good average
response time or no more than 3 seconds for any of the three BSSV web services.
3. Database memory consumption was the limiting factor in these tests. More memory was
required for the dedicated connections that was established for the 900 user test
4. The BSSV WebLogic statistics of CPU, memory and average Java heap size used are an
average of the two WebLogic cluster servers used in the high availability testing.
5. EnterpriseOne Memory consumed: 3.7GB-4.4GB
6. Oracle Database Memory consumed: 6.2GB-6.6GB
7. BSSV WebLogic Server Memory consumed: 3.1GB-3.8GB per WLS server
Page 15
JD. Edwards EnterpriseOne BSSV Cluster Project Result Analysis
Figures 12, 13, and 14 show the metrics collected for varying packet size requests and number of
users for a typical sales order SOAP request.
10
Response Time (sec)
4
1.672 1.592
2
0
50U-10 line 50U-50 line 100U-10 line
BSSV Users
6000
4000
2000
0
50U-10 line 50U-50 line 100U-10 line
BSSV Users
Page 16
JD. Edwards EnterpriseOne BSSV Cluster Project Result Analysis
40 EnterpriseOne CPU
30
Database CPU
20
10 BSSV WLS CPU
0 BSSV Java (Used)
50U-10line 50U-50line 100U-10 line
BSSV Users
The main observations that can be stressed with the above diagrams
1. Good transaction rates and response times were collected for the 10 line item entries, even
when the user count increased from 50 to 100 users
2. A poor response time of 11.12 seconds was observed with the 50 user and 50 line test
scenario. This was accompanied by a large database memory consumption statistic as well.
3. An opportunity for performance tuning is presented for the 50 user and 50 line 124K SOAP
packet size messages scenario. A better response time could be achieved if the required
transaction rate was lower or additional tuning employed. These options were not explored
in this project.
4. EnterpriseOne Memory consumed: 3.1GB-3.3GB
5. Oracle Database Memory consumed: 5.3GB-6.6GB
6. BSSV WebLogic Server Memory consumed: 3.1GB-3.8GB per WLS server
Page 17
JD. Edwards EnterpriseOne BSSV Cluster Project Result Analysis
Figures 15, 16, and 17 show the metrics collected for varying packet size requests and number of
users for a typical purchase order SOAP request.
20
15
9.484
10
5 2.272 2.016
0
50U-10line 50U-50line 100U-10 line 100U-50 line
BSSV Users
7000
6000
5000
4000
3000
2000
1000
0
50U-10line 50U-50line 100U-10 line 100U-50 line
BSSV Users
Page 18
JD. Edwards EnterpriseOne BSSV Cluster Project Result Analysis
60 EnterpriseOne CPU
40 Database CPU
20 BSSV WLS CPU
0 BSSV Java (Used)
50U-10line 50U-50line 100U-10 line 100U-50 line
BSSV Users
The main observations that can be stressed with the above diagrams
1. Good transaction rates and response times were collected for the 10 line item entries, even
when the user count increased from 50 to 100 users
2. A poor response time of 9.5 and 24.9 seconds was observed with the 50 line test scenarios.
This was accompanied by a large database memory consumption statistic as well
3. Similar to the Sales Order Entry results, additional tuning or lower transactional rate
adjustments could be employed to lower the response times of the high packet size SOAP
messages of purchase order entry. This was not explored with these results.
4. EnterpriseOne Memory consumed: 3.2GB-3.4GB
5. Oracle Database Memory consumed: 5.3GB-6.6GB
6. BSSV WebLogic Server Memory consumed: 3.3GB-3.8GB per WLS server
Page 19
JD. Edwards EnterpriseOne BSSV Cluster Project Result Analysis
In all the testing scenarios, the SOAP message response time was found to be the first indication of
a performance or configuration bottleneck. The following are basic steps of performance tuning
that were performed in this testing characterization.
With low packet sized SOAP messages, much higher transaction rates can be achieved. In the
testing of the BSSV processes, 150K transactions were achieved for 900 users in a mixed low load
scenario. This high rate was achieved because the BSSV architecture handles packet sizes of 2K-
12K bytes extremely well. This is intuitive because the amount of resources, memory allocation,
and network transmission of the larger packet sizes between the BSSV WebLogic Cluster Server,
EnterpriseOne processes, and Oracle Database is much greater.
Keep in mind that when tuning the architecture to handle a high transaction load when SOAP
message packet size is low is different than tuning for a high packet size SOAP message for high
transactional rate loads.
EnterpriseOne Tuning
EnterpriseOne tuning for low and high packet sized SOAP messages and high transaction rates
centered solely on the performance of the Call Object Kernels. In general, maintain the rule of 10
BSSV concurrent users per COK kernel.
An addition metric to view is the number of active threads per COK kernel. This metric is
retrieved from the EnterpriseOne Server Manager Console. To improve performance metrics such
as response time and transactional rates, 6 or 7 threads per COK kernel gave good results. If the
number of threads exceeded this, the number of COK kernels was increased. The cost of the
additional COK kernels would be an increase in the memory consumption of the EnterpriseOne
server. When response times of the SOAP message requests exceeded a reasonable limit of 3
seconds, the threads were reviewed for performance interests.
Larger SOAP message packet sizes required an increased amount of memory needed per user for
each COK kernel. Larger SOAP message packet sizes (100K bytes or greater) tended to require
more COK kernels as well or on average of 6 concurrent BSSV users per COK kernel were needed
to achieve better response times.
Page 20
JD. Edwards EnterpriseOne BSSV Cluster Project Result Analysis
Lesser tuning concerns of adjusting the number of network processes and security kernels were also
performed. In general 100 BSSV concurrent users per network process and security kernel was
configured.
The further tuning of the JDE.INI setting of JDENETTimeout from 60 to 120 seconds was found to
be beneficial with larger SOAP message sizes.
It is recommended that the Oracle database advisors and other tuning tools be used for tuning the
Oracle Database Server.
Larger SOAP packet sized messages were found to have a higher percent JAVA heap size for the
same amount of users as with a low SOAP packet size. More frequent and longer cost to JAVA
garbage collections were also seen with larger SOAP packet size messages. Each of these have an
impact to performance, so adjusting them according to the transaction rate and size of SOAP packet
size is important.
In general, when the JAVA heap size usage exceeded 80%, then the response time dramatically
increased. Increasing the JAVA heap size or increasing the number of WebLogic Managed Servers
in the cluster is a way to counter this.
The heap size chosen on a 4.3GB WebLogic Server Cluster with 2-nodes was 768MB. This
allowed memory overhead on the server to not exceed 90% for the scenarios presented in this
document. Having two 4.3GB WebLogic Server Clusters allowed for a total of 4-nodes to handle
BSSV requests. Normally this value would range from 1024MB to 4096MB per Managed Server
Node depending on the total amount of memory on the BSSV WebLogic Server.
Page 21
JD. Edwards EnterpriseOne BSSV Cluster Project Result Analysis
The lesser tuning of MAXCLIENTS and SERVERLIMIT was also done in order to avoid any
concurrent JAVA user maximum configuration limitation. These are adjusted depending on the
limits of how many concurrent BSSV users that will be anticipated per Managed Server node. The
number of BSSV concurrent users per Managed Server Node will depend on balancing the
following concerns:
Size of the JAVA heap size – more users can be handled if the heap size is larger
Size of the SOAP message packet size – more users can be handled if the packet size is
small
Transaction rate – more concurrent users can be achieved with lower transaction rates
Desired response time – more users can be achieved when the response time threshold is
increased
Amount of memory on the server – leave enough memory on the server and avoid operating
system paging, operating system paging decreases performance
JAVA heap usage – noticeable performance degradation occurs when JAVA heap usage is
above 80%
Additionally, increasing the enterpriseServerTimeout in the JAS.INI file from the default of 90 to
180 seconds was found to be beneficial with larger SOAP message packet sizes. Performance and
faster response times occurred by allowing more time for the EnterpriseOne business functions to
complete when the transaction rate was high.
Page 22
JD. Edwards EnterpriseOne BSSV Cluster Project Result Analysis
Summary
The following conclusions summarize the findings.
EnterpriseOne Logic
4. The number of Call Object Kernels (COK) for BSSV users is equivalent to normal
individual interactive users. Depending on SOAP packet size of the message, this can be
from 6 to 10 concurrent BSSV users per COK kernel process.
Reliability
7. The BSSV environment was shown to be reliable during 8 hour testing
Page 23
JD. Edwards EnterpriseOne BSSV Cluster Project Result Analysis
Glossary
DB - Database
DIL – Day in the Life (Refers to pre-build E1 database for load/stress testing)
E1 – JD Edwards EnterpriseOne (ERP Software)
OAS - Oracle Application Server, OracleAS
OEL – Oracle Enterprise Linux
OEM – Oracle Enterprise Manager (Monitoring/Configuring tool for Oracle DB)
OHS - Oracle HTTP Server
OVM – Oracle Virtual Machine
RAC – Real Application Cluster
UBE – Universal Batch Process
Page 24
JD. Edwards EnterpriseOne BSSV Cluster Project Result Analysis
J.D. Edwards EnterpriseOne BSSV Cluster Copyright © 2011, Oracle and/or its affiliates. All rights reserved. This document is provided for information purposes only and the
Project Result Analysis contents hereof are subject to change without notice. This document is not warranted to be error-free, nor subject to any other
February 2011 warranties or conditions, whether expressed orally or implied in law, including implied warranties and conditions of merchantability or
fitness for a particular purpose. We specifically disclaim any liability with respect to this document and no contractual obligations are
formed either directly or indirectly by this document. This document may not be reproduced or transmitted in any form or by any
Oracle Corporation
means, electronic or mechanical, for any purpose, without our prior written permission.
World Headquarters
500 Oracle Parkway
Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.
Redwood Shores, CA 94065
U.S.A.
AMD, Opteron, the AMD logo, and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices.
Worldwide Inquiries: Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license
Phone: +1.650.506.7000 and are trademarks or registered trademarks of SPARC International, Inc. UNIX is a registered trademark licensed through X/Open
Fax: +1.650.506.7200 Company, Ltd. 1010