Home

About

Careers

Clients

Consulting

Training

Support

Articles

Blog

OBIEE Regression Testing An Introduction

January 17th, 2014 by Robin Moffatt

In this article Im going to look at ways to test changes that you make to OBIEE to ensure that they dont break existing
functionality. In all but the simplest IT systems its common for one (planned) action to inadvertently cause another
(unplanned).

Search the blog

Recent Posts
Analytics w ith Kibana and
Elasticsearch through Hadoop
part 3 Visualising the data in
Kibana
Analytics w ith Kibana and
Elasticsearch through Hadoop
part 2 Getting data into
Elasticsearch
Analytics w ith Kibana and
Elasticsearch through Hadoop
part 1 Introduction
UKOUG Partner of the Year
Aw ards

What IS Regression Testing?

When we make a change to a system we use functional unit tests to ensure that it does do what it is supposed to do. We
should also make sure that the same changes dont do what theyre not supposed to, that is, cause functionality already
existing in the system to change behaviour. If this does happen it is known as a regression and is something we want to
ensure doesnt happen without us knowing. Some examples of regressions seen in standard OBIEE development changes
include:
Reports stop returning data, showing an error instead
Reports start to show the wrong data
Some combinations of dimensions and facts to no longer show data, or show an error

open in browser PRO version

Are you a developer? Try out the HTML to PDF API

Oracle BI Cloud Service for SaaS

Application Reporting Part 1:
Integrating BICS to
Salesforce.com using REST APIs

Top Posts
OBIEE 11g Security Week :
Managing Application Roles and
Policies, and Managing Security
Migrations and Deployments
Upgrading OBIEE to 11.1.1.7
OBIEE 11gR1 : Architecture and
Use of WebLogic Server
OBIEE 11g Security Week :
Connecting to Active Directory,
and Obtaining Group Membership

pdfcrowd.com

Some combinations of dimensions and facts to no longer show data, or show an error
Dashboards that reference a particular analysis stop working
As well as these, less common system changes can also cause regressions, for example:

and Obtaining Group Membership

from Database Tables
Analytics w ith Kibana and
Elasticsearch through Hadoop part 3 - Visualising the data in
Kibana

An OBIEE version upgrade causes certain types of graph to render in a different way from the previous version
An OBIEE patch introduces a bug in the front end user interface

What drives Regression Testing?

The requirement for regression testing OBIEE broadly comes from two different types of change:
New binaries that is, an upgrade (or patch) of OBIEE
New application code changes to the RPD, the underlying database schema, and so on.
These two requirements have the same aim make sure nothing breaks when we make the change but differ in ways that
make how we address them important:
1. Frequency : OBIEE may get patched once or twice a year, and upgraded every few years. Compare this to development
changes made to the RPD et al, which users would often like to see happening on a frequent basis (sometimes daily
at the beginning of an implementation). If these changes are happening with great regularity then (a) we dont want to
be the ones causing the bottleneck because we cant regression test them and thus (b) we need to find a repeatable
way to perform these tests accurately and quickly.
2. Delta Visibility : When Oracle change the OBIEE code base, we are blind as to what has gone on under the covers.
Sure, we know whats changed in the documentation, but as a starting point for what might have broken we can only
assume everything has and test accordingly. Conversely, in a planned development we know exactly what we
changed and we can therefore work out the scope of the necessary testing.
The points in bold above are what I aim to address in this article. Regression testing OBIEE doesnt have to mean one
technique alone it can be refined based on what we know has changed.

Random Posts
Visual Regression Testing of
OBIEE w ith PhantomCSS
Create a Mobile BI App for iOS,
Android or BlackBerry w ithin 5
Days!
Looking at the ODI12c Hadoop
Demos in the New Oracle
BigDataLite VM
New s and Updates from Oracle
Openw orld 2014
Oracle Database 12cR1 - New
Features for BI & DW

Tags

11g Big Data Appliance

BIP BI Publisher dw em12c

Endeca exalytics
extremebi git

goldengate
hadoop Hive init.d install
linux MDS XML monitoring
new features nqcmd OBIA

Why Regression Test?

obiee odi odi12c

If you dont regression test then you place a wager that youll b e ab le to fix any prob lems that arise. As soon as they arise. In
Production. With angry users on the phone. And the project manager screaming b lue murder b ecause their change is getting

Applications

b lamed for b reaking everything.

This is a recipe for compounding errors upon errors, not a stable system. Testing, in all flavours, is about gaining
confidence about the impact of a proposed change to a system. Functional testing reassures us that the change will do what

open in browser PRO version

Are you a developer? Try out the HTML to PDF API

opatch Oracle Oracle BI

oracle data

integrator Oracle
Endeca Oracle Endeca
Information Discovery ow b

performance Real Time

Decisions

pdfcrowd.com

it was designed to do. Performance testing helps us understand how a system behaves from a response time and capacity

Decisions replication

perspective. Regression testing gives us the confidence that a change, whilst doing what it ought to, isnt going to affect
something else.

ReportService RTD runReport

The confidence in what is (and isnt) going to happen when we deploy a change enab les us to make these changes more

sampleapp screen scripting

security startup testing

training XML

frequently as required by the users. Instead of a long development cycle with a huge number of changes bunched in
together, and one big bang test and release, we can take a more rapid, flexible, and responsive approach to development
and release because we have the confidence that an individual change is going to work.
In addition to confidence in additional releases to new deployments, a good regression testing framework enables us to have
confidence in making changes to long-standing big ball of mud systems. So long as we understand the relevant interfaces
points in OBIEE, we can build a pass/fail test framework on top of the most complex RPD/schema.

Targeting Regression Testing Effectively

Regression testing is easy. You pay a troop of monkeys to sit at a set of computers and run every single dashboard, build
every permutation of adhoc report, and if youve just upgraded or patched OBIEE, go through the user interface with a fine
toothed comb. After the appropriate period of several weeks, any differences they find from before your change was made is
a regression. Congratulations. All you need to do now is fix the problem and then of course, regression test your new
change. So monkeys are one option, but theyre expensive (you should see the wholesale market peanut price these days),
theyre not infallible (monkeys get distracted by YouTube too), and they are slow.
Better than monkeys is automated regression testing, targeted smartly at the area of OBIEE that has been changed. We will
now take a look at which changes can cause regressions in which area, and from that derive a list of testing methods
appropriate for each type of change made.

Regression testing points in the OBIEE stack

To understand how we can regression test OBIEE, let us look at where a regression can be detected. The following diagram
illustrates the request/response flow through the components in the OBIEE stack. We can use it to see where regressions
may expose themselves, and thus understand at what points we can consider testing for them.

open in browser PRO version

Are you a developer? Try out the HTML to PDF API

pdfcrowd.com

Starting from the point of view of the actual end user:

The user interface may regress. They may be actual bugs that werent there before, or regressions in the sense that
functionality or icons/layout have changed. These changes would typically only come about through software changes
(patching/upgrades).
Regressions could also occur if you are manipulating the UI through the analysis itself (eg narrative view) and the
behaviour changes, but this type of UI modification is less common.
Regressions caused by changes to the underlying data, RPD or analyses are going to manifest themselves through a
dashboard. This could be in the data or the presentation of the data (tables, graphs, etc).

open in browser PRO version

Are you a developer? Try out the HTML to PDF API

pdfcrowd.com

Considering a dashboard by its constituent parts, an individual analysis could exhibit differences in its data or the
presentation of the data

Next to consider is that each analysis sends a Logical SQL request to the BI Server. It is not common, but it is possible that
a change to the binaries (version upgrade/patch) could introduce a regression that caused the Logical SQL to be generated
incorrectly. Specific changes to the RPD can also cause the Logical SQL generation to change, potentially erroneously.

open in browser PRO version

Are you a developer? Try out the HTML to PDF API

pdfcrowd.com

The Logical SQL that is generated is executed by the BI Server which in turn returns the requested logical resultset data.
This resultset may expose a regression in how the BI Server is handling the logical request.

A Logical SQL request on the BI Server is parsed through the metadata layer, the RPD, and one or more Physical SQL
statements are sent to the underlying data source(s). An error in the RPD could result in the Physical SQL being generated
incorrectly.

open in browser PRO version

Are you a developer? Try out the HTML to PDF API

pdfcrowd.com

Finally, each Physical SQL request at the data source returns data back to the BI Server. Any errors in changes to the
physical sources will show themselves through the physical query failing, or the results being incorrect.

open in browser PRO version

Are you a developer? Try out the HTML to PDF API

pdfcrowd.com

Regression testing opportunities

To summarise the previous section, our testing points for regression are as follows.
1.
2.
3.
4.

The logical query generated by Presentation Services for an analysis

The physical query/queries generated by the BI Server to retrieve the data from the data source(s)
The data supplied by the data source to the BI server
The data supplied by the BI server for an analysis (logical resultset)

open in browser PRO version

Are you a developer? Try out the HTML to PDF API

pdfcrowd.com

5. User interface, including the dashboard/analysis, taking into account both rendered data and presentation/UI.

Regression testing is based around comparing one state (before a planned change) to another (after the planned change).
In considering how we are going to perform our testing, lets take a very simplistic view on what we need to test:
1. Does it look the same
2. Are the numbers the same
Of these two, one is very easy to get a computer to do (and conversely, very laborious to perform manually), and the other is
very difficult to explain to a computer (and relatively easy to do manually): Telling a computer to fetch some data twice and compare the first result with the second is bread and butter automation.
Trying to explain to a computer what a page looks like, or what a user interface does is extremely time consuming, and
inevitably specific to the single item in question. Of course, we can programmatically compare the underlying code for a
dashboard before and after a change, but the question I pose is whether we should.

open in browser PRO version

Are you a developer? Try out the HTML to PDF API

pdfcrowd.com

Computers are blind

The user interface for an OBIEE end user is a web browser, and OBIEE builds its web pages through a set of languages and
protocols that used to be quaintly referred to as Web 2.0. It uses HTML, CSS, XML, and JavaScript, taking plentiful
advantage of asynchronous page loading and in-flight modifications to the Document Object Model (DOM) too. AJAX is a
term which certainly covers some of the magic that goes on. The resulting user interface is pretty slick with drop down
menus, expanding hierarchy trees, and partial dashboard rendering as data is returned rather than waiting for all analyses to
complete. All of this omits the knockout blow that is Flash, used for rendering all graph objects in OBIEE and the subject of at
notable UI bug in OBIEE 11.1.1.6. The Developer Tools option in modern web browsers gives us a glimpse into what is
going on under the covers. We can see the number of resources that go into rendering a single page

open in browser PRO version

Are you a developer? Try out the HTML to PDF API

pdfcrowd.com

and how many layers there are to the object model:

open in browser PRO version

Are you a developer? Try out the HTML to PDF API

pdfcrowd.com

Getting a computer to interface with all of this, simulating a user interaction and parsing the response is possible with
functional testing tools such as Selenium, Oracle Application Testing Suite, and HPs QuickTest Professional. Each of these
tools is capable of simulating a user (often by recording a session as the starting point) and parsing the responses from
OBIEE.

open in browser PRO version

Are you a developer? Try out the HTML to PDF API

pdfcrowd.com

But, there is a fundamental complication to using these tools. For all the AJAX/CSS/DOM magic to happen, the page that
OBIEE generates is littered with element identifiers (so that the JavaScript code can identify the element to manipulate). For
example, the following table cell has the ID in this particular execution of e_saw_14485_10_1_0_0:

Some of these IDs may change between report executions or sessions, but either way, cannot be relied on to be consistent.
Therefore, getting our testing tool (such as Selenium) to compare the before/after results to detect a regression becomes a

open in browser PRO version

Are you a developer? Try out the HTML to PDF API

pdfcrowd.com

whole heap more tricky. It is possible to work out element paths based on their relative position within the page rather than
an ab solute ID, but that becomes even more page specific and complex to implement. Therefore to compare a before and
after page programatically we have to either
define a particular part of the page alone to check remains the same (and risk chucking the baby out with the bath
water, that is, missing other genuine regressions elsewhere on the page)
or we have to
compile a list of elements that we expect may change but that we dont count as a regression (i.e. exceptions).
The latter is going to be prone to causing false positives (i.e. failing regression tests that arent genuine regressions)
because it relies on reverse engineering the full Document Object Model of the OBIEE page. All of this is also without even
taking into account software patching and upgrades so far as Oracle are going to be concerned how a page is rendered is
their own business and thus at full liberty to completely change the internal structure of a page as they desire. Given this
above complication, it becomes clear that building a test against a single page is time consuming, and it will typically be
specific to that page only. This becomes a problem the greater the scale of the deployment you are trying to test. Hardcoding
the testing for one specific page might be fine, but given more than a handful of pages you risk ending up with a large
inflexible regression test code base (that itself may become error prone and need regression testing when its changed).

Conclusion
So, we come back to not how we test the front end but more should we, in every case? Given a finite amount of time, what
are you going to get most benefit from in your regression tests? In the next post I will demonstrate one of the ways you can
get the most bang for your buck when regression testing OBIEE, by concentrating your automation efforts on the query part
of the OBIEE stack, and not the front end. Stay tuned!

Many thanks to Gianni Ceresa for his thoughts and assistance on this sub ject.

Related Posts:
Automated Regression Testing for OBIEE
Visual Regression Testing of OBIEE with PhantomCSS
Incremental refresh of Exalytics aggregates using native BI Server capabilities

open in browser PRO version

Are you a developer? Try out the HTML to PDF API

pdfcrowd.com

Tags: bi server, logical sql, obiee, Presentation Services, regression, selenium, testing
Share

10

Tw eet

33

Like

Tags: bi server, logical sql, obiee, Presentation Services, regression, selenium, testing
Posted in Oracle BI Suite EE, Testing | 5 Comments

Comments
Gerard Nicolas Says:
January 18th, 2014 at 12:40 pm

Hi Robin,
Beautiful article with a beautiful start picture. (The baby one where we can see a baby regression testing from a proud
father)
I have a couple of remarks. First, the bread and butter link doesnt work.
The second remarks is on the regression testing definition that you give:
Regression testing is based around comparing one state (before a planned change) to another (after the planned
change).
For me, regression testing was just to ensure that we didnt have introduce an error by resolving one.
Therefore, running all the unit tests (differences between what you get and what you expect) were sufficient. Why do you
need to compare ? The before change state may be wrong.
I stay tuned, promise.
Good weekend

Robin Moffatt Says:

January 18th, 2014 at 3:13 pm

open in browser PRO version

Are you a developer? Try out the HTML to PDF API

pdfcrowd.com

Hi Nico,
Thanks for the comments. Ive fixed the link.
Regarding your second comment, if Ive understood correctly, youre saying that regression testing could actually be done
by re-running unit tests and any failures would indicate a regression? If so, I would agree, but (a) not all *cough* BI
projects have unit tests written for them, (b) unit tests will often not have 100% coverage, and (c) by comparing before and
after states, even if before is wrong, at least it will still be correctly wrong ;-)
Hope that makes sense, let me know if Ive misunderstood your point.
cheers, Robin.

Daniel Says:
January 23rd, 2014 at 11:28 am

Hi Robin, spot on. I have used this approach taking advantage of OBIEE web services to test OOTB, custom and super
user personal reports.
A few times, we found false positives or fake defects where a report works correctly in the upgraded environment
(wrong results largely undetected in the base environment).
In my experience, the ability to discover data discrepancies before the users while applying a patch or going for an
upgrade is priceless.
Looking forward to the 2nd part.
Cheers,
Daniel

Bibin Kumar Says:

January 23rd, 2014 at 7:09 pm

Robin, excellent article. You have brilliantly explained the nuances of testing BI Applications. Typical customer base of
500 odd reports estimating regression testing effort is difficult. I look forward to future articles that involve test
automation and how to measure a before & after delta.
cheers!
Bibin Kumar

open in browser PRO version

Are you a developer? Try out the HTML to PDF API

pdfcrowd.com

Todd Ryan Says:

January 31st, 2014 at 8:14 pm

Very good suggestions/comments. We use Loadrunner to script some of this with each release, helps find issues and
validate performance..

Call us now to talk about your BI project:

+44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA)
or +61 3 9596 7186 (Australia & New Zealand) or
+91 997 256 7970 (India) or +32 280 882 11 (Belgium)
Hom e

Services

About Us

> Consulting
> Training
> Support

>
>
>
>

About us
About our team
Contac t us
Our clients

Consulting
Services
>
>
>
>
>
>

Projects
Expert Services
OBIEE 11g
Sustainability
On Discoverer?
Orac le DW

Training

Resources

Blog Authors

> OBIEE
Bootcamp
> OBIEE End-User
> Exalytics
> ODI 11g
Bootcamp
> Oracle BI Apps

> Articles
> Blog
> OBIEE 11g

>
>
>
>
>
>
>

Mark Rittman
Venkat J
Peter Scott
Borkur S
Mike Vickers
Robin Moffatt
Jon Mead

Rittman Mead Consulting ltd.

Registered Office : Suite B,
First Floor Moore House,
13 Black Lion Street,
Brighton, East Sussex,
BN1 1ND, United Kingdom
Company No. : 6032852
VAT No. : 900 3839 48
Rittman Mead America, Inc.
Registered Office : 4550 North Point Parkway Suite
390 Alpharetta, Georgia 30022, USA
Rittman Mead Oceania Pty Ltd.
Registered Office : 12 Moore Street,
Brighton East,
Victoria, 3187, Australia
Australian Company No. : 149 458 935
Rittman Mead Consulting Pv t Ltd.
Registered Office : Unit 105-106
Regent Prime
Whitefield Main Road

open in browser PRO version

Are you a developer? Try out the HTML to PDF API

pdfcrowd.com

Whitefield
Bangalore
560066
Rittman Mead Belgium
Registered Office : Chausse de Louvain 426
1380 Lasne
Belgium

2010-2011 Rittm an Mead Consulting.

Privacy Policy

E: info@rittm anm ead.com

Website Design & Build: tymedia.co.uk

open in browser PRO version

Are you a developer? Try out the HTML to PDF API

pdfcrowd.com