Agile Approach in Data Intrinsic Development

developerWorks article template using

Microsoft Word

Type of Submission: Article

Title: Agile Approach in Data Intrinsic Development
Keywords: Agile Methodology, RUP
Topics:
Product Family: Rational
Products: Rational Unified Process
Skill Level: 1

Given: Dinesh
Family: Mohata
Job Title: Senor Consultant
Email: dinesh.mohata@in.ibm.com

Dinesh Mohata is a Senior Consultant in the Oracle Retail Practice within Distribution Sector.
The practice is responsible for providing project management, end to end services in retail
industry, and business solutions. Dinesh has over 9 years of IT & consulting experience in
software design, development, deployment and testing. He has retail industry experience of
over 5 years. Dinesh has played a role of Technical Architect.

His areas of interest include Agile Development Methodology, Design Patterns &
Implementation Data Cutover. He has earlier published a paper “Application Agility: A Design
Perspective” with Oracle. Dinesh can be reached at dinesh.mohata@in.ibm.com

Company: IBM India Private Limited

Photo filename: DineshMohata.jpg

Abstract:
This paper details out various Agile Software development methodologies like XP, SCRUM,
RUP etc. The second half of paper details out agile approach taken in one of the real life
projects. It covers the background of the project, approach taken, benefits gained, challenges
and conclusions. This paper will benefit the audience in providing varied perspective for agile
adoption. The tailoring done to the agile approach for data intrinsic development will definitely
provide users with some tips to take and may implement in their projects as well.

Page 1 of 14
Agile Approach in Data Intrinsic Development

Page 2 of 14
Agile Approach in Data Intrinsic Development

Table of Contents

1. Introduction............................................................................................................................ 3
2. Agile Software Development Methodologies..........................................................................3
2.1. Extreme Programming (XP)........................................................................................... 3
2.2. SCRUM......................................................................................................................... 4
2.3. Agile Modeling (AM)...................................................................................................... 5
2.4. Agile Data (AD).............................................................................................................. 5
2.5. Rational Unified Process (RUP)....................................................................................5
2.6. Open Unified Process (OUP)......................................................................................... 5
3. Agile Approach....................................................................................................................... 5
4. Case Study: Traditional to Agile Approach in Data Intrinsic Development..............................6
5. Benefits Gained.................................................................................................................... 11
6. Challenges........................................................................................................................... 12
7. Conclusion........................................................................................................................... 12
8. References........................................................................................................................... 13

Page 3 of 14
Agile Approach in Data Intrinsic Development

1. Introduction

Agile software development challenges traditional software development approaches. Rapidly

changing environments characterized by evolving requirements and tight schedules require
software developers to take an agile approach. These methods include practices such as short
iterations, frequent releases, simple and emerging design, peer review, and on-site customer
participation.

Traditional software development methodologies (SDMs), also referred to as software

development processes, are being replaced by light or agile SDMs. Agile SDMs are characterized
by iterative development, continuous code integration, and the ability to handle changing
business requirements [1].

Extreme Programming (XP) is the most well-known agile methodology. XP is based on a series of
concepts that include: having the business customer on-site, pair programming, collective code
ownership, continuous code integration, small releases, designing tests before writing code,
standup meetings, refactoring, and 40-hour work weeks [2].

Disciplined agile software delivery processes are Dynamic System Development Method
(DSDM), Open Unified Process (OpenUP), and agile instantiations of the IBM Rational Unified
Process (RUP) process framework. All of these methodologies function differently from traditional
SDMs and help organizations meet the challenges of the digital economy [1].

This paper covers the background of the project, approach taken, benefits gained, challenges
and conclusions.

2. Agile Software Development Methodologies

Agile methodologies are used to produce higher quality software in a shorter period of time. Agile
methodologies were developed to streamline the development process and remove barriers to
accepting business requirement changes during the development process. Agile methodologies
do not require that business requirements and design details be locked in for the duration of
development [3]. Agile SDMs share several features including prototyping, iterative development,
and minimal documentation [4].

2.1. Extreme Programming (XP)

XP is a collection of practices for software construction, include refactoring, test-first design, pair
programming, on-site-customer, continuous integration, whole team, and collective ownership.

XP Practices

o Test driven development:

o Focusing on Thinking about what the code should do before doing
à High Quality and defect minimization
– Regression test baseline
– Create confidence in development team
o Refactoring:
– Improve quality, reducing code entropy

Page 4 of 14
Agile Approach in Data Intrinsic Development

– Re-organize design, to lessen the cost of change

o Simple Design:
– Avoid paying the cost of potentially useful, but not business critical
feature.
– The design evolves according to client business requirements
o Pair Programming
– High Quality standards through constant Code Review
– Cross Training of the whole team

2.2. SCRUM
The focus of Scrum is project leadership and requirements management. Scrum proponents
claim that it is a process framework, but if so then it is a sparse and incomplete one at best. A
more accurate description is that it is a high-level lifecycle for construction iterations (what Scrum
calls “sprints”), see Figure 1, and several practices such as daily stand-up “Scrum” meeting,
product owner, product backlog, iteration/sprint planning, and potentially shippable software.

Figure 1 Scrum Construction Lifecycle

alt=Agile lifecycle as used in SCRUM

The above figure has been referenced from http://www.ambysoft.com/essays/agileLifecycle.html

Scrum Approach

 Accept:
o This is a complicated, unpredictable world and development environment.

Page 5 of 14
Agile Approach in Data Intrinsic Development

o You can't predict or definitively plan what you will deliver, when you will deliver it,
and what the quality and cost will be.

 Assume:
o You can estimate, and then negotiate according to various risks as you proceed.

 Understand:
o Following any cookbook approach won't improve the definition of "best", and will
only hinder appropriate responsiveness to the complexity and unpredictability.
o the delivery will be the best possible given the circumstances

The above Scrum Approach content has been referenced from http://www.controlchaos.com/old-
site/philo.htm

For implementing scrum visit www.implementingscrum.com for further details.

2.3. Agile Modeling (AM)

AM is a collection of practices for light-weight modeling and documentation, including
requirements envisioning, executable specifications, active stakeholder participation, prioritized
requirements, and prove it with code. See www.agilemodeling.com for details.

2.4. Agile Data (AD)

AD is a collection of practices for database development, including agile data modeling, database
testing, database refactoring, and continuous database integration. See www.agiledata.org for
details.

2.5. Rational Unified Process (RUP)

RUP is a comprehensive process framework for iterative software delivery which can be
instantiated anywhere from a very agile form to a very traditional form as your situation warrants
[6]. RUP practices include risk-value lifecycle, whole team, test-driven development (TDD), and
business process. See http://www-01.ibm.com/software/awdtools/rup for details.

2.6. Open Unified Process (OUP)

OpenUP combines and extends practices from Scrum, XP, AM and RUP for co-located agile
teams which are building business applications. OpenUP practices include whole team, daily
standup meeting, prioritized work items, risk-driven lifecycle, TDD, active stakeholder
participation, and continuous integration. See www.eclipse.org for details.

3. Agile Approach
Agile Approach has to be tailored to the needs of the project one is working upon. Agile process
adopted in this project was culmination of various precepts pronounced in various agile SDMs.

Page 6 of 14
Agile Approach in Data Intrinsic Development

Various precepts applied in the current work with respected to Agile SDM were as follows:

 Requirements Envisioning
 Redefining Stake Holder Participation
 Incorporation of Late Requirements
 Work Prioritization
 Continuous Integration
 Pair Programming
 Refactoring
 Cross Training of whole team
 Applying TDD
 Technical Excellence

The above approach is explained in detail with all due examples in the subsequent case study
section of this paper.

4. Case Study: Traditional to Agile Approach in Data

Intrinsic Development
A retailing client made their first foray into ERP implementation by implementing Oracle packaged
solution a.k.a. Oracle Retail. This program has been aimed to implement various modules of
Oracle Retail suite formerly called Retek. Oracle Retail Suite will form the technological backbone
of their retail venture. The process of this retailer is quite ad-hoc in nature. The IT infrastructure
required to make this has to be robust and scalable to meet the dynamic changing requirements
for the customer. This project will transform and streamline the business and IT processes for this
retailer in lines with world class best practices in Retail industry

Retail is all about business agility. Requirements change on the fly. To beat the competition
retailer has to change the dynamics of its organization at rapid pace. This requires systems and
processes to be agile. IT has to be an enabler for this happen. This in turn requires IT systems to
follow sound agile process to deliver value to the customer in the minimum time possible. Retail
being highly margin sensitive area retailer has to capture lot of data to optimize its pricing engine
to get the best value to the customer and in turn protect its margins. This requires huge data
churning within the retail organization. This required defining and proving the agile processes
work in packaged implementation. Moreover, the sound agile processes adopted during the
implementation will become the sound backbone for the retailer for future.

This section details about the various facets of agile software development applied for this
retailer. The tenets of agile development have been picked and experience around this has been
penned down in the paragraphs below.

Requirements Envisioning

Ascendant Methodology was adopted to do requirements envisioning. During inception of the

project a high-level requirements modeling was done to understand the scope and identify the
initial stack of requirements. This project being a packaged implementation, it was critical to
understand the fitment of the customer requirements with the product. Any deviations were duly
captured and communicated to the customer as gap. This also made the implementation team
aware of the gaps coming forward beforehand and just in time.

Redefining Stakeholder Participation

Page 7 of 14
Agile Approach in Data Intrinsic Development

Agile approach in traditional environment is really tough. People are not willing to understand the
agile approach as such. Customer is not fully aware of the approach and has no willingness to
here it as well. Approach taken was internally we will satisfy our customer i.e. functional
consultants using agile approach. Agile approach was to internal customers of the project team.
Stake holder for the development team is the internal teams which certify the product before
releasing to the external stakeholder.

Incorporation of Late Requirements

Initial requirements were prototyped. The design was envisaged keeping in mind the changes
coming late in the project to be incorporated. Each report was driven by an Oracle Database
View. Report were predefined by customer in ms excel templates. These templates were
uploaded in the database. Any additional column required a change in template and subsequent
upload a CLOB database object type. Since each report was based of database view it was just
matter of adding the additional column in the view from the table already in use in the view
definition.

Work Prioritization

Changing the stakeholder definition to internal customer helped team to prioritize the work. Entire
team was co-located and any suggestions on work priority were discussed right at the moment to
decide team’s future course of action. Real stakeholders were intimated on the details of the work
scheduled for the team. Communication to real stakeholders was done only after having internal
stakeholder approval.

Continuous Integration

Continuous integration was key approaches adopted in the data intrinsic development. In all the
instances the code was deployed based on few iterations of develop, build and test cycle. The
development was done based on requirements provided by the customer. The requirements were
then prioritized. Based in these prioritizations the development would work with iterations of
development. The end product was continuously integrated with the bigger system to ensue this
integration does not break the larger system. This resulted in smooth transition during the
movement of products in production environment.

There was an instance wherein application was developed in the corporate environment using
intranet network. This application was supposed to be deployed at a remote location for the
customer. This application was working perfectly fine when the server was accessed using the
local area network (LAN). But application just was not working when deployed in the remote
location. The connectivity to the server was not appropriate for certain protocols required in the
application. This was a serious issue since without opening up this protocol in all the routers to
server this application wouldn’t work. Design was re-organized to attain this in two step process.
This design was also thought through with customer in mind. The process using the protocol was
replaced with a two step process. The first step was making a direct database entry of the data to
be transported from local client to the server. The second step was creation of the file at the
server side using database server file utilities. These two steps isolated the protocol error
occurring in the application. Cost of change was quite less since this kept the other steps in the
application intact and re-designed a portion of the application process.

During the one of the demonstrations of the code developed during the project the agile approach
was quite handy. Two team members from the team were executing the demonstration to the
customer. One of programmers was displaying the features of the code and performing the initial

Page 8 of 14
Agile Approach in Data Intrinsic Development

tests for the customer on the developed application. This demonstrator was connected to the
projector for the entire audience. Another programmer was sitting just beside and noting the
comments of the audience. There was couple of enhancements requested by the customer
during this session. While the first demonstrator was demonstrating the application, the second
developed already coded these new features in the code and deployed the same code during the
recess break of the session. Post the recess break once more the application walkthrough was
conducted. This walkthrough resulted a huge success maximum of changes requested by the
customer was successfully implemented and demonstrated within a very short duration.

Pair Programming

There was instance in the project were in we had to deploy an unknown application. This
definitely required some initial training to get the job done. Customer arranged for online CBT
training but this did not help. We did form a two member team to get the job done. There was one
programmer doing the typing effort and another person religiously followed the application
deployment guideline to guide the typist. This combined development effort paid off and we were
successfully able to implement the application in very short duration. As a side note I would
emphasize that this required good resources with a good attitude to get the job done.

Cross Training of whole team

All the members in the team were provided an opportunity to work within other modules of the
project. The whole team was collocated in a single venue. Team sitting in a single location
actually helped in various ways. Discussions between team members of various modules by and
large happened in the same room. This resulted in each and every individual having a fair amount
of idea on each module being implemented. As in IBM servers, it applies to humans as well. Not
all the time each individual is fully engaged. IBM virtualization concept utilizes spare processing
power from servers in the stack which are not in use. Applying the same concept in the co-
located, it helped in filling any immediate requirement from a module which would require some
extra effort for any dynamic requirement. This in turn also gave some amount of knowledge to the
cross team member who approached to solve the issue.

Applying Test Driven Development

TEST DRIVEN DEVELOPMENT (TDD) LIFE CYCLE

The below diagram explains the TDD process lifecycle.

Figure 2 Test Driven Development

Page 9 of 14
Agile Approach in Data Intrinsic Development

ADD A TEST

Pass

RUN ALL TESTS

Fail

Pass
WRITE/REFACTOR CODE

Fail

RUN THE TEST

alt=Test Driven Development Lifecycle

The above content has been referenced from http://www.agiledata.org/essays/tdd.html

Add a test

In test-driven development, each new feature begins with writing a test. This test must inevitably
fail because it is written before the feature has been implemented. In order to write a test, the
developer must understand the specification and the requirements of the feature clearly. In all the
developments developer was provided with use cases and stories to cover requirements and
exception condition. This made developers aware of the requirement before writing the code.

An instance of test in current scenario wherein a development of Retail application for financial
accounting was developed. User story of this application was beforehand given to the developer.
The requirement was to develop application that sends data on daily basis of certain attributes

Page 10 of 14
Agile Approach in Data Intrinsic Development

and thereafter updation of flag at the subsriber end. The test condition was written by the
developer stating the attributes of the database required to pull the data from. If any of the
columns required fo certain table does not exist then this test condition will fail. The code had to
be rewritten to take the appropriate column from the table for the test to pass.

Run all tests and see if the new one fails

This validates that the test harness is working correctly and that the new test does not mistakenly
pass without requiring any new code.

The new test should also fail for the expected reason. This step tests the test itself, in the
negative: it rules out the possibility that the new test will always pass, and therefore be worthless.

Write some code

The next step is to write some code that will cause the test to pass. The new code written at this
stage will not be perfect and may, for example, pass the test in an inelegant way. That is
acceptable because later steps will improve and hone it.

It is important that the code written is only designed to pass the test; no further (and therefore
untested) functionality should be predicted and 'allowed for' at any stage.

Run the automated tests and see them succeed

If all test cases now pass, the developer can be confident that the code meets all the tested
requirements. This is a good point from which to begin the final step of the cycle.

Refactor code

Now the code can be cleaned up as necessary. By re-running the test cases, the developer can
be confident that refactoring is not damaging any existing functionality. The concept of removing
duplication is an important aspect of any software design. In this case, however, it also applies to
removing any duplication between the test code and the production code.

Repeat

Starting with another new test, the cycle is then repeated to push forward the functionality. The
size of the steps can be as small as the developer likes, or get larger if s/he feels more confident.
If the code written to satisfy a test does not fairly quickly do so, then the step-size may have been
too big, and maybe the smaller testable steps should be used instead.

The first step is to quickly add a test, basically just enough code to fail. Next you run your tests,
often the complete test suite although for sake of speed you may decide to run only a subset, to
ensure that the new test does in fact fail. You then update your functional code to make it pass
the new tests. The fourth step is to run your tests again. If they fail you need to update your
functional code and retest. Once the tests pass the next step is to start over (you may first need
to refactor any duplication out of your design as needed, turning TFD into TDD).

I like to describe TDD with this simple formula:

TDD = Refactoring + TFD. [5]

Page 11 of 14
Agile Approach in Data Intrinsic Development

The above content has been referenced from http://www.agiledata.org/essays/tdd.html

This Test Driven Development approach was followed across on new developments done in the
project.

Technical Excellence

Customer had proposed for development of 16 reports as part of SOW at an implementation

project at a large India based retailer. Since the reports were not classified for its complexity the
customer proposed very complex reports for development.

The challenges for the report development

1. Each report was culmination of three to four medium complex reports

2. Excel output was required for the reports which was not in purview of development
effort/methodology
3. Huge Technical Challenges

Based on the requirements the report development was started developing the skeleton or
framework for one of the reports. Brainstorming was conducted on technical challenges of the
report and work for the same started. The working report was ready in a day. There were issues
of performance in the report as far as the data display in the screen was considered. The data
was displayed in the spreadsheet on a cell by cell basis. Oracle’s Webutil technology was
leveraged to write data in the excel sheet.

Following steps were taken to foresee any bottlenecks and combat well in advance

 Since each report was based on a database view, SQL tuning the underlying view
database query was indispensable.

 Server Statistics report collected in daily basis check the server health and apply
appropriate check if necessary

 Database performance was continuously monitored by modifying database parameters

 Collecting the data base statistics reports.

 Each SQL session and application session was traced. Oracle database utilities were
used to analyze the performance impact of session parameters

This approach of continuous integration and collective ownership as a team was adopted to
benchmark the technical approach in an agile way.

5. Benefits Gained

Page 12 of 14
Agile Approach in Data Intrinsic Development

By following the agile-tional approach to software development for data intrinsic development
following was gained

 Design and Build of reports were prototyped resulting in creation of working software
before discussing requirements further with customer
 Effort spent on creating this working software was restricted to two days only
 New alternatives/suggestions were advised to customer on first meeting itself
 Realistic expectations were set for the customer from day one based on the technical
limitations of the development
 Functional fitment done well in advance because of requirements envisioning
 Ability to take any additional touch-ups on the code as along as the framework of the
development does not change
 Customer was very satisfied on the approach of creating prototype in initial phase
 Requirements late in the project were easily incorporated
 The working reports were used by the customer for extensive data analysis from their
end.
 They used the reports to further verify the data in their system
 On the fly change during presentation using Pair Demo as part of continuous integration
was just excellent and was highly appreciated by the customer
 Pair Programming resulted in huge customer satisfaction
 Quick Results
 High Quality standards were set through refactoring codes
 Good Clean code that works
 Agility in the Software development process
 Huge confidence in the development team

6. Challenges
There were many challenges faced in journey of applying agile-tional approach to development

 It was difficult to make team agree on the agile approach since the school of thought
was more towards waterfall model
 Getting the buy in from the Project Management was tough and hence workaround
stakeholder idea was devised
 Technical Challenge of publishing an excel report from Oracle 10g server
 Performance of the reports in general and this was dealt with continuous monitoring
of database and application
 Regression Testing of deliverable upon incorporating late feedback

7. Conclusion
Software development can be done in agile way bringing delight to the customer as an end result.
It provided quick turnaround time to deliver quality product to the customer. Willingness from the
all stakeholders is indispensable for successful agile methodology implementation. Requirements
late in the game can also be incorporated with agile SDM. Customer is fully aware of the end
product getting delivered to him.

Agility is more of perspective. It’s really just a mindset. It’s about adaptability to changing
business requirements. Based on the points highlighted in this paper, and supporting points
given, it could be very well stated that agile SDM can be adopted in an organization whilst

Page 13 of 14
Agile Approach in Data Intrinsic Development

providing huge benefits and requiring few cultural changes that will greatly impact the
implementation.

8. References
1. Boehm, B. & Turner, R. Management challenges to implement agile processes in traditional
development organizations. IEEE Software. 22(5), 30-40. 2005.

2. Theunissen, W., Boake, A., & Kourie, D. In search of the sweet spot: Agile open collaborative
corporate software development. Proceedings of the 2005 Annual Research Conference of
the South African Institute of Computer Scientists and information Technologists on IT
Research in Developing Countries. White River, South Africa. 268-277.

3. Lindstrom, L. & Jeffries, R. Extreme programming and agile software development

methodologies. Information Systems Management. 21(13), 41-53. 2005.

4. Holmstrom, H., Fitzgerald, B., Agerfalk, P., & Conchuir, E. Agile practices reduce distance in
global software development. Information Systems Development. 23(3), 7-18. 2006.

5. Agile Database Techniques by Scott Ambler

6. Kroll, P. and MacIsaac, B. (2006). Agility and Discipline Made Easy: Practices from OpenUP
and RUP. Boston: Addison-Wesley

Page 14 of 14