Agile 2008 Conference
Bootstrapping Scrum and XP under crisis
A story from the trenches
Henrik Kniberg, Reza Farhang
Crisp AB, Sweden
henrik.kniberg@crisp.se, reza.farhang@crisp.se
Abstract
During 2006 Tain, a Swedish gaming company,
underwent a fast and dramatic agile transitioning
process driven by a crisis situation. Many of the
lessons learned are described in the book “Scrum and
XP from the Trenches”[1]. This experience report
focuses on the actual bootstrapping process – the
critical decisions and changes made during the first
few months that ultimately transformed a burning and
sinking ship into a fairly well-oiled agile software
development organization.
1. Introduction
Tain is a gaming software company based in
Stockholm. In December 2005 Henrik was contracted
as chief of development to improve the development
process. At that time Tain’s largest and most business
critical project was in a state of crisis. The purpose of
the project was to develop and launch an online poker
gaming system, which at the time involved 42
developers and testers (more than half of the entire
development organization). Reza was leading the 6-
man client team in the poker project.
2. The crisis
The online poker system had been launched to
production prematurely a few months earlier. It
crashed many times every day, causing immense
negative publicity and loss of money to Tain, and
consequently immense stress on the pokerdev team.
The strategy so far had been to throw bodies at the
problem – get more people in, work longer hours.
PDD – pain driven development. This approach
proved to be expensive, unhealthy, and ineffective.
978-0-7695-3321-6/08 $25.00 © 2008 IEEE
DOI 10.1109/Agile.2008.34
When Henrik arrived in Dec 10 2005 pokerdev was
essentially a team of ghosts – pale, stressed, sleep-
deprived developers and testers. Several people had
already been burned out and left the company. The
rest worked insane amounts of overtime. One person
even submitted a formal vacation application for the
upcoming weekend – this was symptomatic.
On December 15 Tain hosted Poker SM, the Swedish
national online poker championship, a highly
publicized event involving thousands of players and a
prize pot of €320,000. In the middle of the tournament
the poker system crashed, causing the entire event to
be canceled. The media had a frenzy.
The media was notified that Poker SM would be re-
run on Feb 1.
A few days after the crash top-level management
decided that they no longer could afford so many
contractors. Henrik was notified that over half of the
pokerdev team would have to be terminated on short
notice. At the same time several new developers would
be hired.
3. Our mission
Our mission was fairly clear. In priority order:
· Succeed with Poker SM on Feb 1 (six weeks away)
· Make the poker system stable
· Make the poker system scalable
· Create an effective and humane work environment
We knew that if we didn’t succeed with the first goal
we wouldn’t be given another chance. If we failed the
pokerdev team would most likely be fired and the
company’s survival would be at stake. We also knew
that, even if we did succeed with Poker SM, we
needed to succeed with the other three goals as well
within a few months or face another crisis.
436
4. Our situation 4.2 Organization

4.1 Interview results The pokerdev organization looked something like this:
PokerDev organization
It was impossible to lead the team without Server Server
understanding the current situation better, so Henrik gamelogic platform
started by interviewing most people involved.
Marketing
Developers, testers, stakeholders, the product owner,
etc.
Support Backoffice Client
The interviews took a couple of weeks. During this
period Henrik started understanding who the key
people were and spent time discussing possible Operations
strategies with them. This was how he discovered that DB Test
Reza was a strong informal leader, so from that point
on Reza and Henrik essentially worked as a pair – CFO
Reza from “inside” as developer and team leader,
Henrik from “outside” as manager and coach. This
Customers
turned out to be a fruitful combination.

The interviews revealed a surprisingly consistent and
dismal picture. The most frequently repeated
comments were:
· “I’ve been doing overtime consistently now for
many weeks, I am about to crash and burn.”
· “I have no idea what is going on – who is doing
what, or what I am supposed to be doing”
· “This is the worst code I’ve seen in my entire
life”
· “I can never get anything done because I keep
getting interrupted by system crashes or changed
priorities”.
At the same time it was surprising to hear that most
team members were actually highly motivated, and
had some fairly good ideas about how they could make
the system more stable. They cared about the product.
They liked poker. They wanted to succeed!
Teams were divided by component or role. There was
an open office area with 4 teams:
· Server gamelogic team
· Server platform team
· Backoffice team
· Client team
Two additional teams had their own rooms:
· DB team
· Test team
There were several people that weren’t associated with
any specific team and had unclear roles – for example
“application manager” and “release manager”.
Requirements and requests were continuously raining
in from different stakeholders such as marketing,
support, operations, the CFO, and customers. Each
with their own agenda. Stakeholders would often
physically enter the room and make demands directly
to specific developers.

Releases and patches were made on a team-by-team

basis without adequate testing and integration, causing
vicious release-crash-patch cycles. There was a sense
of panic in conjunction with each release. There was
no stable baseline to release from, instead each release
was manually assembled as a patch.

437
The testers had very little communication with the
developers, despite the fact that their room was only
20 meters away.
4.3 Summary of our situation
Forces working against us:
· Severe staff reduction. Pokerdev team was being
reduced from 42 to 16 due to quick roll-out of
contractors. 4 new employees were starting
during January.
· Overworked team, at the brink of burn-out.
· Chaotic and ineffective process.
· Short time frame – 6 weeks.
· Immense external pressure.
· Terrible code base
Forcing working to our advantage:
· 100% support from top-level management.
· Motivated team – they wanted to succeed.
· Single clear goal shared by the whole team.
· Willingness to change.
5. Our strategy
Our initial strategy was very simple:
Implement Scrum fast!
The reasoning behind this was:
· We didn’t know the best solutions to our
problems, so we would need to experiment and
adapt.
· The pokerdev team appeared to have the
motivation and skills necessary to solve their
problems. They just weren’t being given a chance
to focus.
· We needed to do something fast. The situation
could hardly get any worse, so any kind of
change would be good.
· One of the other product teams (Casino
development) had been using Scrum successfully
for a few months already. A small team, but they
were happy with it.
· Henrik had positive experience with agile
methods from previous engagements and
believed Scrum was a good fit to the types of
problems this organization was facing.
438
6. Getting our nose above the waterline
6.1 First step – product backlog
There was no point doing anything else until we knew
our focus and priorities. So we started by trying to
create a product backlog.
Here we were lucky. The poker product management
group was strong, they were domain experts and
experienced in Scrum. Since there were several people
in the group we managed to agree on having one
person be the official product owner for poker software
development.
We already knew the primary goal - make the system
stable enough to run poker SM.
Through dialog with key developers and the product
owner we quickly came up with a list of key issues that
needed to be fixed in order to increase stability. Those
became the top items in our product backlog.
It became quite clear at that point that the team had
been wasting a lot of time developing stuff that was
quite unimportant compared to the stability issue.
6.2 Second step – focus
In addition to the top items, the product backlog
quickly filled up with hundreds of other features and
improvements and bug fixes that various stakeholders
had requested (including the developers themselves).
Here we made a key decision and communicated to the
whole company:
The pokerdev team will now focus 100% on system
stability. Until that is fixed we won’t work on anything
else. All requests must from now on be brought to the
product owner, not directly to the team. The product
owner will place each request in the product backlog
and assign a suitable priority, which means terribly
low priority for everything except stability
improvements.
This generated a lot of friction. But we managed to
hold the fort thanks to support from top-level
management, good work from the product owner, and
general awareness throughout the company that we
need to focus to survive.
6.3 Third step – reorganize
We still had four problems in our way:
· It was not feasible for everybody to work on
stability issues. So what should the rest be doing?
· How could we focus? The system was still
crashing every day, causing immense disruptions
as the developers had to scramble to implement
hot-fixes.
· No reliable release process and no stable baseline
to release from.
· Overworked team.
Our strategy was to completely reorganize the teams.
· Move everybody into the open office area, bring
the testers out of their cave and seat them with
the developers.
· Create two cross-functional teams:
o One “fire prevention” team that focuses on
fixing the underlying stability problems.
o One “fire fighting” team (called the support
team) that focuses on protecting the first
team.
· Make the test team virtual. Each tester is
dedicated to one of the two cross-functional
teams. But the test lead can pull together the
entire test team when necessary to do full
regression testing in conjunction with major
releases.
· No more overtime. As manager, Henrik
explicitly discouraged (but didn’t forbid) people
from working evenings and weekends. Working
smart was more important than working hard.
· No releases or patches could be done without
involving the test lead and product owner. The
test lead should at least know the quality of each
release, and the product owner has final say on
whether the release should be made or not.
· All teams would reserve time to fix the baseline,
improve our version control system, and find a
better release process.
The fire prevention team would be a full-fledged
Scrum team.
The support team, however, would be almost 100%
reactive – continuously fixing problems as they
occurred. The support team couldn’t have a product
backlog and had a planning horizon of at most one
day, so we concluded that all-out Scrum wouldn’t be
439
useful to them. They did however do daily meetings
where they discussed and prioritized today’s work, so
in a sense they were doing one-day sprints.
We rearranged the office so that the support team was
seated physically between the scrum team and the
door. That meant that for someone to disturb the
scrum team they would have to physically walk
through the support team which invariably led to a
friendly “Hi, how can we help you? Those guys in
there really shouldn’t be disturbed.”
The new organization looked something like this:
PokerDev organization
Support team Test
(fight fires!) team
(virtual)
Product
owner
Scrum team
(prevent fires!)
Stakeholders
6.4 Fourth step: Start first sprint
The first sprint started on Jan 2 with a clear goal that
couldn’t be misunderstood: Poker SM.
More specifically: release a stable enough version of
the poker system that Poker SM will work on Feb 1
without incident.
Reza was Scrum Master of the fire prevention team,
with Henrik close behind as coach.
It was not a clean by-the-books sprint. It was messy.
The support team managed to hold off many
disturbances, but not all. Sometimes both teams had to
get involved in patching and problem solving.
There were staff changes. Contractors being rolled
out. New employees being rolled in. The size of the
Scrum team started at 10 but was 15 people at peak.
The daily scrums often took 30 minutes or more.
We didn’t have a clear end to the sprint. Did the sprint
end on Feb 1, in conjunction with Poker SM? Or did it
end a week before, when the actual release was made?
Or sometime in between?
Despite the messy nature of the first sprint, it was a
world of difference compared to before. The whole
pokerdev organization, every single person, was
aligned to a single goal. We had two teams with
different roles but a shared goal. There would be no
partial credit – we would all succeed or fail together.
Although the fire prevention team was disturbed from
time to time they did manage to focus enough to make
critical improvements to the system and make several
new releases just in time for the tournament.
6.5 Poker SM – 1 Feb
The tournament succeeded. The relief was
indescribable. Failing was just not an option.
As with most victories, luck had its part. The system
did continue crashing once in a while even after the
tournament. But not as often as before.
We had succeeded with our first goal! The other three
goals remained.
· Succeed with Poker SM on Feb 1
· Make the poker system stable
· Make the poker system scalable
· Create an effective and humane work environment
7. Spawning more Scrum teams
After the first sprint (actually even before it was
finished) we started spawning additional scrum teams.
This was driven by two facts:
· The system was getting more and more stable, so
we didn’t need as many people in the support
team.
· We noticed (to nobody’s surprise) that large
teams don’t work too well.
· After each sprint with a too-large team we
learned more about how that team could be split
up.
440
The diagram below shows the evolution of our Scrum
teams.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
The diagram is a timeline. Each column represents
one week, each row represents one team.
Each box represents one team doing one sprint. The
width of the box is the sprint length. So the first team
did a 4-week sprint, then three 2-week sprints, then a
3 week sprint, etc.
As shown in the diagram we spawned a new team
after every one or two sprints until we reached five
well-balanced teams in week 14. We experimented
with sprint length and finally settled on a
synchronized 3 week sprint cycle, i.e. all teams start
and end the sprint together.
Some teams were truly cross-functional, containing
people from client, server, test, DB, etc. Some teams
focused on one component, for example the first scrum
team evolved to become a server-only team,
continuously improving stability and scalability.
Here’s an illustration of how the organization looked
around week 12.
PokerDev organization
Test
Scrum teams team
Product
owner
Stakeholders
The support team was now gone. The support load
was low enough that each team could leave some slack
each sprint for this.
We kept the virtual test team concept, with testers
semi-dedicated to each scrum team. This was needed
for the time being since the poker system as a whole
still needed many hours of manual regression testing
in conjunction with major releases, so once in a while
the whole test team needed to get together to test a
new release.
8. Solving key impediments
8.1 Scrum knowledge
Lack of Scrum knowledge in the teams was an
impediment. This was quite easily solved. In early
February a group of 5 individuals from different teams
attended a Certified Scrum Master course with Ken
Schwaber.
The course alone obviously didn’t make them experts
at Scrum, but having a trained Scrum Master in each
team was enough of a “seed” to make the teams start
inspecting and adapting their own process. This freed
Henrik from much of the day to day coaching and
management and allowed him to attack specific
impediments instead.
8.2 Versioning and continuous integration
One of our key impediments was that we didn’t have a
stable baseline from which to make releases, and that
our code trunk was in a horrid state. Releasing was a
manual process and therefore time consuming and
error prone.
We spent immense amounts of time cleaning up our
branches, migrating from CVS to Perforce, and
changing our whole way of working. We implemented
a version control model known as the mainline model
and gave each team their own branch.
The article “Agile version control with multiple
teams” [2] describes in detail how we did this.
In conjunction with this we also implemented
continuous integration using QuickBuild. This
reduced, among other things, client build time from 5
hours of manual work to 20 minutes of automated
work.
441
These changes had a significant impact on our ability
to maintain a sustainable pace and release code in a
reliable way. It also made it easier for teams to work
in parallel without stumbling over each other and
polluting the trunk.
This improvement to our development environment
would have been impossible to complete without hard-
necked focus from the teams and the product owner,
and the support that the scrum framework gave us.
Since everybody agreed that these improvements were
crucial to our long-term velocity we were able to
allocate time to this on a continuous basis.
Another crucial aid was that Tain had a highly skilled
configuration manager, and we managed to have him
fully allocated to the pokerdev team for a few sprints
to help us complete these improvements.
8.3 Release process
As we implemented Scrum we realized that our
release process was severely dysfunctional. The
operations team was in Malta and there was no single
person there responsible for the poker product. Instead
they had teams of people working in shifts handling
multiple products on a 24 hour basis, and relied on
detailed documentation from the development team for
each release.
Despite many attempts to improve the hand-over
process there were frequently mistakes made during
the release process, causing costly roll-backs and
downtime for the poker system.
This turned out to be quite easy to solve. All we did
was shift focus. “Individuals and interactions over
processes and tools”.
We threw away the detailed release process documents
and templates describing exactly how the hand-over
should be done. Instead we demanded that Malta
provide one person that is responsible for poker
releases, and named him “Deployment Coordinator,
Malta” (DC). We then allocated a corresponding
person on our side and called him “Deployment
Coordinator, Stockholm”. Our test lead took on the
DC Stockholm role, which was a good match.
DC Stockholm and DC Malta were jointly responsible
for executing releases. They could use whatever
process they like – but there was no hand-over of
responsibility. They worked together until the release
was complete.
We also clarified that release is top priority at all
times. The two DCs could request whatever help they
needed to ensure that a release was successful.
Over time the two organizations – development in
Stockholm and operations in Malta, grew closer and
starting learning each other’s domains better. This
type of clear but informal collaboration, not detailed
process documents and hand-overs, was what was
needed to ensure friction-free releases.
8.4 Test automation & technical debt
Another key impediment was the terrible condition of
the code. There were no tests at all and it was
impossible to understand all the criss-crossing and
circular dependencies riddling the code.
The background to this was the fact that the code was
initially purchased from another company, already in
a bad shape, and then attacked during several months
by over 60 desperate developers working over-time to
make the system work in time for an unreasonable
release date in fall 2005. Not only did the release fail
to work properly, the code was irreparably damaged
and the system crashed every day.
We often considered rewriting the whole system, but
decided each time that it was too risky.
Instead we decided, first of all, to stop making things
worse. We trained the whole team in test-driven
development (TDD), instituted a rule that all new code
should have tests, and that each team should slow
down enough that they have time to write tests and
gradually refactor and improve the code base.
This was made possible by the Scrum rule that a team
chooses how many backlog items to pull into a sprint.
That enables them to leave room for improving the
code base.
We negotiated with the product owner and agreed that
roughly 20% of our time on average would be spent
refactoring the code and improving test coverage.
442
Some of the refactoring work would be quite complex,
sometimes requiring significant changes to the
architecture. In order to coordinate these efforts we
introduced a “tech backlog” and a “poker architect”
role.
The tech backlog listed the key improvements that
needed to be made, in priority order. The poker
architect was essentially the “internal” product owner
who owned and prioritized the tech backlog (in
addition to doing development in his Scrum team of
course). This freed up the main product owner from
having to understand and prioritize technical
refactoring tasks.
This agreement served us well. The product owner
knew that the team spent roughly 80% of their
effective time focusing on the product backlog, and
roughly 20% implementing long term improvements
to the code base. We all believed that this was the only
way to achieve a good velocity in the long term.
Although these rules weren’t always strictly followed
or enforced, the intent was clear. It took several
months to see the results, but by May it was fairly
clear to everybody that both the quality of the code and
our velocity had noticeably improved.
The tech backlog was a temporary solution, once the
most important items had been fixed the tech backlog
was no longer needed, and the team could revert to
more spontaneous refactoring on a continuous basis.
It was clear that the team had now taken ownership of
the codebase. Instead of complaining about the
codebase they were improving it.
During one period we had a dedicated scrum team
creating a test automation framework. For various
reasons this didn’t work out to well, among other
things it was hard to define clear goals and follow up
their work.
9. Delivering business value
During Jan and Feb we focused almost exclusively on
stability and scalability, all new features had to wait.
By March the system was stable enough that we were
able to allocate some teams to develop new features.
This was a critical milestone from a business
perspective – the ability to actually start delivering
new business value and not just repaying technical
debt.
By the end of April we finally completed and released
a clustered version of the poker system. This gave us
the scalability that we needed.
The cluster was also supposed to give us better
stability, since when one node went down players
would be migrated to another node and could continue
playing. Ironically, the nodes themselves were now
stable enough that the failover functionality was
hardly ever needed.
10. Where we ended up
By May the system was stable and scalable. The teams
were working normal office hours, even taking time
off regularly to improve their skills. We socialized and
had fun.
Compared to December the team had now been
reduced to half the size and was working about half as
many hours, but was producing significantly more
business value than before. Very interesting to see!
Needless to say, we had completed our goals and were
happy.
· Succeed with Poker SM on Feb 1.
· Make the poker system stable.
· Make the poker system scalable.
· Create an effective and humane work environment
11. What happened after
Inspired by the success of the poker project, Reza took
over responsibility for the whole pokerdev team while
Henrik moved on to implement Scrum in the rest of
the organization. Many of the lessons learned from
that are described in the book “Scrum and XP from
the Trenches” [1].
Since then Henrik and Reza have moved on and
helped several other companies transition to agile
software development, and learned that most lessons
from Tain still hold true.
443
12. Lessons learned
The most important take-away points from this
experience are:
· Scrum works. A project might fail despite
Scrum, but it will not fail because of Scrum. A
process can’t guarantee success, but we saw
clearly that even a great team will fail if the
process is dysfunctional.
· Scrum is simple but hard. It’s like chess – the
moves and rules are quite simple, but to get good
at it you need to practice, learn some standard
openings, and study professional matches.
· Technical problems are caused by process
problems. Focus on improving the process, and
the technical improvements will follow. In our
case XP followed naturally in the wake of Scrum.
· Good product owner and scrum master is
critical. Without a good product owner you don’t
get focus. Without a good scrum master you
don’t get process improvement. Both are critical
to success. Spend the time and money necessary
to find a good PO and SM.
· Don’t throw bodies and hours at the problem
o If the process is dysfunctional, adding
more developers makes development go
slower, not faster.
o Working overtime on a regular basis
makes development go slower, not faster.
· Get an experienced coach. This experience
report and many others confirm the fact that an
experienced agile coach is an important factor for
succeeding with agile transitioning. This is
especially important when the change needs to be
done quickly.
· Beware of framework teams. Teams that don’t
focus on direct business value should be managed
extra carefully using clear goals and tight
feedback loops. Alternative approaches should be
considered.
· Experiment! Don’t spend too much effort trying
to get everything right from start. Here are some
examples of important lessons specific to our
context that we probably wouldn’t have learned
without experimenting.
o Take testers out of their room, place them
with scrum teams, but keep them as a
virtual test team that can be pulled together
for regression testing.
o Have Deployment Coordinator roles instead
of a detailed written release process.
o Use a tech backlog to coordinate major
refactoring work.
o Export the product backlog to physical
index cards during sprint planning.
o An effective way to do sprint planning with
multiple teams is to rent a large conference
room offsite and do all teams at the same
time.
444
13. What we would have done differently
When implementing Scrum and XP in the rest of the
organization we gained some knowledge and
experience that would have been useful to apply
earlier.
If we could go back in time to December 2005 we
would do many things the same way, but some things
differently:
· Introduce pair programming earlier.
· Skip the test framework team, or at least equip
them with clearer goals.
· Use physical taskboards for the sprint backlogs
from the beginning, instead of Excel
spreadsheets. We discovered later that taskboards
were much more effective.
14. References
[1] H. Kniberg, Scrum and XP from the Trenches, C4Media,
Stockholm, 2007.
[2] H. Kniberg, “Agile Version Control with Multiple
Teams”, InfoQ,
http://www.infoq.com/articles/agile-version-control,
March 2008.