Breaking the Lock

1. The first few months…

I was hired to head the BCG company’s effort to setup a modern Decision Support
System.

The Executive VP had done his homework, or at least he thought he did, and wanted to
build a Data Mart with real data and organize a proof of concept between the usual
suspects vendor companies.

At the onset this looked like a classical scenario: done that, been there, I was game.

The bar for what the Proof of Concept would achieve kept getting raised. We were going
to build a real life Data Mart and have each software vendor build reporting applications
with data from the Data Mart that we would be able to compare with the data published
in the existing reporting system.

So, I sunk my head into understanding the available data elements and their relationships,
identifying the existing data structures, mapping meaningful reports, starting a
dimensional design, etc..

My project was assigned a bright, competent, young Indian “C” programmer who worked
in the billing group since the data I needed would be sourced from specific billing tables.

I was thrilled.

I encountered a few difficulties creating the logical design. I decided to start the work to
build the Data Mart with the tables I knew we would need no matter what and while that
work would progress I would resolve my design dilemma.

The first warning sign came in the form of an apology from the programmer assigned to
the project and a “heads up” from the Application Development Director.

MY developer would not be able to continue working on my project for a week because
of an urgent billing related hot issue.
Being a patient, understanding and accepting kind of guy, I said: “no problem”.

After a few days, earlier than planned, my developer was ready to resume work on the
Data Mart.

And then it happened again, and again and again.

Then I realized that all the developers in the Department were actually assigned to a
minimum of four simultaneous projects of various sizes.

Eventually we completed the Data Mart, the vendors came and did the Proof of Concept.
But by then, the Executive VP’s interest for the whole initiative had waned.

The company was in Chapter 11 and the Executive Team was busy trying to find a buyer
for it.

In December, the Director of Application Development asked me if I would be willing to

work as a Project Manager on general IT projects. I enthusiastically accepted.

Two months later, the executive management of the company decided to downsize. I was
to layoff four developers who were reporting to me.

The worse part was laying off Ms. Wang.

I loved Ms. Wang.

Ms. Wang is a sixty something Chinese lady who made a specialty of maintaining critical
legacy applications.

She was slow, no doubt about it but her code had few bugs. Ms. Wang did not understand
English easily, neither did she understand the absurdity of the security procedures that a
manager is supposed to follow when doing a massive layout: no contacts with other
fellow employees.
So, the Director of Application Development and myself took her in his office and tried
slowly to explain to her what was happening and to answer her questions until she
understood.

We were both crying and Ms. Wang was hurt and angry. I felt like I was firing my own
Grand-Mother.

Rumors were rampant. Even before the lay-offs Fear was running high. Now, the
question on everyone’s lips now was: I am next ?

Soon after, the Executive Management decided to set a firm deadline to the top 4 most
critical IT projects underway. The projects were to be completed by April 15th and two of
those key projects were mine and they had barely started.

The developers attitude was: What happens on April 15th ? What is the point for me to
work long hours on these projects to deliver them by April 15th if after that I am going to
be laid off.

Two key developers decided to leave. The company pulled all the stops to keep them on
board. For one of them it worked. The other decided that it was time for him to move on.
Moral was low among developers, even after a pep talk by the COO and the CEO.
2. The Boss

Dr. Santosh Anand who had the title of CIO ran the Department. Dr. Anand was the
company’s first IT employee shortly after the company was founded. It was Dr. Anand
first job upon graduating from a prestigious Indian University.

Dr. Anand managerial style can be characterized as “benign dictatorial”.

Dr. Anand ‘s day is shared between executive meetings, long periods of time spent coding
specific software components and design review meetings with specific developers. Dr.
Anand is generally a mild mannered individual except when he looses his temper. At such
times, Dr. Anand favorite sentence is: “This is how it is and if you do not like it, feel free
to look for a job somewhere else.”

Based on the uniqueness of the tools necessary to run the company and on a desire to
save money on expensive specialized packages, Dr. Anand has had a long-standing policy
of expanding the efforts necessary to build custom software than buying commercial
software packages. As a result, the company had never bought any significant
commercial software package.
3. The IT Department
The Software Development Group was dominated by Dan Thomas. Mr. Thomas spent 12
years working as a software engineer at one of the US Army labs, where military style
command was the preferred mode of operation.

Mr. Thomas’s favorite put down for non-conforming developers was: “this is no rocket
science”. This would be followed by accusations of building over-complicated,
theoretical code directly copied from academic books with no contact with day-to-day
business reality.

As senior developer/architect in the IT Department, developers felt compelled to follow

Mr. Thomas’ software design instructions, even though they might be riddle with
inefficiencies and devoid of any event logging and/or error handling and recovery
capabilities.

The IT Department day-to-day reality was dominated by feverish efforts to meet

deadlines imposed by executives. Project commitments are made by IS Managers before
detailed requirements are gathered and before a requirement analysis/design is
performed.

In this context, somehow, the IT Department delivered 90% of its projects on time.
Further analysis revealed that most of the projects delivered were short to medium
duration projects with levels of effort below the 100-man hour level. Often, these types of
projects were modifications to existing legacy applications with minor bottom line
impact.

Commitments for Enterprise projects were treated in a similar fashion as smaller projects:
IS Managers would commit to a delivery deadline imposed by executives before detailed
requirements were gathered and before a Software Engineer could work through an
analysis and a design of the project requirements.

Typically, imposed deadlines on Enterprise projects had no relationship with the actual
amount of work necessary to meet the deadlines. And typically, deadlines on large
enterprise projects came and went without being met. Over time, some large enterprise
web based projects were delivered although none of them ever met the expectations of
their customers.

Software Engineers usually worked in isolation on smaller projects. Dan Thomas

frowned upon developers that paired up to work a design or a specific issue as a team.

When software development teams were created to tackle enterprise projects, they tended
to operate in silo, isolated from other project teams working on interdependent projects.

Soon, an attitude of “us vs. them” developed, further fueled by rampant finger pointing
and public put-downs by project managers from interdependent projects.

Things were turning ugly.

4. Methodology and Technical Environment.

Having attempted to understand part of the Production Database data structures, I

realized that no shared knowledge repository existed; no organized knowledge sharing
practices existed and little to no application documentation was available.

Although, software developers cooperated willingly together, knowledge of business

rules, legacy database tables, and specific application process existed only in the head of
specific software engineers.

No process existed to evaluate software development practices and tools. No organized

software development training program or continuous education curriculum was offered
to developers. No differentiation of roles between developers existed, nor was any career
path defined or offered. Software engineers were expected to be their own business
analysts, their own designer, their own coder and their own testers for their projects.

Typically, a scope document was written and approved by stakeholders. Often, but not
always, end users would be in charge of providing requirements for the proposed system.
Software Developers rarely provided Design and Analysis documents. Often end-users
would drastically change requirements mid-course through a project development phase
and no Scope Change-Control process existed. Design reviews and code reviews
occurred sporadically.

Two technical environments were in use: Testing and Production. Some developers
developed and tested their code on their own box, others on the test server.

Prior to deployment in the production environment, new software went through unit
testing and variable levels of internal customer acceptance testing on development
servers. Test plans were seldom written as part of the application design documents and
no automated testing tools were available. Testing was done by each individual coder or
by other developers. No formal testing group was in place. No code version control was
in place. Typically it was assumed that the latest and greatest code was the code in
production.

Managers justified the expediency of organization’s mode of operation by the urgency

and the pressure put on them by the Company’s business executives.
In an effort to minimize computing costs, Santosh Anand bought used servers and “off-
the-shelf” Intel based servers and, a few years back, had decided to move from UNIX to
the Linux platform.

With that decision, Java and JSP were the new logical choice as preferred development
language. Legacy applications were written using C and PowerBuilder.

It is important to notice that new functionalities were continuously added to existing

legacy application.

In spite of Dan Thomas’s best efforts to build Java based applications, core business
functionalities were provided by legacy applications.
5. Impact of the culture and mode of operation

1. Most individual software developers chose to simply submit to Dan’s design

decisions even though they disagreed with those decisions. The rationalizations given for
this position included:

- Expediency: considering the pressures for delivering the application, the proposed
design is the shortest road to meeting the imposed deadline.

- Let’s not rock the boat: this is how it is being done here.

Other software developers chose to comply while resenting that decision. These
individuals mentioned that they knew the proposed designs were fundamentally flawed
and felt powerless in affecting the situation in the absence of any manager or any forum
in the department where their point of view would be heard.

Finally, another developer chose to avoid open conflict with Thomas by concealing the
fact that the applications he built did not include any of Thomas’s design
recommendations.

2. Because of the speed and reliability with which applications were built in
PowerBuilder, legacy applications kept growing and expanding further delaying and
rendering more complex the task of replacing client-based legacy applications by more
modern and efficient web based applications.

3. Occasional trainings to new technology were organized. However, because of the

continuous expansion of legacy applications, developers working with legacy technology
were not getting an occasion to practice and improve their newly acquired skills.

4. Rumors were running rampant among developers.

5. Turnover rate among developers was high. About a third of the staff had to be
renewed over a typical year.
6. Most large web based Enterprise projects failed totally or did not deliver the
expected functionalities and the delivered applications were unstable and inefficient.

Over time, the culture of the IT Department had become a mix of free-for-all, rigid, and
fear-based culture paying little attention to methodology, new concepts, practices or tools.
The chaos created by the absence of consistent methodology and processes was
compensated by software developers’ willingness to sacrifice inordinate amounts of their
personal and family time to get applications in production. Although managers routinely
criticized existing practices, they also held strongly to the belief that a culture change was
not possible.

In other words, we were stuck.

7. The Black Sheep

Sandeep, a thirty something Indian developer had joined the IT Department a few months
before. Sandeep was a friend of mine, he had been unemployed for a while and since the
department had to replace two Java developers, I had submitted his resume for
consideration and he was hired.

I knew that Sandeep had a deep commitment to and knowledge of Object Oriented
Programming concepts and a mind attentive to new more efficient Open Source tools
whenever they would become available.

Soon conflict arose between Dan Thomas and Sandeep on various critical technical
subjects. Sandeep refused to use the Dbconnector class that Dan had build because of its
blaring inefficiencies and lack of error logging, trapping and recovery. Sparks were flying
during impromptu technical discussions the two would have. Since Sandeep was not
reporting to me, I felt there was nothing I could do.
Sandeep first project was a complete re-write of a high visibility reporting system
involving very complex business logic that he totally rethought and delivered on time at
the satisfaction of the client.

Sandeep was trouble because he was not scared to challenge the existing status quo.

7. The Opening

The Opening came fast and unexpected.

As I recall it was a Tuesday morning.

Santosh Anand, Dan Thomas, myself and a couple of the Java developers of the
department we were together sitting in the IT meeting room discussing the need for a
single set of entity objects that everyone would agree on and use. We had just discovered
that several different Java classes written by different developers for different
applications represented the same particular database entity !!!!!???!!!!

Nothing surprising considering what we now know of the shape the department was in.

I turned my head to face Santosh and with one glance at each other’s eyes, I knew he was
OK with what I was going to say:

I proposed that all the department’s Java developers meet together once every week and
discuss together for 1h design or technical issues that face them as a group.

We discussed what time the meeting needed to happen and we landed on the hour
between eight and nine in the morning.

Santosh gave his approval to the meeting and said he would participate to it as well.

I would chair and facilitate the meeting. I made sure that the voice of each developer who
would attend that meeting would be heard.

Dan Thomas was resistant but complied. For months thereafter and even now he still
complains that this meeting was a total waste since, according to him, that hour is the
most productive hour of a developer’s time. Developer’s only gripe was with how early
in the morning the meeting was.

8. The Aftermath

So we met every week, week after week.

We started by tackling the design of these database entity classes and little by little, all the
technical issues that had been carefully ignored and that had been weighing on the
efficiency and the progresses of the developers started to come to the surface.
Soon I kept a spreadsheet with a running list of the issues that were raised during the
meeting. I made my agenda to consistently bring up the top issues from the weekly
developers meeting to the managers’ meetings and to my meetings with Santosh so that
decisions would be taken and that issues would find resolution.

Consciously I had proposed the meeting as a venue to discuss particular technical topics.

Actually I had created a safe forum where developers could bring real issues, related to
object design or to the inconsistencies in the technical environment, work out their
disagreements with Dan Thomas and Santosh and not be fired.

For the first time, Santosh heard directly from the developers what worked, what did not
work and what was needed. To Santosh’s credit he did not seem attached to making sure
that any inefficient artifacts that he had created be kept in use. In other words, we worked
on a level playing field.

1. The Outcome

The first major outcome was the announcement of Dan Thomas’ resignation. Dan had
found another job and he was leaving.

Dan’s departure opened up a major security issue, since his name and password was used
as a generic user name/password for a number of major applications. A small group of
developers and myself spend a number of feverish days before and after Dan’s departure
plugging that security hole.

* It became OK not to use some of Dan’s Java classes that were build into a number of
important applications and that were inefficient and functioned heretically.
* Efforts were made to move all code into a single code repository that provided
version control. Four technical environments (Unit, System, CAT, Production) with their
respective servers were put in place with defined boundary rules and gatekeepers.
* Santosh and the IT Managers group focus on improving software development
methodology started paying off. Managers were sent to methodology training, a
requirements management tool was acquired.
* I designed and co-facilitated a meeting where the methodology framework was
presented and developers broke out in smaller groups to identify the process they thought
were relevant. Developers took ownership defining methodology steps they thought were
appropriate.
* Developers regained a sense of empowerment.
* Moral among developers improved.
* The executive team decided that the company needed to get out of the business of
building its own proprietary software and concentrate on maintaining existing
applications and acquiring tools available on the market.
 * Taking into account the lessons of the failed initial Business Intelligence/Data Mart
proof of concept project, I recommended the selection of an inexpensive Business
Analytics tool that does not require the building of a Data Mart to function. That tool
provides complex analytical reporting after a few days of efforts. Santosh Anand
understood its benefits and how it fit the needs and resources of the organization and
bought the tool. For the first time, business stakeholders have a rapid deployment, web
base, sophisticated business analytics tool at their disposal. They are delighted.

2. Lessons Learned or: What does it take to be an “Inside Change Agent ?

· First, I am going to assume that you are willing to spend the time necessary to
understand the environment to the point that you can describe its implicit norms and
explicit boundaries. You cannot just walk-in, make pronouncements and walk out.

You need to get to know the people, live their lives from the inside until you can name
the unquestioned assumptions that support the set of roles, structures, tools, people,
budgets assembled together make this organization. You might get there quickly, but the
occasion to break the lock might take a while to show up.

· Until the point that you see an occasion to create an Opening in front you, you
might feel like stuck. No surprise: you are just mirroring where the organization is at.
Keep doing what you are doing.

· Trust you guts. If you see an opportunity to create an opening, make it happen.
Don’t spend time agonizing on whether it is THE Opening or not. You will not
necessarily know the impact of what you are doing until much later.

· Don’t pre-judge the form that the opening needs to have. It will be an activity,
whatever the activity might be, shared by all interested parties that will open up
communication channels and build trust.

· In the meetings you chair, separate decision making from facilitation. The decisions
can be taken by consensus or based on technical expertise, you need to stay focused on
the process of the meeting, i.e.: facilitation. This point is actually very important.

By separating decision making from facilitation:

o You allow decision makers to listen to what the people who are doing the work have
to say.

o You ensure that no one is shut out of participating in the discussion and that
everybody’s voice is heard and respected.

With Gratitude.