A Pragmatic Project: Live In Concert

by Andy Hunt, The Pragmatic Programmers, LLC.

All too often we hear of well-intentioned software projects that crash and burn due to a long
and familiar list of common pitfalls. But every now and then we come across a truly Prag-
matic Project where a software development team tackles a monstrously complex problem,
does all the right things that we've written about in our books and articles and emerges tri-
umphant. From version control to unit testing (including a novel approach for fuzzy testing
of complex data) to automation (including a webcam), we'll look at a team that does it
right.

First, let me tell you about the project itself. Early in March 2005, the folks at Zenph
Studios invited me to a nearby recital hall to hear the famous pianist Glenn Gould play solo
piano on a 9-foot concert grand, live for an audience of about 30-40 people. But here's the
twist: the piece was played exactly as he played it on a 1955 recording, and Gould himself
died over twenty years ago.

Was this a pragmatic séance in operation? No. Player piano rolls? No. All the team had to
go on was a well known but scratchy mono recording made in 1955, and here was a live
piano replaying the concert note for note.

And just to prove how identical the performance was, our host, John Q. Walker (formerly of
Ganymede, now president of Zenph) played a recording of the new performance in one
speaker and switched back-and-forth to the original mono recording in the other speaker.
The sound was identical, except for the scratches, clicks, and pops on the mono recording.

From that old vinyl mono recording, the team at Zenph Studios analyzed the audio wave-
forms and extracted every nuance of the original performance: articulation, technique,
pedals; everything needed to recreate the performance. After analysis, this information is
then stored in a high-definition MIDI file (which is a quite a bit more sophisticated than the
more familiar, and now dated, MIDI standard). Finally, this data is fed to a freshly-tuned
and voiced souped-up Disklavier Pro piano (of which only several currently exist in the
world).

This is not the same Disklavier you might hear in a Marriott or Embassy Suites, endlessly
banging out Misty for hapless passersby. This is a high-performance, cutting-edge instru-
ment, capable of reproducing world-class, concert pianist performances.

Challenges

Which brings us to the challenges faced by the Zenph team.

As with any bleeding-edge technology, the Disklavier Pro came with no technical specifica-
tions describing the high-definition MIDI format it uses to drive the instrument. As Zenph

A Pragmatic Project: Live in Concert

Copyright © 2005 The Pragmatic Programmers, LLC
April 2005
Page 1
soon discovered, there were gaps in the data as well. Needless to say, there were no sup-
porting libraries to deal with high-definition MIDI files, either.

Then there's the audio side of it. How do you determine which notes were hit, with what
finger, articulation, the position of the piano pedals and all the rest of that when it's mixed
up in a monophonic audio track with all the ambiance, echo, and crowd noise? How on
earth would you even unit test this sort of thing?

Oh, and modern startup companies, unlike the Foosball-equipped wastrels of the dot-com
era, can't afford to just throw money at a problem, so you need to adopt a very pragmatic
approach.

Which they did.

The Methodology

The Zenph team consists of three developers and one expert user. Yes, the user is very
much part of the team, and in fact, drives the development effort, but more on that a bit
later.

Setup
The team is geographically dispersed across the U.S., connected by a virtual private net-
work (VPN). That's less surprising these days than in years past (and in fact is the rule on
virtually every project we've worked on), but none of our projects have ever featured a
fully-automated recording studio complete with webcam! With this setup, the developers
can feed high-definition MIDI files to the Disklavier Pro, watch and record the results using
the studio microphones, and download the resulting audio file to wherever they happen to
be.

Version control should be the first thing any project implements, and Zenph started with
Perforce from Day One of the project. Unlike many unsuccessful projects, there was never
even a question about using a mature, robust version control system for development.
Again, unlike many others, they put everything under version control, from PowerPoint
presentations for potential investors to their Articles of Incorporation and so on.

Testing
Now we get to the interesting part. How on earth do you unit test in this sort of environ-
ment? JUnit, for example, doesn't have an assertion for "does this audio file sound close to
this other audio file?"

Their goal is to generate high-definition MIDI files that sound like those created when real
performers play at the piano.

Here's their main coding/testing workflow:

• Start by creating a high-definition MIDI file using the given Disklavier Pro technology
(that is, physically play a note or two on the piano keyboard).
A Pragmatic Project: Live in Concert
Copyright © 2005 The Pragmatic Programmers, LLC
April 2005
Page 2
 • Now drive the piano using that MIDI file, and record the results through the air using
regular microphones in the studio to produce a WAV-format audio file.

• Process the audio file using the software under test to produce a MIDI file again.

• Compare the original MIDI file to the regenerated one.

Ah, but there's the tricky part. Unlike standard MIDI, these high-definition MIDI files are
built with around 10 attributes per note, all shifting and sliding around in time as well. Us-
ing simple assert statements as one would find in JUnit, CPPUnit and so on won't cut it: the
high-definition MIDI files don't have to be exact (and in fact won't be); they have to sound
the same.

Now it turns out that some perceptual acoustics come into play. The human ear is very dis-
criminating when it comes to timing, for instance. A difference of even 2-3 milliseconds is
very noticeable in the right context.

So they came up with the concept of a Grader. The Grader is a bit of testing framework
that compares high-definition MIDI using a set of heuristics, examining all 10 attributes per
note, and judging the "performance.” This gave the team a way to quantify the end goal of
does it sound right?

As we describe in our Pragmatic Workshops, you first have to know what "done" means in
order to actually be done. Without an objective measure of success, this team could have
floundered internally for months arguing whether they were converging on a solution or not,
or worse, delivering product that the end user wouldn't accept.

But not only did the team solve that problem, they took the idea one step further and made
the boss write the Grader. In other words, they made the project manager define success
in a quantifiable, achievable manner. That alone can make the difference between a suc-
cessful project and a death march.

It's also important to note that the Grader and its heuristics didn't spring into life fully com-
plete. They added heuristics to the Grader incrementally as development progressed: a
3ms discrepancy is better than a 300ms one, and one isolated discrepancy is better than
five in a row, and so on. They tuned and grew the grader as they went, slowly but steadily
converging on the goal.

Some teams would want to start testing with a Rachmaninoff piano concerto or somesuch,
and that's a huge mistake. You always need to start with small, isolated unit tests before
moving on to more advanced functional or acceptance testing.

In this case, the first six months or so of unit tests comprised some beautiful piano solos
made of just one note. Just one single note at a time, mapping out the full range of the in-
strument. It was quite a while before the unit tests got the software to the point where
they could try "Mary Had A Little Lamb," and quite a while after that before poor Mary got
any rhythm.

A Pragmatic Project: Live in Concert

Copyright © 2005 The Pragmatic Programmers, LLC
April 2005
Page 3
Requirements
Agile methodologies recommend that you have an expert as part of the development team,
so that you can get rapid feedback as the software matures and quick decisions on gray
areas.
But there's a possible down side to such intensive user involvement. As part of the devel-
opment team, the user may begin to lose sight of the distinction between their true needs
and various implementation strategies.

As with any project, the challenge is in drilling down to the user's true need without being
distracted by their view of possible implementations.

The Zenph team has an expert user---now completing a doctorate in Piano, no less. His
close association with the team shortened the feedback gap and let the team move with
great speed at satisfying the requirements.

Developers working in the recording studio.

The Technology

Different programming languages are better suited for certain applications than others; the
trick is to use the right tool for the job. Telecom and audio apps tend to favor C++, so their
production-level audio analysis software is written in C++.

But that's not the end of the story---they don't use C++ for everything. For rapid proto-
typing of low-level algorithms, the developers use a scripting language (they happen to use
Perl; other teams we know use Ruby or Python). They use the same scripting language to
power the automated build and test, and for controlling and coordinating remote-control
access to the studio and equipment.

On the C++ side, they've continued the practices that John and the developers used at
Ganymede: lots of good instrumentation in the code itself. This includes custom runtime
assertions with full call-stack reporting and so on. As to the age-old debate as to whether
you should leave these assertions enabled at runtime, the Zenph team comes down heartily
on the side of enablement. Since they are deploying this software in a service-bureau
A Pragmatic Project: Live in Concert
Copyright © 2005 The Pragmatic Programmers, LLC
April 2005
Page 4
model, there's no downside to displaying very technically detailed assertion failures at run-
time (software designed for your grandmother, on the hand, may need to adopt a some-
what less threatening posture).

Another enduring question that plagues many teams is "build or buy?” When is it most ad-
vantageous to build custom software, and when is it better to just buy it? As developers,
we tend to prefer to build our own stuff all the time (pragmatic author Venkat refers to this
as "Resume-Driven Design").

The team struck a good balance, leveraging open source, commodity-level functionality
where possible. Items such as path and extension handling, threading libraries, GUI wid-
gets, and so on were ably handled by libraries from boost.org, the wxWidgets set, and
others. They wisely reserved custom crafting where it was most needed, including their
core audio algorithms and components such as the MIDI data pretty printer.

Now printing out binary MIDI data in a nicely formatted, human-readable form may sound
like a frivolous extra, or at least something one could get off the shelf. But as the team ex-
plained to me, they didn't want some third party piece of software to mask, distort, filter, or
otherwise lie about the data. They needed to know precisely what was going on, with no
surprises. Too often, we take critical diagnostics on faith, only later to realize that the in-
formation wasn't accurate or complete.

Team Work

The most important ingredient of any project is the people. Not only do you need good
people to begin with, you need people who can work together effectively.

On larger teams, you can afford to have some trainees aboard---folks who aren't at the top
of their game yet, but aspire to grow in that direction. On small teams, you may not have
that luxury.

In this case, with such a small team, each member is a highly-skilled professional. To get
maximum skill coverage, each specializes in separate, non-overlapping areas of expertise:
the high-definition MIDI format, audio processing and acoustics, infrastructure and archi-
tecture, and concert piano performance.

Each team member is good at prioritizing their own tasks and maintains their own to-do
lists. But despite this autonomy, they still hold weekly virtual meetings to share progress
and status and coordinate their plans.

A Pragmatic Project: Live in Concert

Copyright © 2005 The Pragmatic Programmers, LLC
April 2005
Page 5
 Developer team meeting.

To leverage their computing environment and maximize mobility and interoperability, all
their development work is done on laptops (larger servers or grids will be used in produc-
tion). Even better, all the laptops are hardware-compatible IBM ThinkPads. This allows the
team members who are collocated to share parts more easily in case of emergency, and
makes issues such as OS and driver patches much easier to manage.

But while the development hardware platform is standardized, the software isn't. There is
no mandated IDE--nor should there be. Different people will be more productive in different
IDE environments, and these team members tend to prefer powerful stand-alone editors in
lieu of bloated, slow, all-encompassing IDE's (note the complete lack of editorial bias here).

With this technology and methodology in hand, this small team was able to create ground-
breaking software in a relatively short space of time. Let's take a quick recap of the salient
points of their approach.

Lessons Learned

What can we learn from this team? Just that the basics on which we continue to publish
and lecture make a big difference. Your team could be a success or yet another dismal
statistic, it's up to you. Here the practices observed by the Zenph team that we think are
most important.

• Start with good people. If you're lacking in certain skills or experience, invest in
training or workshops to help bring everyone up to speed (if you've got trainees you
need to bring along, make sure they have a chance to learn while keeping them out
of harm's way on the critical path). If you're faced with choosing between a
mediocre team that's collocated vs. a skilled team that's geographically dispersed,
choose the skilled team and invest in a decent VPN.

• Everything under version control. We're getting kinda tired of beating this drum,
but many teams still either don't use version control at all or do so improperly. If
your version control system is getting in your way or slowing you down, you need to
replace it or learn how to use it properly. By the way, my address book is under
version control, as well as my daily to-do list.

A Pragmatic Project: Live in Concert

Copyright © 2005 The Pragmatic Programmers, LLC
April 2005
Page 6
 • Define success. The project manager or sponsor needs to define what "done"
means for the team. And not just in some flowery mission statement. Give devel-
opers quantifiable and achievable goals and they'll meet them.

• Start small. Tracer-Bullet development is all about starting with a full framework,
but with all the functionality stubbed out. Get the small, fundamental basics right
first, operating within a sketch of the larger architectural context.

• Test. There's no excuse not to unit test. If the code is "too hard" to test, then it's
likely too hard to write. Well-designed code is easily testable; poorly-designed code
is not. Don't be afraid to create your own test framework or supporting code (like
the Grader) if needed.

• Learn from the user. Listen closely to the user---their expert knowledge should be
driving the requirements. But their suggestions should not determine the architec-
ture and design, that's the developer's job. Always look past the user's stated re-
quirement to discover their real need.

• Leverage technology. Use open source where it makes sense. Build proprietary
software where it makes sense. Don't be dogmatic in either case. Fancy or expen-
sive IDE's may be a great help, or they may slow the team down. Developers usu-
ally know which is true for them; make sure the team honors that.

• Teams build products. To build a great product, you need to also build a great
team. Whether your team is composed of novices or experts make sure everyone is
allowed to work appropriate to their skill level, and can all work together well (that's
all easier said than done, of course, but that's a topic for another book).

Looking Forward

John Walker says there are a great many historic recordings that were never released to the
public due to minor recording problems---an out of tune string, someone in the audience
coughing up a furball, that sort of thing.

But once converted into the high-definition MIDI domain, the performance can be edited: a
blown note fixed, technique adjusted to match the response of modern equipment, and so
on. When recording the Disklavier, there doesn't need be an audience present. That
means the recording engineer doesn't have to contend with squeaks from the piano bench,
the air-conditioning suddenly kicking on, or anything of that nature. And even if some noisy
event did ruin the take, it's trivial to restart the high-definition MIDI playback and re-record.

Thanks to well-written software that meets the requirements, this technology may well end
up having vast repercussions for historians, performing artists, and fans alike.

That's what we call a successful project.

http://www.pragmaticprogrammer.com/articles/zenph/

A Pragmatic Project: Live in Concert

Copyright © 2005 The Pragmatic Programmers, LLC
April 2005
Page 7