DEVELOPMENT IN FOCUS

ev 2 .0?
at is D o ftw are
Wh o ws
ring h n ging
lo is ch a
Exp e nt
e v e lopm
d

Overview The commercial imperative

This paper looks at how the demands of building and
running Web 2.0 software systems have changed
software development. The extreme commercial
pressures in this marketplace have driven some of those
changes; others have been created by the community-
centric focus of Web 2.0 software. These development
concepts range from the creation of the instant beta,
evolving software and death without a community to
mash-ups by communities.
Web 2.0
Coined in 2004 by John Batelle and Tim O’Reilly
• The web as the platform
• Architecture of participation encouraging users
to add value to the application as they use it.
Gives rise to a “collective intelligence”
• Rich Internet applications using Ajax et al.
• Dynamic content controlled by users
• Metadata to understand the content
• Web standards
Over the last four years, the IT industry has undergone
one of its periodic revolutions regarding the kinds of
software that are being developed. This revolution has
gone under the moniker of “Web 2.0”.1 Web 2.0 is
about enabling communities and the implications of
creating those communities. But it also has wider
implications in the way that software is built and the way
companies try to meet the needs of their prospective
customers.
The companies that created Web 2.0 systems have
found that they can do it with many fewer resources than
before, but only if they change the way they build the
software. Lower start-up costs mean a faster time to
understanding what the market needs, thus a quicker
ROI. Companies create the first version of a product as a
beta that is used to test the market. They are able to
iterate quickly in the search for the sweet spot in their
market space. The software development processes and
architectures that have been created allow start-ups to
be agile and flexible and able to handle frequent
change. Many start-ups try and fail to attract a
community around their offerings; but if a start-up hits
upon a successful market, they have to manage
stratospheric growth and the change that this growth
implies.

- Keyword search This kind of company is very different from most

- Links to related information
- Tags to categorize information
- Syndication using RSS and Atom
- Extensibility via APIs using REST or
SOAP protocols
• Scalability to millions of users
traditional company IT organizations, yet they have
managed to show a potential future of building
applications cheaply and quickly. Web 2.0 has had a
profound effect in both the functionality that is now
expected in software systems and the architectures that
are chosen to deploy them. The functionality has
focused on the social and participatory aspects of
software, while the architecture has been “the web as a
platform”. These kinds of application – regardless of
how they generate revenue – are often described as
“Software as a Service” (SaaS) applications.

1 Tim O’Reilly’s famous definition of Web 2.0: http://oreilly.com/web2/archive/what-is-web-20.html

Page 2 DEVELOPMENT IN FOCUS

Embracing Web 2.0 Companies such as Facebook, Digg and Twitter learned
about real reuse. This reuse was more than reusing
What does it mean to get a company to fully embrace others’ architectural components to speed their own
the concepts behind Web 2.0? It is all too easy to adopt growth. They recognized that turning their own systems
the benefits of the latest hype yet miss the implications into reusable components would work as a fundamental
of such adoption, which are more significant than may at mechanism for spreading their value. So instead of
first be recognized. protecting their software from piracy, they are increasing
The fundamental processes of how software is built have their applications' value by encouraging people to
been challenged and changed. These changes to “mash-up”2 their applications. This desire to be a vital
processes and architectures have been labeled Dev 2.0. component in the web architecture and thus a valuable
The impact of this way of building software has had a property is a driving motif in the community of Web 2.0
reach far beyond the wealth of community-based companies.
products such as Facebook, MySpace, Digg, etc. It has Their architectures gave them other advantages; they
also reached into the B2B enterprise market with even were able to handle delivery and support in very
companies such as Oracle adopting Web 2.0 for their different ways because only one version of their systems
Siebel product range. This new kind of software has existed. This also removed their vulnerability to software
necessitated a new way of building software that goes piracy because they no longer had to distribute their
beyond simply managing a new software architecture. software – they just ran it on their own servers. Their
desire to be “reused” through their API is a legitimate
alternative business model to piracy: It is hard to pirate
something that can be used for free.
The biggest Web 2.0 vendors (Amazon, Google and
SalesForce.com) have built their own software IDEs and
Dev 2.0 technology stacks to enable them to deliver their
systems. Their technology stacks have tended to evolve
• Software development as an evolutionary struggle
significantly as their applications have grown. In
• Reuse others’ applications in yours – mash-ups addition, there is a growing base of commonly used
are king components that are open source such as Hadoop,
• Software as a Service is the model of delivery MemCacheDB and NoSQL databases (e.g. Cassandra).
The technology stacks of vendors such as IBM, Microsoft
• Agile development: Development is controlled by and Oracle have been bypassed since they were found
three- to six-week sprints to be too cumbersome or costly for the job at hand.
• Evolutionary architectures The ability to evolve the applications so fast has led to
- Scaling by necessity many new development practices. Some evolved from
agile, some from coping with the challenges of large-
- No standard architecture scale web deployments, and some are completely new.
- Use what works when it’s needed One of these has been a significant shift away from
static languages to dynamic language environments
• Shadow app to control and monitor the real app
such as Smalltalk, Python and Ruby where the speed of
- Analytics driving development through development and evolution of code is noticeably faster.
quantitative testing and product sampling
There are a range of providers that have taken what they
• API richer than the GUI have evolved and are providing the entire technology
• Real-time bug fixing stack and development tools as a package (e.g.,
GoogleApp and SalesForce.com’s Force.com). These are
• Never roll back – fix and move on often called “Platform as a Service” (PaaS) providers.
• Dynamic languages are the new tools of choice While these have been popular, many organisations are
taking a much more individualistic approach to their
• Perl, PHP, Python, Ruby and Smalltalk development stacks, building what suits their needs
• Integrated participatory frameworks: wikis, logs, from a range of technologies.
RSS, Twitter, Facebook, etc.

2 Mash-up: to create application functionality by incorporating others’ applications into yours.

This is a key feature of Web 2.0 functionality.
DEVELOPMENT IN FOCUS
An application with many faces
“The web as the platform” is a fundamental precept of
Web 2.0. It is the only way that such applications could
have been built, and it is the glue that holds their
communities together. The community that congregates
around a Web 2.0 application contributes information of
value to that community and thus increases the value of
the application. It is the strength of that information and
the community that supports it that represents a core
part of the value of a Web 2.0 application.
SaaS applications are defined not just by the customer
web interface but also by the programmatic API that
other developers can use to access their functionality.
The API provides the ability to mash-up the application,
to enhance its usefulness and reach across the Internet.
They often provide more functionality than the basic
web interface that a web user may see – those user
interfaces are deliberately kept as simple as possible.
SaaS applications are evolved through a process of user
product sampling. User product sampling involves
evaluating new features with a small part of the
community to test their responses to the new features
against the control set using the standard set of
features. This limits the number of people affected if the
changes are negative or harmful. Often the evaluation is
measured in a highly quantitative way to allow a much
more automated evaluation process. Thus usability,
performance or scaling issues as well as commercial
effectiveness can be evaluated and contrasted against
the user base. Quantitative testing has become the
standard way of analyzing the effectiveness of changes
to a system. These sampled changes are evaluated
using a second hidden application (“shadow
application”) that is created for this purpose.
Testing is not what it used to be
Testing has changed, with real-time user feedback being
a critical part of the test cycle. With only one version of
the application running, defects are often of significant
impact to a large number of users and very immediate in
their nature. This has led to test teams being integrated
into the engineering team since bug fixes are required in
minutes at times, and the support team is often using
the most experienced engineer to fix such issues.
Extensive automated test suites are built up, and test-
driven design is seen as fundamental to ensuring that
new features and changes will work in production.
The release cycle has also changed with releases
occurring up to several times a week, even if they are
only for components of the whole system. Releases are
numbered by their time stamp rather than a release
number. The whole development cycle including
development environment and languages must be
capable of supporting this frequent release cycle, and
testing is now a tightly integrated part of that release
process this demanding approach to release
management.
Page 3
The only thing that matters is NOW
All Web 2.0 apps evolve on a daily or weekly basis.
Maintaining control of that evolution is critical to keep
the system working and successful. A couple of
strategies have evolved to enable this to happen: always
rolling forward and tight integration of the test and fix
team.
With changes continually rolling into a system, it is often
easier to fix something going forward than rolling
backward. Applications are rarely rolled back as the
stream of new user data means that new features can
often not be rolled back. Even if it is possible, it is
generally done at a component level.
Testing and defect fixing must be tightly integrated into
the engineering team to ensure as quick a response
time to initial defects as possible. With potential defects
able to bring down the whole system, the need to be
able to fix a system immediately is critical to the viability
of the business.
Agility is all
To handle this rapid evolution of the system, new
development processes have been adopted. Agile
processes like Scrum and Extreme Programming are
now commonly used. Software is built in sprints lasting
three to six weeks with a working deliverable at the end
of each sprint. Progress is controlled through daily
meetings that track development activities. The
development is done with less people, in less time and
with less development artifacts while focusing on
delivering working code, not models or documentation.
Product maps are rapidly evolving and tend to be short-
term. Overall, this means that less money is spent on
development.
Lightweight development tools and languages have
been chosen to support these new processes such as
dynamic languages like Perl, PHP, Python, Ruby and
Smalltalk. There seems to be a general perception that
they are 30% to 40% faster and therefore cheaper to
develop with than more mainstream languages such as
Java and C#3. Malleability of a language and its ability to
allow a code base to be changed rapidly, in sometimes
significant ways, are now a critical features of the
language that one chooses to build such systems.
3 This is a claim that is made very widely and largely accepted without critical analysis.
However a good example of an academic and rigorous analysis of typing’s impact on
development time is http://ecs.victoria.ac.nz/twiki/pub/Events/Plateau/schedule/plateau09-
hanenberg.pdf. The author Stefan Hanenberg is from the University of Duisberg-Essen. His
findings suggest that on a task that took the subjects between 0.79 and 15.83 hours the
benefit of using an untyped language was between 0.67 and 6.7 hours. Hanenberg uses the
expression “untyped languages” in the sense of “dynamic languages”.
Page 4
The new architecture
There is no such thing as a standard platform such as
Java Enterprise Edition once claimed to be. The issues
each business faces cause them to adopt highly
customized approaches to meet their business needs.
The move is toward efficiency and effectiveness over
standardization. The architecture chosen by an
organization must cope with evolution, scaling from
single servers to thousands of servers within an
application’s lifetime.
Such an architecture is often built from many different
elements – both infrastructure components such as
databases, file stores, etc. and other applications that
are embedded in the application’s functionality. To
handle the increasing volumes of traffic, architectural
components may need to be swapped out and the
whole architecture restructured. All of this must be done
without the systems coming down. Twitter has been
brought to its knees on a number of occasions due to
the fact that it chose Ruby as its production
environment. Twitter found that Ruby on Rails could not
scale in certain circumstances4. These well-publicized
problems forced Twitter to make a partial migration to
Scala without taking any of its application down5.
A new component known as the “shadow application”
monitors and analyses the application and is critical to
ensure the architecture grows in a timely manner. You
need to be able to model the size and scalability of not
just the running application but any future changes that
you are planning.
A whole range of new frameworks are needed.
Participation frameworks are vital: wikis, blogs, forums,
RSS frameworks, feedback mechanisms, surveys, Google
gadgets, interfaces to Twitter, Facebook, etc. and multi-
media content support. All of these are collective
intelligence tools and are part of the framework of
modern applications. These components must be
bought or built, and such a decision will impact the
speed at which you can bring your applications to
market and then scale to meet the demand.
Companies faced with these challenges have looked at
languages, IDEs and production environments that make
handling all of these complex issues simpler. These
techniques will spread from the specific domain of Web
2.0 applications into the broader application
development space.
Cincom, the Quadrant Logo, and Cincom Smalltalk are trademarks or registered trademarks of Cincom Systems, Inc.
All other trademarks belong to their respective companies.
© 2010 Cincom Systems, Inc. FORM CSUK1001012 04/10
World Headquarters • Cincinnati, OH USA • US 1-800-2CINCOM Fax 1-513-612-2000 • International 1-513-612-2769
E-mail info@cincom.com • http://www.cincom.com
DEVELOPMENT IN FOCUS
Features of a Dev 2.0 technology stack
By looking at a range of the more famous Web 2.0
companies and their systems, some clear common
requirements for a Dev 2.0 stack emerge:
• Web as a production platform
• Ability to deliver web applications and APIs
• Configuration management that can do daily releases
• Test-driven development and agile methodology
support
• Ability to debug running systems live
• Systems that can be their own description of the
system: Code that is simple enough to understand
• Extremely malleable, generally dynamic environment
• A stack that can evolve with rapidly changing
demands including swapping out components of the
architecture
• Tools to monitor production performance behavior
and frameworks that can be easily plugged into by
shadow applications
• Scale to thousands of servers
• Support for common Dev 2.0 components, e.g. such
as Hadoop, MemCacheDB, NoSQL databases etc.
• Support for all the main participation frameworks such
as RSS, wikis, Twitter, etc.
Outlook
The resulting reduction in cost and time to build systems
that are meaningful to their users has given those that
adopt Dev 2.0 a significant advantage over those still
using the old ways of doing things. Organizations using
Dev 2.0 will evolve better and more valuable systems
faster than those that do not – and that will give them a
fundamental competitive advantage.
If you want to know how dynamic languages, in
particular Smalltalk, can help you realise Dev 2.0
concepts, please e-mail us at
eurosmalltalk@cincom.com.
4There are many references to this story on the web. One of the most informative especially
about Twitter’s issues and their reasoning seems to be
http://www.unlimitednovelty.com/2009/04/twitter-blaming-ruby-for-their-mistakes.html;
however you need to read the comments from the Twitter developers to get the full picture.
5 Their choice of solution was significantly influenced by the need to keep their production
system running while they changed a core component of their architecture.
6A good example of the challenges and benefits are the comments of Twitter’s Jack Dorsey:
http://vimeo.com/5113346 and Digg’s Joe Stump:
http://www.infoq.com/presentations/Digg-Joe-Stump.
Printed in U.S.A. All Rights Reserved