Copyright 2000-2014 The 451 Group. All Rights Reserved.

1

MIS Spotlight
Comparing public and private clouds is like apples and
oranges
Analyst: Owen Rogers, William Fellows Dec 2013
On the face of it, IaaS public cloud is a computational bargain at only a few cents per
hour, how could it be anything but cheap? It's access to infinite, inexpensive capacity to
anyone with a credit card.
However, almost without exception, enterprise end users we speak to are trying to
compare the cost of provisioning services internally with what the cost would be to
source them from external hosted suppliers. The gold standard here is Amazon. Many
enterprises have sought to come up with a price for resources (usually cents per hour
for a package of CPU, server and storage) that can be consumed internally. This figure
is used to benchmark internal IT performance against hosted providers or for budgeting
purposes. Firms as diverse as IBM and Merrill Lynch have executed internal resource
consumption on this basis. The practice predates cloud (think grid), but this utility
metaphor has even more relevance with the abundance of cloud services and the
enterprise adoption of cloud. Yet in all of the examples we have seen, the comparison is
not so much apples-to-apples, but of apples to oranges or some other strange fruit.
But the point is clear. Enterprises want to be able to compare the cost of their on-
premises IT to what it would cost in the cloud. Today, this is somewhat a black art,
according to IT decision-makers who attended our recent 451 Live Enterprise Summit
event. Even comparing the prices of the cloud providers themselves is troublesome.
These IT decision-makers also cited the lack of comprehensive price-forecasting tools
commercial tools provided by cloud suppliers are seen as proprietary. There were
numerous references to the 'bill shock' experienced when a CFO receives the monthly
cloud bill, demonstrating that consumption-based billing has risks as well as benefits.
What's clear is that financial engineering, as much as technology engineering or social
and process engineering, will determine the basis of future IT sourcing. This is precisely
the reason for providing a Cloud Pricing Codex series of digital economist reports.
Today, we examine how these issues impact the accurate comparison of public and
private cloud costs. Through a worked example scenario, we demonstrate these
challenges by comparing a specific use case of AWS public cloud pricing to a Microsoft
private cloud.
Creating a comparison
In our comparison, we will use the Microsoft TCO Private Cloud calculator to determine
the cost of a Microsoft private cloud. We assume our deployment requires 200 virtual
machines and will sit on commodity servers containing two quad-core CPUs and 128GB
of RAM, and that each processor can support eight virtual machines. Why do we use
these assumptions? Primarily so that we can compare virtual machines of
approximately equivalent size between private and public clouds. In this scenario, each
Copyright 2000-2014 The 451 Group. All Rights Reserved. 2
virtual machine will use 1vCPU and 8GB of RAM, and we assign it 2 x 420GB volumes
of storage. This is the closest match we can achieve to any AWS instance using the
Microsoft tool it is approximately equal in memory and storage to an m1.large
instance, although it's impossible to match this precisely without benchmarking,
particularly with regard to CPU capability.
This is our first problem: How do we compare like for like? The metrics used for
quantifying cloud resources vary between providers and cannot be compared easily.
Furthermore, the metric might not fully characterize the capabilities of the resources
two providers might both use GB to measure cloud storage, but read/write performance
might be completely different between the two providers. Responsibility is another
consideration. Some providers may offer more layers of management than others,
which relieves this internal IT department of those duties. A private cloud, unless
outsourced, will have day-to-day management requirements that are likely to be greater
than a public cloud. Each management and maintenance task performed by the
consumer has a cost implication in terms of manpower, but then again, each task
outsourced to a third party will have a direct cost implication, too.
Our second problem is how to use the TCO calculators to derive a fair and unbiased
assessment of costs. Each provider makes assumptions that may not be appropriate for
our use case, and it is possible that biases have been built into the calculator to make
the result favorable to the provider. Providers allow some of these assumptions to be
tweaked, but do not allow others to be changed. Providers also may not communicate
what these assumptions are. For example, on the Microsoft TCO calculator, it is not
possible to change the standard server builds or the ratio of virtual machines to
processors. This leads to bizarre sizing using midsized servers leads to a virtual
machine with a whopping 12.8GB of RAM, but only 60% of a single CPU core. Another
example is the VMware TCO calculator, which allows hundreds of assumptions to be
tweaked, but does not reveal how much memory each virtual machine (or the physical
host) has available to it, which is why VMware is not included in this comparison.
So for the purpose of this comparison, we will compare the cost of the Microsoft Private
Cloud against AWS m1.large instances our aim is to demonstrate the issues involved
in preparing a comparison, rather than to demonstrate one option or provider as
cheaper than another for all situations. There is also another layer of complexity. AWS
offers on-demand pricing, but also offers reserved instances where consumers can
purchase a reduced hourly price for a virtual machine by paying an up-front fee.
Customers have three choices of reserved instance low, medium or high utilization
which provides consumers with the option of paying more up front and receiving a
smaller hourly cost for a virtual machine in return. So we also need to build this option
into our comparison.
Private vs. public on-demand vs. public reserved
Let's evaluate how much the cost of our 200 virtual machines of approximate equal size
over three years will be if we use a private cloud, AWS reserved instances or on-
demand instances. For simplicity, in this evaluation we exclude hidden or associated
costs such as bandwidth. This is our third problem cloud pricing is more than just a
virtual machine, and there are several other costs to consider, which adds further
assumptions and complexity to our comparison. To simplify the analysis in this report,
we consider only the direct cost of virtual machine consumption, but associated costs
should be considered in detailed assessments.
Copyright 2000-2014 The 451 Group. All Rights Reserved. 3
We also assume consumption is constant, and that all capex is paid up front and is not
amortized. We assume capacity is constant from day one, and doesn't increase
incrementally over time as a result of variable demand. The comparison only relates to
the direct cost of the cloud, and does not take into account any internal costs or
savings. Furthermore, the findings of this comparison do not necessarily translate to
other scenarios. These assumptions do remove some realism from our analysis, but for
the purpose of demonstrating issues, these assumptions are appropriate.

Interestingly, in this specific comparison, it would appear that on-demand pricing is the
most expensive by a long way. The Microsoft private cloud is cheaper by $400,000
dollars. However, consumers can still make savings over the private cloud by investing
in reserved instances, with the high-utilization RI (which has the largest up-front cost)
making the biggest cost saving.
Looking at this figure, it seems our comparison is pretty clear-cut. However, there is a
huge assumption we are making in this evaluation we assume every single virtual
machine is being used all the time. In some scenarios, this assumption might be
reasonable for example, in batch processing where tasks are queued, there may
always be a steady stream of jobs ready to take free capacity when it is available.
Applications that are not built to scale might also use a fixed capacity at all times, as
would a cloud deployment that aggregates variability from many different applications to
create an averaged high demand. However, most cloud applications should be able to
grow and shrink as demand dictates; for these applications, it is likely that the utilization
of our capacity will vary over time.
The total cost is an important consideration in building a business case for any cloud
deployment, but this measure doesn't take into account how the deployment will be
used. A fair assessment of cost value is how much each consumed virtual machine
Copyright 2000-2014 The 451 Group. All Rights Reserved. 4
costs individually. So now let's look at how the price per virtual machine changes with
utilization.

This is where things become interesting. At 100% utilization, reserved instances provide
the cheapest option, followed by private cloud and then on-demand pricing. However,
as utilization changes, so does the relative cost of these options. At 60% utilization, the
private cloud becomes more expensive than on-demand because of the capex
investment in hardware, which is subsequently not being realized. At 40%, on-demand
is cheaper than a high-utilization reserved instance, followed by 20% for a medium and
10% for a low RI.
The cost-effectiveness of RIs also changes relative to each other. At 60% utilization, the
high-utilization RI is more expensive than the others. At 40% utilization, the RIs have
reversed now a high-utilization RI, once the cheapest option, is the most expensive RI
option, and a low-utilization RI is the cheapest.
What is happening here? The private cloud and RI options all require capex to be paid
up front. This cost is fixed, regardless of how many virtual machines are subsequently
used. As the number of virtual machines, n, is consumed, the price per machine follows
a 1/n relationship the bigger the value of n, the smaller the cost per machine, and vice
versa. On-demand pricing has no capex, so the cost per virtual machine remains the
same regardless of utilization.
In practical terms, the consumer must make an assessment of risk. If the consumer is
willing to commit a higher amount up front, it will get a bigger discount. But in return for
this discount, the consumer is taking on a risk; if the forecast of future demand is wrong,
the per-virtual-machine cost will increase. An accurate view of future utilization means
Copyright 2000-2014 The 451 Group. All Rights Reserved. 5
the most cost-effective option can be chosen prior to implementation. On-demand
pricing has no risk since no forecast is required, but consumers pay a premium for this
luxury.
This leads us to problem number four: The cost savings achieved by making an up-front
commitment is proportional to the utilization of the platform, which is an uncertainty.
Many TCO calculators will derive a per-virtual-machine cost based on 100% utilization
another assumption to be wary of when using such a tool.
This utilization has some major effects on building a business case. The following
animation shows the cumulative cost of the cloud over a 36-month period.

Notice how the relative payback periods change with utilization. At 100% utilization, the
private cloud becomes cheaper than on-demand after 20 months of usage. However, at
50% utilization, the private cloud will still not be cheaper, even after three years. The
same effect is seen for reserved instances, too. Understanding utilization is vital to
choosing the most cost-effective option, and building a business case that is realistic
and risk-aware.
Conclusion
So, is private or public cloud cheaper? The simple answer is that it depends on how it
will be used, which is why private and public cloud users vary in their opinions on the
cost-effectiveness of public and private clouds. The more complete answer is that it's
difficult to know for certain because of a number of problems in understanding and
comparing cloud services:
Copyright 2000-2014 The 451 Group. All Rights Reserved. 6
Problem #1: How to compare like for like? This is a big issue in general cloud adoption,
and it isn't likely to be resolved soon. The best approach is to benchmark as much as
possible. Use trials of public clouds to perform benchmarking exercises and ask
potential private cloud providers to supply independent benchmarking data, or ask to
speak to an existing customer. Draw up a table of responsibilities showing who owns
which task in each public and private implementation scenario. Attempt to work out the
dollar value of each task so that it's possible to compare the internal cost savings of
each option.
Problem #2: How to relate the TCO calculators to specific needs? They can be useful,
but remember that, ultimately, they are marketing tools just like any marketing
material, they exist to give a biased and polarized view. Check assumptions in the tool,
and change them if you can. Be wary of relying on a provider tool to give an
independent assessment of its competitors some assumptions might be made that are
not unreasonable, but may stack the odds in the providers favor. If you want an
unbiased assumption, and are willing to work for it, use the costs and assumptions
contained in each provider's tool to build your own spreadsheet and evaluation.
Problem #3: How to understand associated and hidden costs? Understanding the costs
before carrying out a comparison is the first step some line items are zero or
negligible, and can be excluded, while others can rapidly mount up. Using a forecast of
likely demand (see problem #4), work out what effect success and failure will have on
associated costs and how this relates to revenue. If a spike in demand causes a rise in
bandwidth, this associated cost might not be an issue if revenue grows, too.
Problem #4: How to assess cost with the uncertainty of not knowing future utilization?
Make a forecast of future consumption that is related to the objectives of the business
if sales are growing, it seems sensible to relate a forecast of cloud consumption of a
sales website. Build worst- and best-case scenarios what will be the cloud financial
impact if sales are flat, what will be the impact if sales grow beyond expectations? Build
this utilization into your own evaluation spreadsheet.
Many assumptions have been used to simplify our comparison in this report. We have
not considered extra costs, hidden costs or the effect of hybridization could
aggregating different pricing models offer best value and lower costs? These add further
complexity to any assessment.
It's easy to see why consumers are still confused by the cost of cloud, and are hesitant
to get in too deep. Cloud brokers might be the ideal entities to sit between consumers
and providers, finding the best execution venue for workloads based on cost and
technical requirements. But then again, perhaps the task is too complicated even for
cloud brokers, and it will be up to consumers to demand simpler pricing and
comparisons from cloud providers before comparing apples to apples is realistic.
In 2014, cloud economics is likely to be become more important as enterprises continue
to embrace the cloud and CFOs become more involved in cloud implementation and
operation. 451 Research will continue to publish reports and analyses on this important
topic, and the Cloud Pricing Codex will be updated and extended to give end users,
developers, CFOs, and service providers further insight into cost and pricing in the
cloud market.
Copyright 2000-2014 The 451 Group. All Rights Reserved. 7
This report falls under the following categories. Click on a link below to find similar
documents.
Company: No primary company

Other Companies: Amazon, Amazon Web Services, IBM, Merrill Lynch, Microsoft, VMware,

Analyst(s): Owen Rogers William Fellows

Sector(s):
Cloud / Infrastructure as a service / General
Hosted services / General