The Cloudscaling blog has a loyal following as can be seen from the website and RSS feed stats. As many of you long time readers know, I’ve been ‘in the game’ working on cloud computing technology or blogging about as long as anyone except perhaps those at AWS. In all of that time, my thinking and assessment of what’s happening and how it’s evolving has changed continuously. What was interesting for me this year is that this continuously changing perspective slowed to a crawl or perhaps even stopped. 2011 is the year that much of my thinking and perspective on cloud computing, particularly infrastructure clouds (aka “IaaS”) hardened.

That sounds tough. “Hardened.” I don’t mean hardened in the sense of rigid, but rather in the notion of wet cement drying. Many things that have seemed up in the air now seem settled and my doubts about the future of infrastructure clouds are gone. They are not only here to stay, but the shape and direction of them seem very clear. I’m not certain everyone else is clear, but I am. Perhaps I will be wrong, but I doubt it.

Let’s take a look back at the arc of my thinking and how things did NOT change in 2011. That will tell us what 2012 is likely to look like.

The Evolution of My Cloud Thinking
My thinking evolved through three clear phases:

Automation -> VMs & Virtual Datacenters -> New IT Paradigm

Phase 1: Automation
About this time of year, in late 2006, a short time after Amazon EC2 launched, myself and others prototyped a cloud application management framework similar to RightScale. At that time RightScale was named something else and had not been funded or publicly launched. These were early days.

As someone with a deep passion for automation, I remember thinking then that a lot of my lifetime interests (networking, storage, security, and systems management) were all converging and being managed by automation. For me, what was happening was all about automation … and lots of it.

Phase 2: Virtual Machines & Virtual Datacenters
Roughly summer of 2008, the first “CloudCamp” was thrown where a number of the cloud bloggers and thought leaders came together for the first time. Unknowingly we all centered about using the term ‘cloud computing’ to explain what this new emerging phenomena was. It was right after this event and over the summer of 2008 that the term “cloud computing” really took hold. This also led to the formation of the “clouderati”  and I simultaneously joined GoGrid as the VP Technology Strategy.

Perhaps GoGrid biased my thinking, but I started to move from a perspective that was cloud application centric back into my sweet spot of physical infrastructure and a focus on virtual datacenters or what I called at the time, “cloud centers”. In this view, virtual machines were king and inevitably, the question was: “how will we model existing datacenter environments?”

Phase 3: Cloud Computing is a new kind of IT
After leaving GoGrid in the summer of 2009 I had the opportunity to step back and take a fresh look at how things had evolved. I wanted to build my own cloud business again, but I wanted to skate to where the puck would be, not where it was today. I also could see that most everyone involved in the cloud computing space was spending time trying to retrofit the notion of ‘cloud computing’ to their existing business models and technology. Simultaneously, I still didn’t see any serious competitors to AWS.

What were they doing that was so different??

It’s not well known, but in the beginning of Cloudscaling’s (re)formation in fall of 2009 into mid 2010, I did a number of strategic and due diligence engagements on various IaaS vendors for VC firms, Platform-as-a-Service startups, enterprises, and enterprise vendors. During that time I was involved in deep technical dives on the technology and business models for these IaaS vendors. They ranged from GoGrid competitors to more of an enterprise cloud model. By late 2010 Cloudscaling, collectively, had deep architectural and business model understanding of roughly 10 different Infrastructure-as-a-Service (IaaS) players, not including the IaaS clouds that we helped build [1]. I am not sure anyone else had or has that understanding today. What we saw, was telling.

My primary takeaway was that even when it came to startups and direct AWS competitors, absolutely none of the infrastructure cloud players were developing their clouds like AWS. For the most part, they were simply integrating common-off-the-shelf (COTS) components to mimic an AWS-like environment. None of them had AWS velocity [2], nor were they paying attention to what made AWS special [3]. All too often, they identified ‘flaws’ in AWS that were instead unrecognized strengths. Examples of this include:

• Constrained feature set
• Standardized instance sizes
• Lack of VLANs [4]
• Ephemeral storage
• Generic load balancing service without fancy vendor lock-in features [5]

That’s when I began to understand that ‘cloud computing’ had less to do with automation or virtual machines/datacenters on demand and more to do with *how* AWS was building their infrastructure cloud.

Ask yourself these questions:

• Why isn’t VMware more successful in the public cloud space if it’s just VMs and VDCs?
• Why isn’t there a VMware-based competitor at similar scale to AWS? Or even close?
• There are now 100+ “VMs on demand” competitors, but almost none have the same growth rate as AWS … why not?
• What do the largest Internet giants (Amazon, Google, Facebook, SFDC) all have in common from an architectural standpoint and how is that different from a typical enterprise datacenter?

Cloud Computing vs. Enterprise Computing
I gave the third opening keynote at Cloud Connect 2011, behind Werner Vogels of Amazon and Lew Tucker of Cisco. That keynote drove much of the discussion during the first day around ‘enterprise clouds’ and their viability. In that talk was also the initial crystallization of my infrastructure cloud thinking:

We didn’t have one way to build infrastructure clouds … we had two.

One was rooted in the old modalities and thinking around existing datacenters and enterprise applications. The other was rooted in a new way of thinking about Information Technology (IT) that uprooted every approach that had gone before.

Enterprise Computing applied to ‘infrastructure cloud’ [6]:

• How do I virtualize and manage my existing datacenter apps?
• How do I achieve bottom line cost savings and extend server consolidation?
• How can my existing vendors help me create a ‘private cloud’?
• How can I be compatible with everything I own today?

Cloud Computing applied to ‘infrastructure cloud’:

• What can we do to allow application developers to experience ‘infinite scalability’?
• How can we simplify the allocation of traditional IT resources of networking, storage, and compute?
• What will it take to help next generation web applications ‘scale’ by simply adding more of these IT resources?
• How do we make it continually less expensive such that application developers can consume as much as they need?
• How can I, the service provider, make my cost of capital equipment and operational management as low as possible so I can pass those savings on to app developers? [7]

These are two very different schools of thought. One is about saving money for existing datacenters and applications. The other is about enabling new revenue streams via new applications and unlocking the potential for developers to add value to the business. The starkest example of this I can think of can be found in my blog interview with Adrian Cockcroft, the chief architect at Netflix on their adoption of Amazon Web Services [8].

A brief aside: This is why I think the NIST definition of cloud computing is such a huge FAIL. It’s focus is on the superficial aspects of ‘clouds’ without looking at the true underlying patterns of how large Internet businesses had to rethink the IT stack.  They essentially fall into the error of staying at my ‘Phase 2: VMs and VDCs’ (above).  No mention of CAP theorem, understanding the fallacies of distributed computing that lead to successful scale out architectures and strategies, the core socio-economics that are crucial to meeting certain capital and operational cost points, or really any acknowledgement of this very clear divide between clouds built using existing ‘enterprise computing’ techniques and those using emergent ‘cloud computing’ technologies and thinking. [9]

How 2011 Unfolded …
Ever since that keynote at Cloud Connect, it’s become more and more clear that not only is cloud computing a new disruptive displacement of the existing IT model (see blog link just above) in the same way that enterprise computing (aka ‘client-server’) displaced mainframe computing, but that it’s directly intersecting with other major trends in technology.

Infrastructure cloud computing directly intersects and either enables or works with:

• Big data, the explosion of data and data processing needs
• The post-PC era, or the notion of the rise of appliances and mobile platforms as the long term predominant platform, and the shift to ‘apps’ from ‘desktops’ [10]
• Consumerization of IT (TrendMicro whitepaper in PDF), or the notion that knowledge workers prefer more adaptive and flexible environments to get their work done such as they experience in their private lives with the large web application providers (Google, Amazon, Facebook, etc.)

I’m probably overlooking other related trends here, but what is blindingly obvious is that all of these trends are new opportunities, not old. Nor are they a re-hash of old opportunities. Every single one of them are driving infrastructure cloud computing growth. From the hidden, such as Apple’s iCloud, to the obvious, such as becoming the de facto platform for building big data or mobile app backend services.

As 2011 draws to a close this weekend, I’m beginning to see the upcoming ‘trough of disillusionment’ or ‘chasm’ as Geoffrey Moore called it.

Writing -> Wall
We are five years in and no one has emerged as a legitimate challenger to AWS’s market dominance. And, frankly, none are on the horizon. The enterprise infrastructure cloud providers I’m aware of have terminally poor growth rates (<10% CAGR in many cases) and most of them won’t see a return on investment before they hit their five-year hardware refresh cycle. Translation: these enterprise clouds are essentially net losses when evaluating them on a 5-year TCO basis. The hardware itself won’t even be paid for during that time, much less the cost of operations.

Meanwhile, AWS will reach $1 billion in revenue this year and those few that are following roughly the same trajectory as AWS have at least similar growth rates, if not scale (see slide 11).

While VCE touts $1 billion in vBlock sales [11], the onslaught of so-called ‘shadow IT’ hasn’t ceased or slowed down if AWS growth is any indication. Most of these ‘private cloud’ deployments have failed to deliver on the promise of cloud computing, hence app developers still adopt AWS in droves. Frankly, it’s stunning how many of the Fortune 1000 are running production apps, mostly next gen web apps or re-architected versions of last gen web apps, on AWS, but won’t talk about it.

The Road Ahead: 2012
In 2012, we’re going to see the gap between ‘enterprise clouds’ and ‘web-scale clouds’ widen as we enter the chasm. At Cloudscaling we are already seeing just about everyone with an ‘enterprise cloud’ out researching ‘low cost’ alternatives. Unfortunately, this is still missing the forest for the trees, as business agility and top-line revenue growth is a far more compelling value proposition for web-scale clouds.

I believe that 2012 will be a time of experimentation, learning, and quite possibly even larger ‘cloud failing’ than has gone before. Before it can get brighter, it’s got to get darker.

I don’t know the ultimate solution, but one thing is for certain, we’re all going to learn a lot making it through the chasm to the other side. The only other thing I can tell you for certain is that mimicking existing enterprise datacenters is a ‘looking back’ rather than ‘leaning forward’ strategy.

In this coming year I plan to spend a lot more time on this blog and in speaking engagements exploring all of these ideas, thoughts, and revelations in more depth.

–Randy Bias
Co-Founder & CTO, Cloudscaling

Let’s put this all to bed right now.  We’re going to look at the issues around multi-tenancy, security, pricing, and positioning.

Market Positioning
I’ll go into depth on this in the near future as it’s tightly related to my recent postings and presentations on ‘enterprise clouds’ (cloud-washed enterprise computing & virtualization systems).  Right now though, the key thing to understand is that AWS is *already* in the business of servicing enterprise customers regardless of security concerns.

Enterprises, like most other businesses, have two key adoption types: greenfield applications and legacy applications.  Greenfield enterprise applications have been adopting AWS and other commodity clouds for some time now.  During that same time, AWS has been very busy adding enterprise friendly features to increase the ability for legacy enterprise applications to adopt EC2.

A great example of this is Virtual Private Cloud (VPC), which originally provided simple layer-2 VLAN/Ethernet emulation combined with a VPN termination point.  Now, as of their latest release it also allows creating complex networking topologies, just like in a traditional enterprise datacenter.

Dedicated Instances are yet another arrow in the AWS quiver that reduces friction for enterprise adoption of existing legacy applications.  This is an enterprise focused feature.  It reduces concerns around security of the hypervisor and ‘sharing’.  Reduces, not eliminates.

It’s also worth noting that while billed as ‘Dedicated Instances’, Amazon has already been effectively selling dedicated VMs/instances in their HPC offering. [1]

Hypervisor Security
Whether you or I believe hypervisor security issues are relevant doesn’t matter.  Some people clearly do and not sharing the hypervisor may be a requirement in some regulatory and audit situations.  Providing customers a dedicated physical server and reducing sharing to only the network and (maybe) storage[2] is seen as a win by some security and compliance people.

For large enterprises, getting over that security and compliance hump is important. Frankly, my recent observation is that when a massive disruption is happening the incumbents will focus on creating Fear, Uncertainty, and Doubt (FUD) in key areas.  One is security.  Many of the threatened enterprise IT vendors specifically throw this up as a reason to avoid adopting public commodity clouds or using their same approaches to build your own cloud.  Dedicated Instances remove this obstacle.

Multi-Tenancy
Perhaps the most pernicious idea out there is that this is somehow ‘Uncloudy’ because the hypervisor is not shared.  I’m not sure how this kind of thing gets started, but at it’s roots it assumes that multi-tenancy is a core property of infrastructure clouds and that it is only achieved via the hypervisor.  Taking aside the definition of ‘multi-tenancy’ and whether it’s a core property, it should be noted that clouds ‘share’ many resources, of which the CPU/server is only one.  They also can share storage, networking, billing systems, etc.

Don’t misunderstand me.  I *do* think some kind of multi-tenancy is important, but there is a spectrum of multi-tenancy from ‘a little’ to ‘a lot’.  Also, what you call a ‘tenant’ is critical.  Finally, tenancy happens differently in SaaS from PaaS and IaaS.  The tenancy models are very very different.

So, let’s dig into this notion of hypervisor tenancy.  I have a couple of diagrams to show my point.  Assume we have 6 customers with 4 instances each on a cloud with 6 compute nodes.  Randomly distributed we see something roughly like this:

Voila!  Multi-tenancy.  Everyone is happy.  We have a cloud, people.

However, if we re-shuffle these instances and ‘bin pack’ them onto dedicated servers, we suddenly turn off the multi-tenancy:

What’s different here?  Have we truly lost multi-tenancy?  Customers are no longer sharing hypervisors and nothing has changed but that we’ve reshuffled the instances.  But perhaps we haven’t lost multi-tenancy.  Networking, storage, and other resources are still shared.

Let’s look at a more real world example, though, since most clouds don’t run at 100% capacity[3]:

Here we have a cloud running at about 75% utilization rate with instances randomly distributed.  This is in pretty good shape, but of course, all of these open ‘slots’ aren’t generating revenue anyway.  Of course, that’s part of the business model, so no harm, no foul.

Time to reshuffle!

Right, so now we’re still running at 75% for the entire cloud, but some customers are 25% utilization for their dedicated servers, some 50%, and some 100%.  Our cloud wide efficiency hasn’t been reduced significantly, but per customer it has.  This also means that customers are going to control the efficiency rate quite a bit more we would like, holding certain physical servers to themselves if this is the same as the AWS Dedicated Instances model.

This is where AWS rather clever pricing comes in.  They simply charge a sort of ‘tax’ across a single region of $10/hr if you choose to use this capability.  This tax effectively makes up for any inefficiency created by allowing customers to hold open a few more instance slots than normal.

AWS Dedicated Instances Pricing
Again, the confusion on whether this feature is ‘cost effective’ mostly comes from the Register’s biased assessment of the costs.  This feature is not targeted at individual consumers, but large enterprises looking to adopt en masse.  For such customers I’ve heard of monthly run rates between 100K-1M in usage charges.  $12M/year for a large enterprise is a drop in the bucket.  $1.2M/year doesn’t even touch the radar.

As an example, if a large enterprise was spending $1M/month and wants to get slightly better security, much better compliance, they would have to spend a whopping $10/hr per AWS region, roughly $7200/month[4].  That’s $86,400/year.  Let’s see, that’s an addition of .7% to their total annual spend on AWS.  Is slightly better security and ability to meet compliance standards worth >1% in additional cost?

If that same enterprise was only spending $100K/month, then we are looking at a 7% addition to total annual spend.  I don’t know what the value is of the AWS Dedicated Instances feature is to such a large customer, but I’m certain it’s more than 1-7% addition in additional spending.  Probably much more.

This pricing makes AWS Dedicated Instances extremely good value for money for a large business.  Combined with the new VPC features and being able to ride Amazon’s innovation curve, constant cost reduction cycle, and the other benefits of a large commodity public cloud provider, it’s hard not to find the whole offering rather compelling.

Conclusion
Better security, better compliance, less impedance mismatch with legacy applications, ability to onboard enterprise customers, and still cloudy.  This is a net win for everyone involved: AWS, enterprise customers, and the cloud community as a whole.

BTW, there are whispers that AWS has significant amounts of other related features that will further reduce impedance mismatch with enterprise clouds.  I expect that anyone sitting on an enterprise cloud (public or private) that doesn’t have an innovation cycle matching Amazon’s is going to get run over in the next year or two.  More from us on remaining competitive soon, though.

UPDATE: Cleaned up some language and fixed some typos.

Just like when we transitioned from mainframes, a *lot* of change is going to happen. In order to highlight this change, we wanted to start a series that really shows what is possible if you fully embrace cloud computing.

Our first interview is with Adrian Cockcroft, Cloud Architect at Netflix. Recently Netflix announced something incredible: their 24x7x365 video-on-demand streaming service, the largest of its kind in the world, had moved from their own datacenters onto Amazon Web Services EC2. See this QCon presentation from Adrian on the technical details.

This interview, however, takes a different tack and looks at why Netflix chose to take this route, examining business drivers, and asking “what do you tell others who wish to follow in your footsteps?”

The Interview

RLB: Could you describe at a high level what Netflix is doing on AWS?

AC: Encoding movies for streaming, log analysis, production web site and API, most everything that scales with customers and streaming usage. Easier to say what we don’t have there: most internal IT that scales with employee count, legacy stuff, DVD shipping systems, account sign-up and billing systems. We use Akamai, Limelight and Level3 CDN’s for streaming the movies, which is a cloud based service. There is an AWS CDN service, but they aren’t a big enough player in this space at this point.

RLB: What was the key business driver that led to deciding to use AWS rather than your own datacenters?

AC: Business agility and the inability to predict how much capacity we would need when and where for a business whose growth is accelerating. Year on year customer growth is 52% (up from 42% last quarter), year on year customers using streaming is up 145% (from ~4M to ~11M). We need to be a small-ish fish in a big AWS pond to leverage AWS investment in datacenter build-out, rather than building our own pond.

RLB: Many folks claim that they can deliver a private cloud at a similar price point to AWS. I assume you ran the numbers yourself. In whatever detail you can share, what does the ROI look like for Netflix?

AC: Our datacenter runs Oracle on IBM hardware, we could have switched to commodity hardware in a datacenter, but skipped that step by going to AWS. There are three points on cost, one is that Oracle on IBM is very expensive, so AWS looks cheap in comparison, and we have flat-lined our datacenter capacity.

The second point is that AWS costs are fully burdened, and we could not have hired enough SAs and DBAs to build out our own datacenter this fast. We have added 4-5x as many systems in the cloud as the total we have in our datacenter over the last year.

The third point is that the costs are elastic, you start paying for a resource just before it goes live, and if you stop using a resource you stop paying for it. If you own a resource it sits around a long time waiting to be delivered and installed, and if you no longer want to use that type of resource you are still paying for it for three years. When Amazon cuts prices, your installed capacity gets cheaper. When they install new instance types you can be running on them in hours, technology refresh in real time.

I don’t believe private clouds can compete with public on price, however if you have a bunch of empty datacenter space or want to re-organize your internal systems to be automated and API driven then there are real cost savings to building your own private cloud. I think VMware and Microsoft are going to own the private cloud space, but Amazon is going to continue to disrupt both of them at a lower price point for public cloud.

RLB: You have a great presentation on SlideShare that describes your development and operations approach. You were willing to rethink your entire approach to the Netflix system. Why was your business leadership willing to take on this kind of risk/investment?

AC: That’s the way they roll… The impetus to do this came from the top down. The leadership saw that the biggest technology risk we had was lack of agility, and wanted to invest in our ability to move faster than our competitors without being held back by scale (e.g. running out of datacenter capacity) or technical debt issues. The leadership had to drag the engineers along to some extent, wean them off SQL and established habits and processes.

RLB: Our belief is that ‘cloud computing’ is a new way of providing IT that displaces ‘enterprise computing’ as enterprise computing displaced ‘mainframe computing’. Your work seems to epitomize this. What would you say to folks who are willing to re-architect/re-engineer their applications for this new world? What are the top 3 challenges they need to consider?

AC: The key challenge is to get into the same mind-set as the Google’s of this world, the availability and robustness of your apps and services has to be designed into your software architecture, you have to assume that the hardware and underlying services are ephemeral, unreliable and may be broken or unavailable at any point, and that the other tenants in the multi-tenant public cloud will add random congestion and variance. In reality you always had this problem at scale, even with the most reliable hardware, so cloud ready architecture is about taking the patterns you have to use at large scale, and using them at a smaller scale to leverage the lowest cost infrastructure.

The second challenge is mostly political, an enterprise CIO and IT/ops department would much rather build datacenters and private clouds than become irrelevant to their customers. However while they are struggling to get their private clouds to work, their end users will be using AWS and setting a fully burdened price point expectation that IT cannot get near to. I have heard of internal chargeback for resources being an order of magnitude higher than using AWS for the same kind of resources [1].

The third challenge is the obvious one that the organizations that run compliance audits are working to a rule-sheet that has no concept of cloud, and it will take a while for some applications and industries (like finance) to get permissions and processes in place. That’s a when not an if.

RLB: I’ve heard from folks who think they are in the know that enterprises aren’t adopting AWS, yet Netflix, a mid-tier enterprise now has ‘thousands’ of servers running its most mission critical service on AWS. What do you say to these folks?

AC: Netflix is pathfinding, others will follow, and tiny startups in the cloud can become very big very quickly like Zynga, who are now one of the biggest games companies, from nowhere. Enterprises that don’t leverage the agility, scale and cost benefits of public cloud will lose out to those that do. For me, if its not multi-tenant, it’s not cloud, and only the biggest organizations should be building datacenters to host clouds, and they should be offering them publicly. If you are doing internal cloud and you have a dominant internal customer then you are doing it wrong, because you have to choose between baking in a lot of unused extra capacity or the risk that at some point that customer will blow up your cloud.

Cloudscaling Perspective
For us, the incredible lesson with Netflix has been how much value can be had in public clouds if one fully embraces the model. We see several kinds of failures to embrace cloud computing that parallel Adrian’s notes above:

1. Failure of understanding (“it’s just [mainframes|service bureaus|outsourcing|virtualization]”)
2. Failure to re-engineer/re-architect applications
3. Focus on building simplistic internal ‘private clouds’ (aka “your mess for less; only virtualized!”)

With understanding, the problem is the same as with the transition from the mainframe. Mainframe operators and the vendors of that time did not want to believe things would change as dramatically as they did. Many of them also looked at enterprise computing as a sort of ‘fad’ instead of seeing the underlying drivers at the time related to flexibility, agility, and expense (sound familiar?).

As far as re-engineering applications goes, we still see clients today asking us to build ‘five 9’ infrastructure clouds to support legacy enterprise applications they don’t want to re-architect. Few of them understand what that means or how expensive a 99.999% uptime cloud infrastructure would be (hint: 10-100x the cost of AWS most likely). I have seen so-called “private cloud” deployments where the 5-year TCO of a single virtual server ran close to USD $20,000. Why virtualize in the first place if the resulting costs are similar to those of running physical servers?

Which brings me to that last point: private clouds designed to accommodate all of the “needs” of enterprise customers aren’t true clouds at all, but simply “your mess for less.” Google can run 10,000 servers for a single headcount because their architecture is horizontal, homogeneous, and commoditized. The worst enterprise clients we have seen manage as few as *5* servers per headcount with the average at *100*. This is largely because of enterprise applications and legacy network architectures that require that the infrastructure is mapped to the applications instead of the other way around. This process combined with heavily silo’ed teams and application architectures results in the operational costs overhead being enormous.

Summary
Adrian and other Cloud Innovators have proven that designing for cloud scale, means designing for failure and the same ‘always-on’ software architecture of an Amazon.com or Google, *not* something as simplistic as virtualizing your existing legacy enterprise applications and moving them over to someone else’s cloud. It’s going to take a while for the market to wake up to this principle and we’ll see lots of failures or as I like to say “blood on the floor” over the next few years.

Those who embrace the disruption and use it to their advantage will win as the landscape shifts. Those who do not will be on the receiving end.

Disrupt. Don’t be Disrupted.

This really comes down to a particular TLA in use to describe grid: High Performance Computing or HPC.  HPC and grid are commonly used interchangeably.  Cloud is not HPC, although now it can certainly support some HPC workloads, née Amazon’s EC2 HPC offering.  No, cloud is something a little bit different:  High Scalability Computing or simply HSC here.

Let me explain in some depth …

Scalability vs. Performance
First it’s critical for readers to understand the fundamental difference between scalability and performance.  While the two are frequently conflated, they are quite different.  Performance is the capability of particular component to provide a certain amount of capacity, throughput, or ‘yield’.  Scalability, in contrast, is about the ability of a system to expand to meet demand.  This is quite frequently measured by looking at the aggregate performance of the individual components of a particular system and how they function over time.

Put more simply, performance measures the capability of a single part of a large system while scalability measures the ability of a large system to grow to meet growing demand.
Scalable systems may have individual parts that are relatively low performing.  I have heard that the Amazon.com retail website’s web servers went from 300 transactions per second (TPS) to a mere 3 TPS each after moving to a more scalable architecture.  The upside is that while every web server might have lower individual performance, the overall system became significantly more scalable and new web servers could be added ad infinitum.

High performing systems on the other hand focus on eking out every ounce of resource from a particular component, rather than focusing on the big picture.  One might have high performance systems in a very scalable system or not.

For most purposes, scalability and performance are orthogonal, but many either equate them or believe that one breeds the other.

Grid & High Performance Computing
The origins of HPC/Grid exist within the academic community where needs arose to crunch large data sets very early on.  Think satellite data, genomics, nuclear physics, etc.  Grid, effectively, has been around since the beginning of the enterprise computing era, when it became easier for academic research institutions to move away from large mainframe-style supercomputers (e.g. Cray, Sequent) towards a more scale-out model using lots of relatively inexpensive x86 hardware in large clusters.  The emphasis here on *relatively*.

Most x86 clusters today are built out for very high performance *and* scalability, but with a particular focus on performance of individual components (servers) and the interconnect network for reasons that I will explain below.  The price/performance of the overall system is not as important as aggregate throughput of the entire system.  Most academic institutions build out a grid to the full budget they have attempting to eke out every ounce of performance in each component.

This is not the way that cloud pioneers such as Amazon.com and Google built their infrastructures.

Cloud & High Scalability Computing
Cloud, or HSC, by contrast, focuses on hitting the price/performance sweet spot, using truly commodity components and buying *lots* more of them.  This means building very large and scalable systems.

I was surprised at the ISC Cloud Conference when I heard one participant bragging about their cluster with 320,000 ‘cores’.  Amazon EC2 (sans the new HPC offering) is at roughly 500,000 cores, quite possibly more.  And Google is probably in the order of 10 million+ cores.  Clouds built around High Scalability Computing are an order of magnitude larger than most grid clusters and designed to handle generic workloads, requiring hitting the price/performance sweet spot when building them.

Grid workloads can be very, very different.

Some Grid Workloads Drive the Grid Community
In talking to the grid community I learned that there are effectively two key types of problem that are solved on large scale computing clusters: MPI (Message Passing Interface) and ‘embarrassingly parallel’ problems.  I’m using terms I heard at the conference, but will use MPI and EPP (embarrassingly parallel problem) so that I can shorthand throughout the rest of this article.

MPI is essentially a programming paradigm that allows for taking extremely large sets of data and crunching the information in parallel WHILE sharing the data between compute nodes. Some times this is also referred to as ‘clustering’, although that term is frequently overloaded today.  Certain kinds of problems necessitate this sharing as the computed results on one node may effect the computed results on another node in the grid.  MPI-based grids, the de facto standard for most academic institutions, are built to maximum throughput and performance per system, including the lowest latency possible.  Most of them use Infiniband technology for example to effectively turn the entire grid into a single ‘supercomputer‘.  In fact, most of these MPI-based grids are ranked into the Supercomputer Top500.

An MPI grid/cluster, in many ways, looks more like an old school mainframe and technology such as Infiniband essentially turns the network into a high-speed bus, just like a PCI bus inside a typical x86 server.

EPP workloads, by contrast, have no data sharing requirements.  A very large dataset is chopped into pieces, distributed to a large pool of workers, and then the data is brought back and reassembled.  Does this sound familiar?  It should, it’s very similar to Google’s MapReduce functionality and the open source tool, Hadoop.  EPP workloads are very commonly run on top of MPI clusters, although some academic institutions build out separate or smaller grids to run them instead.

The majority of grid workloads are of the EPP type.  The diagram below shows this.

I had one person confide in me that “MPI power users drive grid requirements for vendors and assume that if their problems are solved, then the problems of [EPP] users are solved.”
This is interesting since these two types of workloads have different needs.

HPC vs. HSC
The reality is that High Scalability Computing is ideal for the majority of EPP grid workloads.  In fact, large amounts of this kind of work, in the form of MapReduce jobs have been running on Amazon EC2 since its beginning and have driven much of its growth.

HPC is a different beast altogether as many of the MPI workloads require very low latency and servers with individually high performance.  It turns out however, that all MPI workloads are not the same.  The lower bottom of the top part of that pyramid is filled with MPI workloads that require a great network, but not an Infiniband network:

In keeping with Amazon Web Service’s tendency to build out using commodity (cloud) techniques, their new HPC offering does not use Infiniband, but instead opts for 10Gig Ethernet.  This makes the network great, but not awesome and allows them to create a cloud service tailored for many HPC jobs.  In fact, this recent benchmark posting by CycleComputing shows that AWS’ Cloud HPC system has impressive performance particularly for many MPI workloads.

HSC designed to accommodate HPC!

Which brings us back.

The Moral of the Story
So, what we have learned is that scalable computing is different from computing optimized for performance.  That cloud can accommodate grid *and* HPC workloads, but is not itself necessarily a grid in the traditional sense.  More importantly, an extremely overlooked segment of grid (EPP) has pressing needs that can be accommodated by run-of-the-mill clouds such as EC2.  In addition to supporting EPP workloads that run on the ‘regular’ cloud some clouds may also build out an area designed specifically for ‘HPC’ workloads.

In other words, grid is not cloud, but there are some relationships and there is obviously a huge opportunity for cloud providers to accommodate this market segment.  At least, Amazon is spending 10s of Millions of dollars to do so, so why not you?

Here I will briefly cover the following points:

• Disconnects in Telco/SP Cloud Strategy
• ‘Hybrid Cloud’ Still Causing Confusion
• Public Cloud Hits a Tipping Point?
• Enterprise IT Governance

Telco/SP Enterprise Cloud Strategy Doomed to Failure?
How many of the large telecommunications and service providers have ‘enterprise’ cloud strategies? Their basic strategy boils down to:

1. Deploy an ‘enterprise cloud’, usually with VMware
2. Farm current customer base for cloud customers
3. Create a suite of services around the cloud offering that make it compelling

In talking to folks at VMware and various telcos and service providers, I heard a tremendous amount of focus on #3.  Many times I heard that “infrastructure is just a commodity” and “we’ll compete by providing value-added services.”  This is clearly a sound strategy, except that most of these providers are not building commodity infrastructure.  Most enterprise clouds have a very expensive cost-basis for their build-outs.  The most stark difference can be seen with VCE Vblocks clouds, which are almost 10x the cost of a commodity cloud[1].

Taking aside a number of the other issues with these enterprise clouds[2], how can you sell enough value-added services on top of a 10x more expensive infrastructure solution to make up the difference?  The answer is that you can’t and most telcos and SPs with this strategy will eventually have to face the math.

Related to this, #2 above seems to have telcos and SPs focusing on protecting existing customers.  How much more failed can a cloud strategy be than if it’s defensive?  The only good strategy here in response to the Amazon juggernaught is a frontal assault using those assets and capabilities you have and Amazon does not.

It’s interesting that even VMware, probably Amazon’s key competitor, is preaching along with EMC, a ‘Journey to the Private Cloud’ and pounding the pulpit for enabling developers.  This quote from Paul Maritz, CEO of VMware is choice:

In the final analysis they [purchasers] are not the people making strategic decisions in the business. Developers have always been at the leading edge, because that’s where business value is generated. Things that don’t differentiate you at a higher level will be SaaS apps – which will also be purchased at a higher level. The differentiated stuff you have to do yourself, and that means software development”.

I very much hope that telcos and SPs will start to develop some strategies and cloud solutions that are ultimately competitive.  The worst thing that can happen here is to have a GOOG/AMZN duopoly. (Please see my earlier post on the rumor of Google launching their own EC2-like service.)

‘Hybrid Cloud’ Confusion
In a panel at Interop on adoption of public clouds by enterprise customers there was a heated debate about the meaning of ‘hybrid cloud’.  This debate, mostly between myself and Paddy Srinivasan of Cumulux, was helpful for attendees, although as in most conversations of this type there is danger of devolving into an argument on semantics.  I made some pretty strong assertions about the general lack of usefulness in any context of the term ‘hybrid cloud’[3].  Essentially I simply reprised my posting from February this year: Hybrid Clouds are Half-baked.

Why stick on this?  For me, this is a question of straight talk.  We all live with the confusion of ‘cloud’ every day in our work, but when vendors use the term as something new to denote simple Service-Oriented Architecture (SOA) or the joining of two clouds (aka ‘cloud bridging’), that muddies the conversation further. The arbitrary creation of fuzzy marketing terms and pretending as if they have meaning does a disservice to all those who are trying to understand how to move forward in this new world.  Even the Wikipedia entry for Hybrid Cloud is a mess.

Another key reason to push on avoiding this term is that it ‘over promises’ on cloud.  Most companies don’t need a hybrid solution at the moment.  Certainly, some services, like identity management and authentication need to ‘bridge’ the firewall, but a single app doesn’t need to exist in both places nor does it need to move back and forth.  Neither do virtual machines (VM).  In fact, if you are following best practices using tools like libcloud, fog, or jclouds to manage instances and Chef and Puppet to package your app deployments, then you can deploy to any cloud on demand.  This approach makes far more sense than trying to move large VMs back and forth across wide area networks.

It’s just like buying two Internet connections from two separate ISPs, which certainly isn’t a ‘hybrid network’.  Using multiple clouds is a best practice enabled by proper tooling that increases portability & interoperability while reducing risk, not some kind of ‘hybrid’.

Public Cloud Hitting the Big Time?
Just before the enterprise public cloud adoption panel Brian Butte, the moderator, informed me that a poll of the audience had found that 95% of the enterprise attendees were using public clouds or planning on it.  A stark turn around from the beginning of the year when most were focused on private cloud development.

This parallels our experience that most enterprises will ‘fail forward’ trying to build private clouds.  By fail forward, we mean here that IT departments will attempt to deploy highly automated virtualization systems thinking they are private clouds, but not hitting the mark.  As Nick Carr pointed out earlier in Does IT Matter:

Of the more than eight thousand systems projects Standish examined, only 16 percent were considered successes—completed on time and on budget and fulfilling the original specifications. Nearly a third were canceled outright, and the remainder all went over budget, off schedule, and out-of-spec. Large companies—those with more than $500 million in annual sales—did even worse than the average: Only 9 percent of their IT projects succeeded. [ed. emphasis mine]

So what we’re probably seeing now is that enough early attempts at private cloud have failed or have been so slow that business unit owners are pushing for public cloud solutions and demanding immediate success.

Enterprise IT as Governor, not Control-Freak
I hit on this point repeatedly, but I also heard it from a number of other folks.  We’re clearly moving into a world of mixed IT capacity.  Some will be onsite and much will eventually be offsite and run by a multiplicity of cloud vendors; infrastructure, platforms, and applications.  In this new world, it’s more important for enterprise IT to provide governance rather than direct control.  This is very similar to how modern manufacturing or facilities management works.

Apple, Inc, for example, does not manufacture it’s hardware.  Instead, this key capability is outsourced, yet Apple has become an expert in managing a large extended supply chain of vendors and ultimate responsibility for delivering high quality hardware goods.

Similar to the process of managing global manufacturing relationships, I predict enterprise IT will shift to spend more time governing a large supply chain, not running each individual solution themselves.

Wrap-Up
All of this just further reinforces my thinking about Cloudscaling’s general approach to the market place.  We want to help telcos and service providers compete with AMZN/GOOG, while helping enterprises to understand how to embrace and manage the transition to cloud computing.  We think this means that telcos and service providers have inherent advantages that AMZN/GOOG can’t compete with.  Unfortunately, when these advantages are coupled with expensive ‘enterprise cloud’ infrastructure, much of the potential competitive opportunity is lost.

Imagine instead, that the inherent advantages of a large telco, such as geographical dominance, access to wireless networks, advanced networking capabilities such as MPLS networks, cheap IP backbones, were coupled with a cost-competitive Amazon EC2-like cloud infrastructure.  This ‘consumer cloud’ infrastructure combined with the telco’s natural advantages makes for a formidable competitive advantage in picking up all of the new cloud apps, particularly those that service mobile device developers.

When looking at how the marketplace appears to be unfolding, it seems clear that enterprises will continue to be confused with hype and promises such as that of ‘hybrid clouds’, miss the boat on delivering in the short term, pushing IT departments into adopting public cloud solutions whether they like it or not.  It also seems clear that servicing these new cloud apps is also critical.

To us, this means it’s important to have *both* an enterprise cloud to capture and retain existing enterprise customers while also deploying a low-cost commodity cloud, built like Amazon and Google, targeted at consumers, developers, and new cloud applications.  Particularly those apps that drive the burgeoning smart phone market and play to any telcos inherent strengths.

Let’s dive in and I’ll explain.

Cloud Stack Evolution & ‘Camps’
Amazon Web Services (AWS) spawned a huge ecosystem of knock-offs, management systems, tools, and vendors. They include, but aren’t limited to:

• AWS API compatible ‘cloud stacks’ including Eucalyptus, Open Nebula, and others
• Cloud management systems for the AWS APIs and services such as RightScale and enStratus
• Cloud services layered on top of AWS services such as Jungle Disk (S3), Heroku (S3, EBS, EC2), and more

Prior, I wouldn’t have called the AWS ecosystem a ‘camp’ per se, but if you read our most recent article on Google’s foray into cloud storage, you know that it seems likely they will provide a 100% compatible version of S3 and EC2 this year. Imagine the impact of Google Compute & Storage with Amazon Web Services compatible APIs. Already the Google Storage API is nearly 100% compatible with S3.

Together, as a block, Amazon and Google could create a de facto duopoly for infrastructure clouds, which isn’t good for anyone. We need competition and more than two major players.

Up against the Amazon camp is VMware. In my article on Amazon vs. VMware last year I highlighted how these two businesses were on a collision course. Nothing has changed and competition is mounting between them. The reason is that telcos and service providers are under increasing threat from Amazon and soon Google. They need viable solutions and VMware is attempting to provide a competitive ecosystem.

The VMware cloud initiative, vCloud, is designed to arm enterprises and service providers to be competitive, but has not quite delivered yet. VMware has had a number of problems providing a full cloud stack. The software, now in beta, is codenamed ‘Redwood’ has had significant delays in getting to market. Their strategy for cloud infrastructure does not appear unified outside of delivering compute virtualization.

VMware, as a business, understands they need to make their customers competitive. They have made a number of strategic open source acquisitions such as SpringSource, RabbitMQ, and Redis. There are also murmurings that they have some special projects inside that are ‘up the stack’ from their virtualization offerings. In total this shows that VMware ‘gets it’ in that they want to create a competitive ecosystem. While each of these is currently a point solution, there is yet to be a coherent story here. Can VMW build a consistent story and strategy around these disparate pieces? Only time will tell…

Besides these two camps, there is a long tail of clouds running various frameworks vying to establish themselves such as Cloud.com’s CloudStack, 3Tera, Hexagrid, Abiquo, OpenNebula, etc. John Treadway recently had posted a roundup describing all of the various cloud stacks out there.

OpenStack is stepping into the ring as a viable third camp. In particular, the OpenStack Storage solution is a clear contender to Amazon S3 & Google Storage. Many service providers and telcos have struggled to find a viable solution using commodity hardware that was price competitive. Suddenly, there is a viable proven solution.

Yet this is only storage. How can it create an effective ‘third camp’ alternative to Amazon and VMware for an entire cloud?

Lock-in, Architecture, Standards and The Truth about Interoperability

Interoperability for infrastructure clouds is poorly understood. Most believe that the problem lies in the on-disk image format (e.g. VMDK vs. VHD vs. qcow) or in the ‘hypervisor’ (although people don’t really unders/tand what this means). The truth is that lock-in has little or nothing to do with disk formats or the hypervisor. Most on-disk image formats are simply representations of block storage (i.e. disk drives). That means converting between a VMware VMDK and a Citrix XenServer/Hyper-V VHD is relatively trivial.

What about booting the converted disk image up on a new hypervisor? Guess what, since most hypervisors now rely on hardware virtualization (HVM) [1] using Intel-VT/AMD-V, that means that by default most will work with unmodified operating systems out of the box. No changes needed. The only downside of this is that usually the resulting performance is poor. This requires new paravirtualization (PV) drivers in the converted image. What does that mean? After converting the image from one format to another, you simply have to install the PV drivers for the correct OS. A process that requires being methodical, but is in no way technically challenging.

Where is the lock-in then? If it’s not the hypervisor, what makes moving from one cloud to another so difficult? Simply put, it’s architectural differences. Every cloud chooses to do storage and networking differently.

For example, if you wanted to move a virtual machine from GoGrid to Amazon, converting the GoGrid image to an AMI is not difficult. Unfortunately, GoGrid uses two networks, a ‘frontend’ and a ‘backend’ where your cloud storage system is connected to via the backend network. Every Amazon virtual server has only a single network interface. If your application assumes a separate backend network then what happens when it moves to a cloud without one? Or vice versa? Similar architectural incompatibilities exist between Rackspace Cloud, Savvis, Terremark, Hosting.com, Joyent, and all of the others.

The problem here, to be a bit more succinct, is that we need reference architectures for how infrastructure clouds are built. Amazon is one such reference. VMware’s vCloud is potentially another. Now there could be a truly open option with the gravity to gather community support.

More on The Third Camp

OpenStack’s potential to build a real community and a set of reference architectures drives towards greater standardization and interoperability. Perhaps more important than a cloud storage alternative, is this possibility for a true OpenStack community to form a critical mass such that a similar level of developers contributing to it as Amazon or VMware. Then commercial and alternative offerings, such as Cloud.com, Hexagrid, and OpenNebula can match their APIs and architectures to this set of reference architectures.

Will it happen? It’s hard to say, but the opportunity is there. Rackspace and others are putting serious weight behind this initiative.

What This Means for Telcos and Service Providers

For Telcos and SPs this means an alternative to VMware’s vCloud for commodity service offerings. A way to compete and operate at scale like Amazon and at a similar price point. Standardization through a similar reference architecture means greater compatibility between service provider clouds, which means greater benefit for customers and less lock-in, making them more desirable than the walled gardens.

You don’t want to differentiate on the basic compute, storage, and network offering. You want this to be as standard and interoperable as possible, just like 3G networks, TCP/IP, and similar service provider technologies. By creating a common open platform that everyone uses there is a better opportunity to facilitate wider adoption, create a competitive infrastructure service marketplace where providers work on differentiating in areas where they have an inherent advantage:

• Service and support
• Network & datacenter tie-ins (e.g. MPLS, hosting/co-lo)
• Bundled service offerings
• Differentiated value-added cloud services (VACS)

This is a game that all telcos and service providers understand. They have been playing it for the past 15+ years.

Conclusion

OpenStack, with a strong community behind it, should be an important tool for service providers and large telcos to compete at scale with the Amazon and Googles of this world.

We believe OpenStack and the reference architecture(s) associated with it will allow service providers (SP) to get their undifferentiated cloud offerings up and running early. For this reason, Cloudscaling will put real resources into supporting this effort. Getting basic cloud offerings up early then means providers can focus on support, services, bundling, and differentiated services as soon as possible, while embracing as large a customer base as possible. This is just as they compete on top of basic TCP/IP services today.

[1] Clearly, the market leader, Amazon, does not use HVM. They use PVM, a fully paravirtualized mode of Xen. However, even they seem to understand that HVM is the future. Their latest offering, designed for HPC, which is performance sensitive, uses HVM and supports unmodified operating systems. The reality is that the Intel-VT and AMD-V capabilities on the latest round of processors is incredibly fast and will only get faster. The battle is over. HVM and silicon won in this case.

• Amazon’s EC2 is clearly generating Real Revenue (TM) and could be at 500-750M in revenue for 2010
• Google has a massive global footprint and is north of one million servers
• The support structure for these servers includes a huge investment in datacenters, networking, and related
• The Googleplex houses an extremely large number of talented engineers in relevant areas: networking, storage, Linux kernel, server automation, etc.
• Google Storage recently went into BETA and is accepting developer signups

This last is perhaps one of the more telling signs. As you may be aware, Amazon’s Simple Storage Service (S3) pre-dates Amazon’s Elastic Compute Cloud (EC2). When Amazon launched in Europe they first deployed S3 followed by EC2. The same happened with their Asia/Pac deployment.

Amazon has built AWS in such a way that all of the services are synergistic, but in particular, EC2 is dependent on S3 as a persistent storage system of record. EC2 AMIs originate from and are stored in S3, it’s the long term backing store for Elastic Block Storage (EBS) and EBS snapshots, and it’s safe to assume that many other kinds of critical data that AWS relies on are stored there.

Would Google take a different approach? It’s doubtful. Amazon’s S3 is built to be a highly scalable storage platform[2]. Google’s own GoogleFS and BigTable server similar purposes. It’s certain that Google would use related design principles and hence we could see the Google Storage as a prelude to a Google on-demand virtual server service (Google Servers???).

Combined with the rumor I heard from a reasonably informed source I think we can look forward to an EC2 competitor, hopefully this year.

What I want to bring to folks attention here is that if another credible heavyweight enters into this market it will have a tremendous impact in further driving the utilitization of cloud services. In the medium term it will also threaten hosting providers and ‘enterprise clouds‘.

Why? I think what many hosting providers fail to understand is that Amazon and Google, particularly if fueled by direct competition, must grow up into the enterprise space. Just as in the Innovator’s Dilemma, they will eventually provide most of the features of any ‘enterprise’ cloud, which means that if you aren’t building to be competitive with Amazon and Google, you aren’t in the public cloud game.

Much more detail on this in a future posting.

In his presentation, James focuses on power: utilization, distribution, etc., and while an important element, like him, I don’t think it’s the most important factor.

I also want to dispel the myth that only the largest companies can achieve these economies of scale. Don’t get me wrong; providing a cloud service is a scale game. It requires a certain amount of buying power to compete. However,you don’t need to be MSFT, YHOO, AMZN, or GOOG to compete effectively. Buying power can be had at levels much lower than you might think.

In this article, I refer regularly to Jame’s comments in his presentation, so I suggest you watch his video first. In order to minimize confusion, I’ve borrowed some pictures from  his slides and inserted them here for your reference. This is a long entry, but it will be worth the read as I’ve got numbers for you which I hope you will find interesting.

Background
Like James, the Cloudscaling team has a history of building large scale services. I’ve worked in this area for 16+ years as has our COO, Adam Waters, and several of our team members. Understanding of the economies of scale, especially for service providers, cloud or otherwise is fundamental to our DNA. For example, see my previous piece describing how oversubscription works.

Enough of that! Let’s dig in and look at where you can achieve economies of scale, identifying areas James Hamilton may have neglected, and clarifying areas where I think there is still confusion.

Economies of Scale
There are a number of areas where you can achieve economies. James touched on a few in his talk.  While this is not a 100% complete list, here are key areas of opportunity that I see:

• Datacenter and facilities architecture (power & cooling)
• Buying power (COGS) for Networking, Compute (Servers), and Storage
• Development & Labor Costs
• Standardization & Homogenization
• Cash Flow

In James Hamilton’s model (see pie chart below) server costs are the dominant cost, but he critically left out development & labor costs.  This can be as much as 10% for a cloud and while it’s possible for large clouds to drive this down to a marginal cost, in practice there are no Infrastructure-as-a-Service clouds of sufficient size to achieve this yet. While James focuses primarily on power & cooling in his presentation, let’s take a closer look at some other areas.

Jame's Hamilton's Distribution of Cloud Datacenter Costs

Networking
There are  two key areas of networking where you can achieve economies of scale:

• Buying IP (network) transit (OpEx)
• Capital expense (CapEx)

Unfortunately, James provides one example with numbers from 2006 which compare two companies purchasing bandwidth. The larger company purchased bandwidth at $15 per Mbps per month at the 95th percentile ($/Mbps/mo) vs the smaller company’s expense of $95/Mbps/Mo. James uses this number to show a 6 or 7 to 1 buying power difference.

The IP landscape has changed dramatically since 2006.  It’s a dirty little secret in the hosting and cloud world that bandwidth is dirt cheap[1]. In fact, you need very little buying power  to get rock bottom prices. Street price for high quality Tier-1 IP transit is <$5/Mbps/Mo if you buy 1Gbps commits. That’s a mere $5,000/mo, which is well within the spending range of even small companies. My local coffee shop could probably afford it.  Yes, it’s quite possible larger buyers are getting even lower rates than $5/Mbps, but there is a bottom and it’s not much less than $5/Mbps/Mo so the disparity in buying power is closer to 3:1 or 2:1.

To give you some perspective, I know for a fact that some Yahoo! datacenters push upwards of 40Gbps.  A lot, certainly, but at $1-3/Mbps, well within the buying capacity of even medium size businesses.

The second area is using buying power to purchase network hardware at much reduced rates. When buying bulk quantities of network gear, the theory is that network costs can be significantly reduced, hence larger players have significant cost advantages.  Unfortunately, this only works partially in practice.  Network equipment costs are still increasing significantly, not reducing as with compute and storage equipment costs.  Combined, networking equipment (CapEx) and power usage (OpEx) make up 21% (in the piechart above) of datacenter costs and both are steadily increasing.

Large cloud providers are beginning to address this by systematically removing brand name networking vendors like Cisco and Juniper[2]. It is now possible to buy very high quality, exceptionally cheap networking gear direct from Taiwanese manufacturers. Many of these are the original equipment manufacturers for name brands.

Most people don’t realize how marked up Cisco/Dell/HP/Juniper gear is. For example, these Taiwanese OEMs have networking gear with price points as low as $100/10GE port. Yes, $100 per 10Gig Ethernet port. That’s about 1/10 the Cisco price point. At the same time, the quality of the gear is quite high and in some cases the components and chips are a generation ahead of what’s available from the name brands.

In other words, times are changing. We’re going to see a significant drop in the prices for networking gear across the board for the first time in ages and hopefully networking will get in line with the standard Moore’s Law curve.

Servers
Flogged. Dead Horse. There isn’t any significant buying power to be had in the commodity x86 server market. x86 server vendors, particularly those providing commodity offerings, have thin margins to begin with. 25% is typical. An Amazon or a Google can push these down somewhat by buying in bulk, but not enough to give them more than a marginal advantage. Anyone who can buy $1M USD of servers at once can negotiate a pretty steep discount. Many businesses can afford to buy at that price point.

James Hamilton understands this and points out where the real buying power is: buying customized hardware in bulk that allows for datacenter optimization and cost reductions in power, cooling, and space. By purchasing customized server offerings from the likes of an SGI/Rackable or Verari that include well spec’ed components, designing for their particular datacenter environment there are significant savings to be had. That’s where the real opportunities lie.

Vendors like SGI/Rackable and Verari can afford to build to spec in large quantities and amortize that customization across large orders.  These vendors are learning from the large clouds what works best and productizing it.  You will be able to benefit from this learning and productization too. In fact, we help our clients figure out these kinds of issues every day and know that these opportunities are within the reach of all types of cloud businesses.

Development & Labor Costs
Although touched on only briefly in the presentation, but I think this is the heart of the matter. Amazon leads the pack in rapid development of cloud services (see my post “Is Amazon Winning the Cloud Race?“). Their ability to innovate both automation and technology allows them to drive  significant economies of scale. This implies that development is a much larger cost of a cloud than might be expected.

Amazon’s EC2 was initially built using a 15 person engineering team. I estimate that AWS as a whole probably has 50+ software engineers and 20-30+ support and operations staff [3]. Last year, I estimated there were approximately 40,000 servers at a target price of $2,500 each for EC2. That’s 100M in CapEx on servers. A reasonable estimate of engineering and operational labor costs for AWS are probably close to 10-20M over the past 4 years. A not inconsequential number compared to the CapEx costs.

One client recently asked us to compare the operational expense costs for server administration between large clouds (e.g. Microsoft and Google) and a typical enterprise. Usually, enterprises can manage 100-200 servers per admin. Microsoft’s stated goals are 1,000-2,000 for their Chicago datacenter (confirmed in the presentation). Google is managing at the scale of 10,000 servers per admin and trying to get to 100,000.

If you do the math, the basic cost for administration is $75/mo/server for the enterprise, $7.5/mo for Microsoft, and a mere $.75/mo for Google, a 100x difference! When calculating the long term TCO, you’ll find that investing heavily in automation is a “no-brainer” for those whose core competency *is* building at scale and operating IT at the lowest cost possible[4].

The lesson here, which James alludes to in his presentation (see his map of AWS releases in 2009 below), is that one major economy of scale is the ability to have significant resources deployed for software development purposes. The outcome of most cloud software development is generally automation or technology that enables the business to scale more efficiently.

James Hamilton's map of 2009 AWS releases

Standardization & Homogenization
Often overlooked is that businesses built at cloud scale *must* run relatively homogeneous environments. By standardizing, they can achieve reasonable scale. For example, Google is reputed to run as little as five hardware configurations across its one million+ server base. In contrast, a typical enterprise  has hundreds of configurations across a much smaller server base, increasing operational overhead and expense dramatically.

Did you know that the primary driver behind Yahoo!’s cloud computing initiative was to normalize and cleanup their 800+ configurations? It’s impossible to operate at massive scale without homogeneity and standardization. A huge benefit of virtualization in cloud environments is that it allows the standardization of the physical hardware platform while running a plethora of operating systems at the virtualized layer. This is also why I am occasionally a little sad when I see large enterprise IT shops insisting on purchasing their x86 server hardware vendor of choice. It just doesn’t matter any more. Not if your cloud (internal or external) is designed correctly.

The Cash Flow Problem
A less understood, but just as interesting area of business scalability for clouds is the interplay of growth, speed, and cash reserves. There’s no question clouds can be very profitable and attractive for operators given their high margins and 100% compound annual growth rates (CAGR). Typical payback periods on installed cloud hardware average a fast 3-6 months. However, new entrants need to plan for an extended period of ever increasing hardware expenses that stay well ahead of free cash flow. Larger clouds will require $10M in liquidity to meet their rolling hardware acquisition needs, and even small clouds need to think about acquiring hardware at $1M per step. One key in building a highly profitable cloud lies in minimizing the lag between hardware acquisition and revenue generation. The difference between days and months is the difference between profitability and disaster.

A Brief Aside on ‘Utilization Rates’
And that brings up an interesting point I want to address, which is not technically an economy of scale, but worth discussion.  I’m intrigued by the notion that  ’utilization rates’ are almost completely meaningless in the context of public cloud providers.  James Hamilton’s claims 30% server utilization (presumably CPU??) as a high metric even for cloud service companies. However,  this doesn’t matter when you sell your capacity like an IaaS cloud does.  Here’s why: whatever your particular cloud business model is (e.g. selling RAM, CPU, or ‘bundles’ like Amazon’s instances), you *must* sell as much of it as possible at any given time.  Anything else is business suicide.  You can’t overbuild and only have 30% of your capacity sold.

Most IaaS providers target 75-80% sold capacity. Their cloud is therefor ‘utilized’ at 75-80% in that the capacity is sold, even if not at 100% usage itself.  Unlike a business where unused capacity is waste,  in a business model that sells capacity, unused capacity is unsold capacity and hence, not competitive.  Not all usage of the term ‘utilization rates’ is the same, especially when discussing an IaaS cloud[5].

Conclusion
It’s critical to understand the potential economies of scale for cloud providers. They can achieve these economies through size and focus. While larger players have some advantages,  many businesses can afford to buy servers and network in enough bulk to see significant price savings. More important than sheer size is the ability to focus on innovation.

Public cloud providers have a core competency that involves delivering IT services at a very cost effective price point. They are the new IT utility companies of the near future. Their ability to focus and spend development resources to achieve ever newer economies of scale will be something that traditional businesses can’t compete with. Traditional enterprise IT vendors will likely continually be playing “catch up” and be unable to provide competitive solutions in time.

Economies of scale are why other business infrastructure in the past, e.g. railways, telecommunications, and shipping, have consolidated into businesses who focus on delivery of the infrastructure as a core competency. To think IT is any different is to bet against history.

UPDATE: Added footnote to clarify that AWS is currently aggressively hiring so my initial estimates on engineering staff size are way off.

To be clear, the top three U.S. clouds do not use iSCSI SANs: Amazon’s EC2, Rackspace Cloud, and GoGrid, all use local RAID subsystems.  This is common knowledge.  Of the early cloud pioneers, as far as I’m aware, mostly the U.K.-based clouds such as ElasticHosts and FlexiScale use iSCSI SANs.  The latest set of new cloud entrants, such as Savvis, Terremark, and Hosting.com all use either iSCSI or Fiber Channel-based SANs.  This is also commonly known.

Your Mileage May Vary on these performance numbers.  I’m not trying to highlight any ‘right’ way to build a cloud here.  I’m simply trying to show what the difference in performance is between a single SATA disk and a VM disk drive backed by an iSCSI SAN over a single Gigabit Ethernet.

This is not a robust performance and benchmarking analysis.  It’s a simple “run the numbers and compare” blog posting.  These are by no means authoritative performance numbers and that’s not their purpose either.  Their purpose is to highlight how performance differs between a single spindle and many in a RAID configuration, even when that RAID is available via a SAN over Gigabit Ethernet.

Please avoid overly critiquing the testing technique here.  It’s not meant to be robust, so nitpicking it serves no purpose.

Setup & Methodology
This is a very simple test in the Cloudscaling hosting & cloud lab environment.  Both servers running the test are on latest Ubuntu Jaunty Jackalope release.  One is a physical server with a single SATA disk and the other is a VMware vSphere VM backed by an iSCSI LUN.  The iSCSI LUN is provided by a ZFS-based SAN product called NexentaStor from Nexenta Systems.  This is an OpenSolaris derivative and a very cost effective alternative to say a NetApp or EqualLogic system.

The iSCSI SAN hardware is a simple Sun x2200 M2 with a Sun J4200 JBOD and 6 15K RPM SAS drives.

The bonnie++ command line was as simple as possible:

bonnie++ -n 512

Basic Numbers
Here is a basic high-level chart showing the numbers.

Figure 1. High level of SATA vs. VM disk

The first thing you will notice, of course, is the two big spikes for sequential and random file reads.  These numbers are artificially inflated as clearly 325,000 IOPS for sequential and 460,000 IOPS for random reads are ridiculous.  This is likely due to caching either in the OS or the controller on the physical box.  bonnie++ is supposed to account for this, but for some reason, in this instance it did not.  So it might be a little easier to evaluate the relative performance on a logarithmic scale:

Figure 2. Logarithmic scale for test Results

Much better.  What is easier to notice here is that the VM generally performs better on both standard measures of disk speed: raw throughput and disk operations (I/O per second or IOPS) with the obvious exception of the two aberrant data points.

Removing those two data points will give us an even clearer picture:

Figure 3. Normalized test results

Great.  Now this is very clear.  As you can see, the first half of the chart shows raw throughput (Kbytes/second).  When reading blocks from the VM disk we’re nearly saturating the gigabit ethernet link which should top out at 125Mbps theoretical, and we’re hitting 107MBps on average over 10 runs, so this is quite acceptable.  The SATA disk, in comparison gets just over 60MBps, which is about right, even though the SATA spec and controller are capable of more.  Sustained block reads from SATA disks will typically be 60-80MBps in the real world.

Much more interesting is the number of IOPS.  Many real world disk workloads, like a database spend the majority of their time doing large amounts of their ‘seeking’ from one position of the disk to another, meaning lots of random file access.  They will bottleneck on waiting for the disk ‘head’ to move from one position to another on a disk drive and read new data.  It’s hard to tell the difference above because the SATA disk is so slow it barely registers on the chart.

If we change to a logarithmic scale again the data becomes much easier to read:

Figure 4. Normalized logarithmic scale test data

Now you can see that doing random seeks (i.e. moving the head of the disk drive from one location to a new one to read a piece of data) are starkly different.  A single SATA disk gets about 185 IOPS while a set of 6 SAS disks in the SAN is right around 10,000 IOPS.  This is a huge performance difference.  There are several reasons for this.  One, a typical SATA disk has an average latency of 8.5ms and a 15K SAS disk has only 3ms.  Also, with 6 disks in a RAID configuration, I have 6x more disk heads to read with.

It’s still a bit hard to see with this chart, but for most of the rest of the IOPS tests above, the SAN solution is roughly 3x the performance of the single disk.  For example, Sequential File deletion is 2,573 (SAN) vs. 840 (SATA).

Rather than going through the entire set of results, I recommend you download my simple spreadsheet.

Note that for Amazon, Rackspace, or GoGrid, local VM disk results will likely look very similar to the iSCSI SAN results for IOPS and sequential read/write (first half of chart) will be much higher.

Amazon’s Elastic Block Storage (EBS) would have similar performance characteristics to the iSCSI SAN above and hence you can see why it can be acceptable for running a database.

Summary
My point here is very simple.  I want to highlight the difference between purchasing a dedicated server with a single (or small number of) SATA disks vs. going with a cloud solution that uses a shared iSCSI SAN or local RAID on a single physical node.  Purchasing your  own dedicated server solution with a RAID can be extremely costly compared to a similar cloud solution.

More importantly, for those workloads that require random I/O and file access, like database applications, RAID is clearly a winner.  That’s why using a shared RAID (via an iSCSI SAN or a local RAID) on a physical node for your cloud VM can be a clear advantage of the cloud today.

Regardless, a bit more light has been shed on Amazon’s controls and measures in their recent security webinar.  You can access it here.

At a high level, CJ Moses, who presents the webinar talks to the core areas they covered in the control objectives, which are:

1. Security organization
2. Amazon employee lifecycle
3. Logical security
4. Physical security
5. Environmental safeguards
6. Change management
7. Data integrity, availability, and redundancy
8. Incident handling

This looks pretty reasonable at a high level.  Of course, it would be nice to see the actual controls and objectives, but at least they are covering the appropriate areas of security.  I do notice that there isn’t much around perimeter or related security.  I’m guessing they are trying to gloss over the AWS distributed firewall.  It would be nice if someone besides Amazon was vetting the way this was built.  They appear to consider it a piece of core intellectual property despite the fact it would be trivial to reproduce.  I’m not exactly certain why.