For those who are new to OpenStack or simply haven’t had time to delve into it’s innards, Nova uses a core asynchronous RPC/messaging system to communicate between components.  Asynchronous message passing systems are key tools in building scalable systems.  They are used in many different ways depending on the type of system you are building.  Nova has traditionally used RabbitMQ, which is an excellent choice for many.

Cloudscaling felt that RabbitMQ added unnecessary complexity to Nova deployments given what the requirements were for an asynchronous RPC message passing mechanism.  Rabbit provides a number of value added features such as store-and-forward of messages, which can actually be problematic at larger scale.  In addition, Rabbit is a centralized, broker-based model that requires deploying it in High Availability (HA) pairs and combining this with clustering (multiple HA pairs in parallel) for scaling.  We felt this was not an ideal scenario for larger Nova deployments.

The result is a completely broker-less, peer-to-peer, asynchronous message passing system based on 0MQ (ZeroMQ).  This open source pluggable driver for Nova has no centralized broker, requires no HA components, and is distributed amongst all of the Nova components.  The ZeroMQ plugin that Cloudscaling created is roughly 700 lines of Python code that plugs into the Nova Queue Abstraction Layer (QAL).  It leverages the ZeroMQ library (written in C), which is fast, portable, and proven at scale.  You will find that this plugin not only removes complexity in how Nova does messaging, but also has its performance scale linearly with the number of components.

As a brief aside, this is an example of how Cloudscaling intends to help create choice and increase the robustness and scalability of OpenStack, while reducing complexity.  Last year, we pushed back the pluggable QAL interface to OpenStack in Diablo with a driver for RabbitMQ, which maintained all functionality, while allowing us to have a place to plug in ZeroMQ in the future.  Now, as of Essex, Stackers can choose RabbitMQ or ZeroMQ depending on what they think their needs are.  We hope this is an approach that others in the community will choose.

Eric’s presentation on the ZeroMQ RPC driver for Nova is below.  It describes how ZeroMQ works and provides more depth on how it was implemented.

That was in July of 2010. What I said was:

“OpenStack, with a strong community behind it, should be an important tool for service providers and large telcos to compete at scale with the Amazon and Googles of this world.”

It’s nearly two years later, and there is definitely a ‘strong community behind’ OpenStack. More than 165 companies have joined the project, from startups to multinational giants. And they’re not just lurking and watching. They’re contributing code. In fact, more than 200 Stackers from 55 companies contributed to Essex, the release announced last week. Want more diversity? There are more than 20 global user groups, and there are 2,600 members of the technical community.

But is OpenStack becoming ‘an important tool for service providers and large telcos to compete at scale’?

There’s no doubt about it. Companies from KT and Internap to Deutsche Telekom and HP have launched production-grade deployments on OpenStack. Beyond these marquee deployments, consider the 100,000 downloads to date.

And remember, it’s not even two years old yet.

Today, the OpenStack community is taking the next step toward becoming the most profound open source movement since Linux. Eighteen companies have stepped forward as platinum or gold members of the OpenStack Foundation. These companies are making a financial commitment to help assure the long-term, independent success of the community. Each of them committed code to Essex. They’re vested.

One of the things that makes OpenStack work is its open development process where technical  meritocracy drives the code. The community is simply too diverse for one company to impose its will. With the formation of the foundation, this philosophy becomes codified into the DNA of how OpenStack functions.

Credit Where It’s Due
I’d be remiss if I didn’t call out Rackspace for their thoughtful and balanced stewardship of the project to this point. Jim Curry, Mark Collier and their team have guided OpenStack toward today’s foundation announcement. Rackspace, and the entire OpenStack community know that the formation of the OpenStack Foundation is the correct next step to move the mantle of leadership to the community itself.

Getting on With What’s Next
It’s as true today as it was in July of 2010. OpenStack will succeed because of the diversity, engagement and energy of the community building it. And we’re not done building that community. There are tough questions to answer about governance and competition. But as the community takes care of its internal business, the market will take care of the competitive issues.

To my fellow Stackers, we’ll see you next week at the Summit to get on with the business of self governance. And join us on Monday night for a party with our co-hosts SolidFire and RightScale. There’s plenty to celebrate.

The basic premise of the article was that Citrix was making pretty outlandish claims about being the only one who “even comes close” to “meeting [the] requirements]” for “AWS-style architecture”, “proven at scale in real production clouds”, and is “compatible with the Amazon architecture.”

These are big claims on Citrix’s part and an attempt to set themselves apart from everyone else primarily on the strength of their Amazon compatibility.  To let such claims go unanswered, when they are formed on such a weak basis, is not possible.

The Article & It’s Reception
There isn’t much reason to retread my article on GigaOm.  It speaks for itself.  I focused not on CloudStack vs. OpenStack, but endeavored to talk about the claims around AWS compatibility and Amazon-style architecture.

Tweets were exchanged with the chief cloud architect from Citrix and Christian Reilly, one of their big customers, proponents, and briefly, an employee of Cloud.com/Citrix.  The result was that Citrix made it clear that AWS compatibility *is* possible with CloudStack 3.0, since February 2012, if you buy a Citrix NetScaler.

In other words, on an open source project to open source project comparison, CloudStack has no more additional AWS network compatibility than does OpenStack.

Here’s what I said in the GigaOm article[1]:

 The default AWS networking is a flat layer-3 network, which CloudStack can support, but typically does not. Much of their value-added functionality (e.g. load balancing, network area translation) disappears when using flat networking. Default CloudStack deployments aren’t network compatible with AWS.

My stance was clear in the original and stays the same.  If you download CloudStack, turn it on in flat network mode, like Amazon networking, you lose a bunch of CloudStack’s purported value add and are not any more compatible with AWS networking than OpenStack.  Citrix stance is simple: buy a NetScaler to fix this.

“Designed From the Ground Up”
Here’s the CloudStack 3.0 release notes declaring support for the Citrix NetScaler.  CloudStack 3.0 (Acton) was released on February 28th, 2012.  So native support of their “designed from the ground up for AWS compatibility” for ELB, Elastic IP, and NAT in flat networking mode (Amazon’s default) is now 2 months old.  This support is provided by purchasing a Citrix hardware appliance, which is the only load balancer currently supported by CloudStack today.

Citrix Isn’t More Designed to be AWS Compatible Than Anyone Else
I called Citrix out on some outrageous marketing claims that are minimally supported by the evidence.  CloudStack was not designed from the ground up to be AWS compatible or in the Amazon architectural model.  Nor is it significantly ahead  of projects like Eucalyptus or OpenStack in terms of AWS API features supported.

This isn’t about OpenStack vs. CloudStack though some are trying to make it that.  This isn’t about whether CloudStack is more open source than OpenStack or vice versa.  It’s about these marketing claims.

It comes down to this: You can build an Amazon-style and possibly even somewhat AWS compatible cloud using CloudStack IF you also buy the Citrix NetScaler.  The same can also be said of OpenStack and possibly even Eucalyptus with a modest amount of integration work.

Adding integration to hardware projects in the last two months does not make true the claims of CloudStack software being “designed from the ground up” to be the “only one who comes close” to AWS compatibility.

[1] This is backed up when you read the CloudStack 3.0 Advanced Installation Guide, which shows that you must purchase a ‘physical’ device to get NAT, ELB, Elastic IP, and firewall capabilities.

As I write this, Citrix is preparing a big announcement tomorrow. The details are sketchy, but apparently they are joining the Apache Software Foundation (ASF) and contributing back Cloud.com’s CloudStack infrastructure-as-a-service (IaaS) software. It’s possible they may even contribute Xen, although I don’t have any indications of such.

Citrix paid a pretty penny for Cloud.com. At reportedly 200M[1] and possibly as much as 40x revenue, it was a huge bet and now it appears destined to be transferred over to ASF lock, stock, and barrel. At the time of the acquisition, it wasn’t clear what the Citrix strategy was given they had made a big public commitment to the OpenStack project (whither art thou, Project Olympus?). It was never clear if they could bring the two projects together as they claimed they would given the likely tissue rejection (Java vs. Python anyone?). Now, it appears Citrix is not only going their own way, but possibly planning to leave OpenStack behind.

So, what’s happening? Is this Citrix throwing in the towel or is it a new opportunity to drive greater awareness of open source and to foster the cloud computing space in general?

It’s hard to say, but we can be certain of a number of things.

First, Citrix’s focus has always been VMware. Their virtualization strategy has largely been focused on building a fast-follow model (XenServer+XenCenter) to VMware’s wildly successful vSphere system. The more cynical might see the Cloud.com acquisition as a similar move to create an alternative to VMware’s vCloud Director.

Second, Citrix has struggled to integrate CloudStack and OpenStack or to have a cohesive strategy or statement on how integration could even be done. Meanwhile, the pressure to see a return on investment for CloudStack must be very high. Remember, this is the second major open source project they have invested in. It’s unclear if XenSource, acquired 4.5 years ago for $500M, was ever a net win.

Third, many believe there is currently a three-way ecosystem race: Amazon Web Services, VMware, and OpenStack. Certainly Citrix doesn’t want to be left out of the party. If they are moving away from OpenStack and can’t move towards VMware, that only leaves two options: move towards Amazon or try and create a fourth ecosystem. Perhaps that is a major motivation for joining ASF?

Fourth, while Citrix’s CloudStack has been officially open source for a while now (GPL, to be exact), it hasn’t seen the uptake or contributor community traction that OpenStack has seen. As the fastest growing open source project in history, OpenStack can make some serious claims in terms of the number of contributors and velocity. CloudStack, meanwhile, has been predominantly an open source project of one, first Cloud.com and now Citrix. Joining ASF adds a sense of open source legitimacy and may be a defensive strategy on Citrix’s part to avoid obsolescence in the face of OpenStack’s mindshare dominance, while trying to gain more mindshare against their old rival, VMware.

As an aside, it’s great to see more validation that open source solutions are highly desirable for building open cloud infrastructure. We welcome as many companies into the fold as possible. Hopefully CloudStack will have more success as an Apache Software Foundation (ASF) open source project than it has enjoyed solo.

So what’s happening here? Is Citrix giving up? That seems unlikely. Are they giving new life to CloudStack as an open source project? All of the momentum continues to be with OpenStack at the moment. Joining ASF seems like a bold and smart business move, but it could also wind up being quite a big flop if a community doesn’t form.

Bold move or brash decision? Only time will tell.

[1] Their most recent 10-K says 158M in cash with 500,000 in stock + options or roughly 40M at today’s close of $80/share.

Story sourcing clarification: Some folks in the twitter verse were annoyed that I broke this story early.  These bloggers/pundits had been briefed in advanced of the Citrix announcement.  I was not briefed in advance.  I was notified through a tip.  I confirmed with a second source and then wrote this article.  Neither myself nor my sources were under any kind of embargo or NDA.

I applaud Joshua McKenty’s recent Open Letter. Cloudscaling has had similar concerns about the OpenStack Foundation governance model. We participated in a variety of discussions including the recent governance meetup during Cloud Connect, discussions with other community members, and discussions with Rackspace–the current Foundation steward–in private and public. We encourage all frank, even-handed, and open dialog about the governance of such a critical community as OpenStack. We look forward to working with all of the companies, both small and large, that are bettering their businesses on the success of OpenStack.

A rising tide raises all ships, and a stable, fair, and equally representative governance model is the foundation of all successful communities. We believe the OpenStack Foundation and the community it represents is at a cross-roads now where we can work towards an equitable and fair model that represents the interests of all participating members, be they large or small. It’s incumbent upon us to embrace such a model.

We will not be able to achieve such a model if we don’t have further conversation about the ultimate governance model of the OpenStack Foundation. A conversation that understands that a level-playing field is, ultimately, in the best interest of all. Even the large players.  A plutocracy will not facilitate OpenStack’s ultimate success.

We will have more to say on this soon.

–Randy Bias, CTO & Co-founder, Cloudscaling

Jesse Robbins of Opscode gave an exceptional 20-minute keynote at Cloud Connect last month. Check out the video of his keynote on the Opscode blog.

Also, the short video embedded below is a good summary about Opscode Chef and cloud infrastructure automation.

Finally, be sure to check out #ChefConf May 15-17 in San Francisco.

Much of the OpsCode team will be at their upcoming Chef conference, named, appropriately enough just #ChefConf.  Come learn from the best, share your experiences with Chef, or even sponsor the conference.  I think you will find it’s well worth your time. We have a number of Cloudscalers attending.  The early bird price is at a 30% discount  until March 31st!

This article aims to talk about complexity.  I hope for cloud building teams to understand this issue better.  In particular, I hope for many of the OpenStack folks to gain some insight into this way of thinking.

It will help us all to make a better open source project.

Systems & Complexity
I am no expert on systems theory.  A lot of what I know is based off of 20+ years of hands-on experience or anecdotes with building ISP backbones, datacenters, security systems, storage systems, and writing automation tools.  (Lots and lots of automation tools. I never could stand doing anything by hand.)  That said, over time I have slowly learned and inferred a number of lessons, the most important being

a. Complex systems fail.

b. People love to build complex systems.

In particular, many engineers see understanding and developing complex systems as a rite of passage.  In reality, the true test of a great engineer is their ability to make things simpler, not more complex. In software development, this is talked about as “elegance” or “code elegance”.  An excellent definition can be found in SearchSOA[1]:

“The word elegant, in general, is an adjective meaning of fine quality. Refinement and simplicity are implied, rather than fussiness, or ostentation. An elegant solution, often referred to in relation to problems in disciplines such as mathematics, engineering, and programming, is one in which the maximum desired effect is achieved with the smallest, or simplest effort. Engineers, for example, seek the elegant solution as a means of solving a problem with the least possible waste of materials and effort.” [emphasis mine]

Complexity is the opposite of elegance.  Complexity breeds failures.  Systems that are not designed for failure, which are complex or sprawling, will fail catastrophically.  Frequently catastrophic failure will turn into a cascading failure.  “High availability” (HA) as typically implemented in an enterprise datacenter will not protect a system from cascading or catastrophic failures.  More importantly, in my experience, HA is a leading cause of catastrophic failures. Traditional HA stems from the idea of risk mitigation, predominant in how enterprise systems are designed today. However, it is simply not possible to ensure robustness by predicting what could go wrong and adding complexity to handle a predicted range of failures. Cloud systems must embrace risk acceptance and planning, the new emerging approach for building reliably unreliable cloud systems.

Failures in Systems
There are a number of works on systems and failures, but my favorite is Systemantics or the Systems Bible by John Gall.  Considered somewhat facetious or tongue-in-cheek by many, much of its kernels of wisdom ring true.  The entire body of work is best summarized by “Gall’s Law”:

“A complex system that works is invariably found to have evolved from a simple system that worked. The inverse proposition also appears to be true: A complex system designed from scratch never works and cannot be made to work. You have to start over, beginning with a working simple system.”

Gall’s Law is a synopsis of Systemantics ‘laws’ 12-16 listed on the Wikipediapage:

12. A complex system cannot be “made” to work. It either works or it doesn’t.

13. A simple system, designed from scratch, sometimes works.

14. Some complex systems actually work.

15. A complex system that works is invariably found to have evolved from a simple system that works.

16. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over, beginning with a working simple system.

The inherent truth in Gall’s Law should be self-evident, but what’s important to understand is that many of the current approaches to building Infrastructure-as-a-Service (IaaS) clouds are deeply rooted in complexity.  In this regard, they look similar to how enterprise datacenters and applications are constructed today: heterogeneous, sprawling, multiplicity of silos with no prevailing design patterns or reusability. These kinds of approaches are difficult to scale, to secure, or to maintain with high levels of uptime. Systemantics talks very specifically about failures.  The subtitle of the book is “How systems work and especially how they fail.”  I want to bring a few of the more interesting, but pithy ‘laws’ to your attention including some brief commentary, particularly in the context of large cloud systems (numbered according to how they are found on the Wikipedia page for reference; will not match original text):

#22 – “A system can fail in an infinite number of ways.”

While infinite is an exaggeration, the larger a system is and the more moving parts it has, the more likely you are to encounter unplanned for edge cases.  You cannot plan for all edge cases.  You can only try to reduce the number of edge cases by having simpler systems with fewer moving parts.

#2 – The Fundamental Theorem: “New systems generate new problems.”

Systems are created to solve a problem.  Unfortunately, new systems bring new problems in addition to the ones you are trying to solve for.  People, and particularly engineers, want to solve problems by designing new systems.  Whenever possible, use existing systems to solve a problem.  If you must build a new system to solve a problem, make it the smallest possible solution possible to solve the problem.  Solve it as elegantly as possible.  Minimalism.  Simplicity.

Not in WP list – “If a problem seems unsolvable, consider that you may have a metaproblem.”

If a problem doesn’t look tractable then there is a deeper underlying problem that should be addressed.  A great example of this is that AWS doesn’t guarantee virtual server uptime through ‘HA’ or ‘server HA’.  Rather than trying to make virtual servers automatically recover, they simply addressed the metaproblem: the cloud app should handle server failures itself.

#31 – “Loose systems last longer and work better. (Efficient systems are dangerous to themselves and to others)”

This is what’s behind the notions of ‘loose-coupling’ and ‘failure domains’.  We see transient, isolated failures when systems are loose.  We see big, catastrophic, cascading failures when systems or subsystems are tightly bound or integrated together.  A few transient failures that are expected and planned for are always better than a very small number of very big, very spectacular failures.

Complexity in Software Systems
I can only touch on this briefly, but suffice it to say that software system complexity most often comes from more code.  In this regards, one of the main metrics in the OpenStack community (code check-ins and/or lines of code) is a bit broken.  What we really want is a metric for how people are simplifying the OpenStack code base, and making it more elegant.  Perhaps code check-ins where lines of code are REDUCED while maintaining functionality? Less code means less complexity, which means fewer edge cases and fewer failure points.

To see simplicity in action, let’s go back to AWS.

How Amazon EC2 Started (Simple)
When AWS EC2 launched, in August of 2006, you could get an m1.small for $0.10/hr in one region using an API.  That was it.  That was *all* of it.  You couldn’t even get m1.large or m1.xlarge instances until over a year later. There is a key lesson here: No one will compete successfully with Amazon by trying to build AWS as it is in 2012.  You’re going to build AWS as it was in 2008, or perhaps 2009, and then iterate fast.  As fast as you can.

Examples of AWS EC2’s Design Elegance
Even as AWS grows in scope, Amazon maintains simplicity in individual services, such as EC2. Some highlights:

• Only one hypervisor is supported [2]
• The default networking model is a simple flat layer-3 routed networking model
• Instance disk storage is ephemeral, meaning there are no SANs or NAS, just regular old DAS
• Elastic Load Balancing (and similar services) are ‘lowest common denominator’ capabilities: you get just simple L4 load balancing, not a complex L7 load balancer
• EC2 evenly subdivides physical hosts and ‘bin packs’ VM instances onto the same kind of physical hardware designed for that workload/VM-type
• Every VM instance has one network interface (NIC)

Some will undoubtedly point to the complexity of AWS’ Virtual Private Cloud (VPC) as a counter-example.  It’s important to note that VPC is a set of very complex networking designed to provide an emulation layer for legacy enterprise applications that were built in an era that required this kind of solution.  Most of these apps are not highly elastic and do not have large scale needs.  Solving for them with a VPC style solution is appropriate and sensible.  But you won’t ever see Netflix or use cases like Netflix adopting VPC.

Examples of Failures in Designing Scalable IaaS
This article is really becoming tl;dr, but before summarizing, I want to discuss some examples of what I see to be missteps in building IaaS cloud software and systems.

“Multi-hypervisor”

End-users don’t care about hypervisors.  Hypervisors are a commodity.  They are the new ‘bare metal’.  Multi-hypervisor adds code, complexity, creates edge cases galore, and creates all kinds of new requirements such as maintaining VM image repositories for each kind of hypervisor.  This is bad inside of a single system.

If you need multi-hypervisor, the right solution is to do what Christian Reilly did at Bechtel: deploy a number of systems and then use a cloud management system like enStratus or RightScale to bridge them with a unifying UI.

“Redundancy for redundancy’s sake”

Redundancy can be evil if done wrong.  Every redundant system adds complexity.  Redundancy adds moving parts.  Determine what is an acceptable and expected failure for the system and don’t add any more redundancy than necessary. For example, if we consider an individual compute node as disposable, investment in redundancies at the hardware level is not the best use of resources.

Build It Right
There are many other examples I could give, both positive and negative.  The challenge in building robust and scalable IaaS systems isn’t “will there be enough features?”, it’s “will we simplify and grow organically what we know works?” Right now I see more of the former and not nearly enough of the latter. In looking at your own systems, I would keep asking questions about the main inputs of complexity at the outset:

1) Features: What is the minimal set of services that you need to provide to your users to be a viable solution?

2) Options: Do your users really need more than one way to do things, to start? Flexible systems are rarely simple, ask Larry Wall.

3) ‘Best’ practices: Tried and true IT practices for small systems do not automatically improve a production cloud system.  They can, in fact, weaken it!  Context is key.

Let me know your take on this. Thanks.

[1] See also this piece on elegant code.
[2] Although in two modes now: PVM and HVM

The 12-minute podcast covers a lot of ground. Enjoy!

(Note: Requires Flash.)

Regardless, I got some props from Adrian Cockcroft, one of the primary cloud architects for Netflix.  I think very highly of Netflix approach to re-architecting their application to be ‘cloud-ready’ and to achieve high levels of uptime on AWS EC2.  If Adrian thinks this is right on, then it must be.

Enjoy.