What would Cisco do?

The biggest problem with Linux server operating systems, whether its Red Hat Enterprise Linux, Novell's SuSE, or appliance new comers such as rPath Linux, is that they are all cut from the same cloth. There is no real innovation here. Sure, Red Hat occasionally will throw us a bone with things like GFS, or some cool innovative features in the kernel. There are very few operating systems willing to dance outside the safety of the status quo.

The problem is with the school of thought. The school of thought across all the developers and great folks that work at these companies is the same. There is nothing new there. If you look at rPath, you have Erik Troan, he was one of the original authors of RPM, the Red Hat Package Manager. Billy Marshall, CEO of rPath, he is ex-Red Hat too. Red Hat Enterprise Linux and SuSE, both RPM based, as is Mandriva. Even Ubuntu and Debian, while not RPM based, still have developers that essentially came from the same community with the same concepts that we've all seen Open Source grow up with.

What about Virtualization I hear you say? Virtualization is new. Yes it is, but what do our friends do? They bring package management and the very same line of thinking into Virtualization. It all comes down to one thing, its safe. The big companies like Red Hat and Novell, simply can't change over night. They are heavily invested in RPM, they have third party software vendors, certifications, and in essence, changing from package management, would be an admission that they were wrong. Why do startups follow this Lemming-like path? Well it makes it easier for customers to jump ship.

On the other side of the fence, you have companies like Cisco. Wait, what has Cisco got to do with Linux distributions? Whether you like it or not, Cisco is trusted with a major chunk of the Internet and corporate networks. Corporations are funny, they won't typically turn their business over to just anyone.

The Cisco 7200 series router is the workhorse for many businesses. Just like their other router and switching products, the 7200 runs IOS, their Internetworking Operating System. They use package management right -- NOT! IOS is shipped as a firmware image, a self contained system. When you upgrade from one version of IOS to another, you are completely replacing the system and rebooting. That IOS is specifically tested not only to work as a system, but to work on the very hardware that it runs on.

Look at all the businesses that provide critical wired or wireless infrastructure - Cisco, Nortel, Aruba Networks, Foundry, F5, Redback, Motorola etc. Look at your wireless router at home, it too provides a firmware image - self contained and tested.

This is the proven, time tested method for Enterprise networking, so why doesn't someone apply this to servers? Plain and simple -- school of thought. The folks behind the Linux distributions don't come from these backgrounds, and the few that may do, simply stick with the status quo.

That is of course with one exception -- AppOS, developed by Spliced Networks. For the past 5 years, Spliced Networks has applied the very same principles that make Cisco routers -- mission critical / enterprise ready to the Linux server. With AppOS 4.0, we'll make that very same solution available, de-coupled from the hardware!

Goodbye Package Management

Package management on an appliance is wrong -- plain and simple. The whole concept of an appliance is to have a contained system that has been fully tested and quality assured. When you introduce package management into such a system, you immediately lose the integrity and quality of the system.

Why ? A network-enabled package management system increases the number of variables involved for testing exponentially. A customer running a version of package X and a version of package Y , might decide to upgrade the version of package X, but has some custom application that upgrading package Y will break. This has produced a new type of system, one that has version 1 of package Y, and version 2 of package X. This might seem fairly simple, just create a QA test case for this scenario. Most Linux systems have hundreds of packages, as you start to scale up, you see that its no easy task to actually test each one of these scenarios. The moment you run update with yum, apt-get or conary, you've immediately stepped away from the QA blessed version of the operating system.

The problems however, don't stop there. Package management is making live changes to libraries, and executables on your system. The package management software is essentially, removing and install, or installing over the system. With automated systems such as conary, and systems you could automate like yum / apt-get, you run into a situation where human error is a factor, as is the integrity of the repositories you have to trust. What happens if in a rush to get a security fix out, a package isn't fully tested with the entire system, you grab your automated update, and next minute, your PHP applications stop working. Your only option is to roll back, and rolling back with package management (ask anyone who has used rpm for a few years) is a very hairy situation to be in.

What happens if your system is running a third party application, or your own custom application. It might have files open, that the package management system is trying to upgrade, and hence the package update fails unless those applications release the lock on the files in question, such as libraries.

Then of course there is the problem with your configuration. What happens if the system decides to update maybe bind, and this new release of bind may have changed a particular option you are using. Unless the package management system is designed to cater for this scenario, and in testing, we've found none that will do this effectively, your system is rendered unusable without manual intervention.

Package management is perfectly fine for the desktop, if something gets upgraded and it turns out to break your system, while its annoying, its not catastrophic. The same can't be said for the server or the appliance.

While some vendors would like you to think that a package management system is the way to go, its not. The only real advantage to package management on an appliance, is not to make the administration easy, but to make the development easier. In fact, package management adds a new point of failure and adds complexity to administration.

The real benefit is for the developer, yes thats right, the folks who are supposed to know their stuff, that you are entrusting the integrity of your business to, package management dumbs down their tasks for them. You might say, but this whole packaging system has complex configuration, how could it be dumbing things down?

Things are being dumbed down because the developer just needs to use a nice web interface, pick the packages that they need, perhaps add a few packages of their own and suddenly they are pitching to you this "Enterprise Ready" appliance solution. The fact of the matter is, they don't know when they imported openssl, who built it, if it contains patented algorithms, is it exportable and what extra patches it might contain. In some cases they could find that information out, but honestly, when they can click and roll, do you really think they have?

So whats going on here is essentially the "dumbing down" of the developer, the very person(s) who you trust your business to. Package management is bad all around for appliances, and it is far from Enterprise ready. The next time someone tries to tell you Ubuntu Server is Enterprise ready, ask them if Ubuntu has QA'd all the package management combinations, and you'll get a resounding... NO..

High Quality Articles Prevail

Despite the fact that we didn't make it to the front page of Digg.com or Slashdot.org, we were still able to bring home the bacon. As of this morning, o3 magazine had 208,898 readers of Issue 6. This is not bad at all considering we were off on hiatus for a year, and the issue is highly specialized!

For those interested in the geographical stats, here you go:

North America = 36%
Europe, Middle East, Africa = 54%
Asia Pacific = 10%

We expect to see at least another 100,000 readers pickup Issue 6 over the next 90 days. With past issues, not everyone is interested in a particular issue, but a future issue will get them hooked. They'll like the high quality articles, and head back for more by checking out past issues!!

Issue 7 is right around the corner.. stay tuned!

o3 magazine is back

On Monday we released Issue 6 of o3 magazine, the FREE digital Open Source / Business magazine that Spliced Networks started back in November 2005. So far, I am very pleased with the results since it has been almost a year since the last issue. The magazine is regaining momentum, and after only a few days.

Mayank Sharma is doing a great job with the magazine, Issue 7 will be out in a few days, covering Agile Product Management, again this is Open Source solutions that we are using in-house, so we're still drinking our own Koolaid!!

The GSLB solution is doing very well, considering it was pieced together over a couple of weekends. Nginx is an awesome piece of Open Source software, and I'm glad I found it. Hopefully I will get some time to submit a few patches to help polish its load balancing capabilities!