Originally Published: Monday, 11 December 2000 Author: Andrea Glorioso
Published to: featured_articles/Featured Articles Page: 1/1 - [Printable]

Worth 1000 words

Andrea Glorioso proves that a picture is worth a thousand words as he offers a rare inside glimpse at Linux Service Companies, compares service offerings and tells us what the future holds for them.

   Page 1 of 1  

It's often been said that these days we are living in are the epitome of "the virtual": everything is digital, intangible; in a word, "virtualized". The rise of the so-called "Linux Service Companies" are maybe one of the best examples of this trend: business companies which do not provide neither tangible goods, such as cars, nor less tangible but nevertheless "branded" goods, such as software. Instead, what they sell are services, and this article tries to give a brief overview of what the heck that means, and if this could be the answer to the everlasting question "How can you make money with software you can get for free?".

Why am I entitled to talk about this?

Freedom of speech would allow me to talk about Linux Service Companies (LSC from now onwards, to follow our glorious tradition of geekdom and acronymitis) even if I didn't know anything about them. The fact is, I actually worked for one of them: Linuxcare Italia S.p.a., the Italian subsidiary of Linuxcare, Inc. Linuxcare Italia was closed together with all European operations something like a month ago- and I'm writing this to clarify my position- but the time I spent with them has taught me something about what an LSC actually is and does.

What services are
A rose by any other name would smell as sweet, but would still cost money

A "service" is just what the name itself says. From Webster's dictionary, a service is "a useful labor that does not produce a tangible commodity". So, an LSC can be thought as a business that, instead of pushing "products" on the market, listens to customers' needs and reacts accordingly, "producing a useful labor" which satisfies the counterpart's requests. We'll see that the distinction between "products" and "services" is sometimes quite blurred, especially from a marketing point of view; but it is nonetheless a useful difference to start with.

One thing that usually distinguishes LSC from other Linux players in the field, namely distribution companies such as Red Hat, SuSE, Mandrake, and TurboLinux (just to name a few of the most important), is that service companies like to stress their "distribution independence". For a LSC, this makes a lot of sense, because their first goal is to furnish services for Linux- not for Red Hat Linux, nor for SuSE Linux, nor for any other distribution-peculiar Linux. Distribution companies usually have their own channels for tech support, and some (like RedHat and SuSE) are investing a substantial amount in the "services" area. The fact remains, nonetheless, that LSC like to stress their distance from any single vendor, and while we can have various degrees of "independence"- from Linuxcare, whose motto is "Linux is Linux is Linux", to VA Linux, which happily installed Red Hat Linux on their servers before offering to mount Debian, too- "distribution independence" is a trait by which you can easily discern LSC from other Linux businesses.

Someone once said that a picture is worth a thousand words, so let's give a brief overview of which kind of services LSC usually have in their portfolio.

Technical Support

It's no surprise that one of the strongest obstacles that have been standing in the way between Linux and the enterprise is the lack of support. Linux has always been used in many big companies, often deployed in quite important roles, even if marketing reasons caused such companies to present themselves as "Windows NT'' or "Solaris" or "Whatever else has a nice logo" powered. IT decision makers would hardly even consider the possibility of "officially" adopting Linux as a solution to their problem. Whilst this has always been a reason of scorn for many experienced hackers who were forced to work on crappy software, the suits had their reasons. In almost all IT environments, especially those with a medium-high level of complexity, what really matters is not that everything works well (this should be a prerequisite of having a good IT environment) but that you have someone to call when something breaks: it's in these situations that Murphy's law reaches its full potential. Inevitably, something will break, and when it happens, you want someone who knows his job to solve the problem for you.

Although this attitude will surely cause a natural feeling of repulsion in the hacker part of our soul, it's actually quite a sensible way on thinking when you have something more than three servers to look after; it's called risk management.

It's obvious that what we are talking about here is not your usual toll-free number to call when you can't manage to squeeze three floppies in the drive at once ("Help! The instructions say I have to insert all the floppies I found in the box, but I can't manage to put more than two together!"). "Serious" LSC, if they want to survive in an environment that is becoming harsher day after day, have to provide a very professional tech support structure, with a multi-level, scalable and medium-independent (e-mail, web, telephone) approach.

Linuxcare, for example, started as a pure technical support company, and later on switched to a full-blown service company (not without having its problems in the process, by the way). Today, Linuxcare's tech support offering is perhaps one of the most variegated ones on the LSC market, presenting tech support solutions which range from its searchable web-based knowledge base, to the "experts-exchange" powered forums, to more standard multi-level tech support, both via telephone and via e-mail. Linuxcare offers various contract types, starting from "one-shot" phone calls up to extended tech support contracts, as well as the possibility for a company to completely outsource its own tech support structure to Linuxcare- the end user phones to Foobar, Inc.'s tech support, but it's Linuxcare employees who actually answer the call with a smiling "Hello, Foobar tech support, Greg speaking. May I help you?".

Tech support is usually modeled with a multi-level approach, i.e., when the question arrives, first line employees assert its reception and read it. If it's one of the "standard" questions (that's what a central problem database is useful for, fulfilling exactly the role of FAQ lists on USENET, except that you cannot tell the customer to RTFM) the problem is closed without further hassle. If this is not the case, however, the issue is "scaled up" one level, where "techies'' analyze the problem and usually give an estimate of how long will it take to at least say if they can resolve it or not. In the latter case, the issue is scaled up again one level, until it reaches the level when it's not a "tech support" problem anymore and it's food for the Professional Services unit (more on this later on- note that, of course, not all LSC use the same terminology, although "Professional Services" is a surprisingly common term to express the same meaning).

I cited extensively Linuxcare in this short presentation of tech support services just because it's something I know quite well. Actually, VA Linux tech support offering is very similar, apart from the different names (but as you know, a rose by any other name.).

The interesting point about tech support offers from LSC is that it's not something you put up in a hurry with two friends and a cell phone (although some LSC admittedly began this way: but nowadays the game is a bit more harsh). It needs coordination, a very good infrastructure (PBXs, issues tracking software, central problem databases and so on) and a bunch of people who know what they are talking about.

Actually, it looks like tech support is what LSC are doing best on the "services'' front, at least until now. While many companies had to take some drastic decisions regarding other units, tech support staff seems to always be a scarce quantity, not to be cut without a very good reason. My personal opinion is that this is very coherent with what we said earlier, i.e. that the IT business, more often than not, is not happy with a solution which works, but needs a solution which works and which is guaranteed to work, or at least which is guaranteed to have someone to blame if it doesn't work- and Linux, from this point of view, is still an almost totally virgin market.

Courses and Education
Knowledge is power but you must know how to manage it

So you don't trust your IT personnel enough to deploy Linux on critical nodes of your business. On the other hand, you can't afford to call your preferred LSC technical support number (or write them an e-mail, or have a look at their web forums, or whatever else they came up with this time) both for economical and for scheduling reasons- when you have to pick up the phone for every single character your should type at your terminal, deadlines tend to slip forward in definitively.

Or, you are on the other side of the social pyramid of your IT infrastructure, and you are sick of being a puppet in the hands of those merciless tech support guys who, as soon as you try to pose a second question, gently stop you telling that "your support contract only allows for one question for each call".

Or, you are a training company, and you continuously receive requests for courses on Linux, such as introductory classes, system administration, development, specific software packages, and so on. You don't have the resources in-house to offer such courses, so you call a LSC which has courses in its portfolio (and almost all of them have several kinds of "education offers", although named differently, but we already know the story about the rose) and outsource them the "human element" of the courses (the teachers and the material) whilst furnishing yourself the material infrastructure needed, i.e. workstations, network connectivity, slide projectors, and so on.

As you can see, there are numerous reasons why a LSC would want to include some kind of education-based offer in its portfolio- and that's what actually happens, with various degrees of importance given to the education division. For example, Linuxcare- again, I speak of them just because of my past experience; I don't endorse any particular business company- has always stressed very much the importance of Linux education with its "Linuxcare University" business unit and a detailed portfolio of courses, which had amongst its goals LPI exams training. (Actually, Linuxcare hired Dan York, of LPI fame, although he doesn't work for them anymore.) Innominate AG, a European based LSP, is moving in that direction, too.

The main problem with this kind of service is that teaching is a very difficult task. As anyone who has ever attended some kind of "education facility" (be it high school, university, private lessons, talking to a colleague) will know, you can be a top level guru in your field and still not be able to transmit your knowledge to your students. Technical difficulties apart- it's sometimes difficult to manage large classes when you have to watch after each student's workstation, checking why his/her "ping" program doesn't behave like it should and why the student next to him/her has got a nice "kernel panic" smiling on the screen- there are very human reasons why a course can be a success or a total failure.

And it's in cases like this that the Human Resources department of a company has to show its full talent, trying to quickly determine if a person, even if technically able, is suited to the delicate task of representing the company educational business to an audience composed of complete strangers, where the feedback and the potential for disasters is much more present and less "coverable" than with other kind of business. In the education field, an awkward move can be your last move: maybe not today, given the relative scarcity of (supposed) "Linux experts", but not too far in the future.

My personal impression is that LSC, although pushing a lot on education and courses, are not fully aware of the potential for disasters that a hasty move could produce. This is an opinion from the trenches, of course: but LSC tend to adopt the same strategies they use for other business units to courses- and that's a gross error.

Professional Services
Call them as you like, it's when the customer wants to order something off the menu

As my tentatively ironic title says, "Professional Services" (also called "Consultancy" and many other, even stranger, names) cover those jobs which don't fit very well in the other business units of a LSC. This is not to say that they are the "Cinderella'' of a service company; instead, as long as a company actually has the manpower to fulfill the kind of tasks which professional services can be called up to, this can be a big source of revenue.

I must admit that I'm a bit biased (in a positive sense) towards professional service, partly because my job at Linuxcare Italia, which I enjoyed as long as I could do it, was as a "consultant" in the proserv business unit, i.e., the technical interface between the customer and the developers, discussing what the customer actually needed, what we could do for him, how long would it take for us to do it, how much it would cost, and so on.

"Professional services" can cover a very wide range of topics. If we examine three companies' proserv offerings we can get an idea of what LSC have in their portfolio.

Table 1: Linuxcare Proserv offering
Open-Source strategy consulting
Linux and Open Source development
Open-Source deployment
Custom Solutions (Linux Distributions) program

Table 2: VA Linux Proserv offering
Initial Code Offering -- taking proprietary software to Open Source
IT Assessment Practice
Security Assessment

Table 3: Innominate Proserv offering
Embedded Systems

My first impression, when I had a look at the proserv offerings of the different companies (of which those presented are just a part), was that they all had a certain "vagueness'' in common. Although, to be honest, a certain part of this lack of precision (what does "Open-Source strategy consulting'' mean, for example?) can be figured by a careful visit to each company web site, the various "professional services" business units are not very well defined.

This vagueness is at the same time the main strength and the principal weakness of LSC professional services. It is a strength because it encourages many types of customers to ask for help from a LSC: if an advertisement said that company Foobar only did network device drivers, for example, it wouldn't make much sense for a customer to ask them advice for a security audit of their IT infrastructure. On the other hand, this can very well be a weakness, because it's common sense that no individual and no group can have competence in every possible area of expertise.

This wouldn't be a problem if company Foobar, faced with a task for which it has no specific competence, just told the customer the truth. What usually happens is that the larger a company is- and LSC tend to be quite large anyway; if you plan to build up at least two business units, you need a good economical investment, and at that point being "large'' is both a good marketing move and a public relations incentive towards those who invested in you- the bigger is the lack of communications between "sales'' and "delivery'' people (I use the terms loosely). So, what usually happens is that the sales department is genuinely convinced that Foobar is capable of handling projects the most disparate, while the delivery departments are continuously faced with different kinds of jobs. They try to do their best and, thanks to the general lack of true accountability for these types of projects (I'm not saying that developers are liars: it's just that it's harder for a customer to understand whether certain requirements have been met, especially since they are often blurred from the very beginning) they usually manage to come up with a decent solution; but this way they are not able to deepen their expertise in a specific field: and the situation gets worse every time.

Given that for many LSC the proserv unit is a central point in their business strategy, a lack of focus can be really disastrous. You need very capable managers, at all levels, to handle such situations- especially because Linux and open-source geeks in general are not the kind of guys you can tell "do this and don't ask me why'' to. USENET old timers will surely know the old motto about newsgroup moderators: it's like "managing a herd of cats". This kind of difficulty can very well arise in a LSC, especially when focus on specific areas is not taken as a primary objective.

I won't go as far as to say that the difficult moment which several service companies are traversing right now is due to this lack of focus. But all other external factor notwithstanding, speaking from the trenches again, this is a problem for proserv units.

The fact that the majority of LSC are basing at least a part of their offerings on specific "products" (more on this later on) instead than on "services" is a clear sign of this problem: at a certain point you have to clear out the vagueness and focus on very specific tasks, or you risk to perish altogether.

There is no rule without its exceptions

As I hypothesized above, Professional Services units as LSC define them nowadays are way too generic to survive on the long run. Whilst a certain degree of selection (and, hopefully, strategic management decisions) will contribute to focusing the proserv offerings, a common trend in LSC is to guarantee at least basic revenues (both actual revenue and future revenue, based on marketing, branding and support strategies) with so-called "products".

A "product" is quite a different thing than a "service". The main difference is that a "product" can be branded much more easily, i.e. the company can associate itself with that particular product- something which a service doesn't allow very easily. It's like when someone says "Photoshop" and you think "Adobe", or "crap" and you think "Micros." -woops, my fingers just slipped.

Usually "products" and "services" can be and actually are combined- for example, VA Linux offers several servers, each one tailored to a specific task (Application Servers, Database Servers, Web Servers, and so on), whilst Innominate offers Linacle (an Oracle-based server appliance) and Lintero (an all-in-one router/firewall solution); each company usually offers customization and support services on these products, too.

I don't think that products will be deleted from the various LSC portfolios anytime soon (not that I would like this to happen, anyway: often they are quite nice pieces of software/hardware). It's nonetheless interesting to note that the "S'' in LSC hasn't been yet totally obeyed to- so, should we call them LSPC (Linux Service and Products Companies)?

Is this whole LSC business successful or not?

This is a tricky question. Honestly, I don't want to proceed on with a detailed economical analysis of the current situation and of the possible future outcomes, for two good reasons: first, this is not "The Wall Street Journal'', and second, I just did a basic economy exam while back at University.

What I can say to those who had the patience to read till the end of his article are just my personal impressions; and my personal impressions are that Linux Service Companies have an enormous potential waiting to be fully unleashed, but they have several problems which have to be addressed if they don't want to perish or, to be optimistic, just continue to float around waiting for the next venture capital firm to buy them.

My personal impression is that tech support and courses will continue to be a strong source of revenue for LSC. As in all businesses, new ideas- together with the management ability to organize them in a coherent fashion, and the staff capability to implement them- will help tech support business units to guarantee a certain level of revenue. As was mentioned in the article, simple toll-free numbers, whilst a good starting point, are not sufficient anymore. Web-based knowledge bases, USENET-like newsgroups, "experts-exchange" style forums are all good ideas which have worked well and are still doing their job: but if LSC want to stay abreast in the tech support market, they will need both to improve the quality of these services (good search engines for web-based knowledge bases, for example) and find new ones, which will have to be appealing for the suits and for hackers: because the latter will use tech support services (especially if they will be offered via IRC. wow!) but the former will pay the check, and they won't do it for something they don't understand and hence don't find valuable.

Courses are will be a big source of revenue for a long time. Ours is an era where one of the most valuable assets is knowledge- hence, either you pay for someone else's knowledge (consultancy) or you try to get that knowledge. While self-study is sometimes a good choice, it makes sense for a company to plan Linux (and open-source software/methodology/development) training in an organized fashion. My only concern with current LSC approach regarding courses is that in my experience they consider it as some kind of "product"; and while you might as well think about it this way for billing and accounting purposes, education is very different from software development both in terms of methodology and of the people you choose to deliver it.

Professional services are a good source of revenue, as well. In my opinion, they are one of the most interesting and funny jobs you get to do with Linux (and Open Source software), especially when you get to work with the customer from the very start: on the other hand, I already mentioned the problems deriving from lack of focus, so I won't repeat myself another time here (I've already done that enough).

Maybe my experience with a LSC has really taught me are two things: first, that Linux Service Companies shouldn't be too worried to present the customer with open source solutions, whenever that's technically possible. What sometimes happens, instead, is that when having the possibility of choosing between offering two solutions, one closed source and one open source (and that's not always possible, especially when you play with the "big boys" in the business arena), a company is tempted to offer the closed source one because it seems more "respectable". Well, my personal opinion from the trenches, again, is that respectability is something you earn by showing that your solutions work as expected (not better than that, you don't want to accustom the customers to well) and cost less than your competitors, and not by hiding your commitment to open source (if that exists, but let's suppose so for the sake of discussion) in fear of being considered just another "leenooks hacker". Open source methodology and tools give a company all the possibilities it needs to gain this respect, and the bottom line is that either you believe this or you are just trying to make money with a fashionable idea; but, fashion is by definition quite volatile, while a company is not (or should not be).

The second point is that everyone should do their job: hackers are hackers are hackers, and managers are managers are managers- and this applies to all relevant roles in a company. You don't send a salesman to do a developer's job, nor do you assign a developer to manage accounting issues or make marketing decisions. If a company is healthy, there will be a natural flow of communication that will allow for each one of these roles to gain from knowledge of each other party. If you want to start play in the big league, you better know the rules, or hire someone who know the rules of both worlds: the open source and hacker culture, and the business culture. Whilst it's very difficult to find such people, I think that a LSC who doesn't at least strive to do this is doomed to die in a short time.

Footnote: Many acute philosophers have tackled the issue of what "virtual" can actually mean. If you are interested in this subject, I suggest you check "Le virtuelle", by Pierre L'evy; you could be surprised by how much philosophical thinking applies to everyday IT tasks.

Andrea Glorioso is a brilliant young Italian with a refreshingly wry sense of humor, an impressive Curriculum Vitae and a penchant for Barilla pasta. You can reach him at sama@aglorioso.com.
Copyright (C) 2000 Andrea Glorioso. All rights reserved. Redistribution, in whole or in part, is allowed only with prior notification to and written consent from the author.

   Page 1 of 1