# Battles with Proprietary Software Support

It always amazes me how much pain reliance on proprietary IT
solutions can cause, yet people go back for more time and time
again. Having worked in the past with a large, managed security
vendor, I've seen first-hand the kind of trouble relying on
proprietary solutions can cause. Here are just a couple of examples:

* This vendor's managed security service relies on some closed
components at its core, including backend databases, ticketing, and
alerting. Over the past year, the security industry has seen a
drastic change in what people expect from managed security vendors
(which is now a lot more than it used to be) - this vendor has been
unable to change quickly enough for the demands of the market, and
is now feeling competitive pressures from those firms that are using
Open Source components and can quickly adapt to what the market
demands. Internal development resources are underfunded to begin
with (the thought process was "Oh, good, an all-in-one solution, I
don't need developers, let's lay them all off". They then regretted
it when a solution had to be developed, because the old bunch of
proprietary components no longer fit with, well, anything).

* Providing support for proprietary software is a losing battle. The
aforementioned vendor provides some of the best Check Point firewall
support around, and has in previous years taken direct support
business away from Check Point themselves. Where the support falls
over, though, is the inevitable problem caused by software bugs
(there seem to be about 30 of those nasty Check Point bugs that
impact this vendor's clients each year), which they can **only** get
resolved by going to Check Point for support. As you might imagine,
since Check Point is the only company in the position of fixing
software bugs in their closed-source product, they have no impetus
to provide good support. Delays of a month or more are common,
regardless of the severity. Non-critical bugs sometimes never get
fixed.

Nokia's IPSO platform is the same way - while based on FreeBSD
originally, it is completely proprietary, and support vendors and
clients are unable to get bugfix support elsewhere. I have
personally been involved with two IPSO kernel bug-fixes (simple
ones, at that) that have taken over a year each to resolve.

Now, you might think "So what, don't buy it if you don't like the
support lock-in". But at some point this is like sticking your head
in the sand. If you believe Check Point, every single Fortune-100
firm uses their firewall software. How long will it be before a
serious bug or security vulnerability affecting Check Point impacts
not just the firm using their software, but that firm's clients and
customers? At what point do we look at the risks imposed on users of
technology, and give the potential impact of those risks more weight
than rights of the technology implementers?

This question is at the heart of most hot-button topics centered
around Free/Open Source software (FOSS), including the BSD/GPL
debate. The BSD license favors the rights of the developers, while
the GPL favors the rights of the users. Keep in mind that
"developer" and "user" might be third or fourth generation, so a
BSD-licensed piece of software that stayed that way through three
vendors can suddenly become proprietary - the users have just lost
most rights they had prior to that, if they use that software. A
GPL-licensed piece of software allows it's end-users the same
rights, no matter how many vendors have touched it (assuming they
each honor the license).

I had personal experience with some incidents that made me _really_
see the direct value that FOSS can bring to a business (as opposed
to just reading about such ideas). No really new ideas here, but
experiencing the problems with proprietary software first- or
second-hand, while being in the middle of a support nightmare as a
product support provider - is, I think, much more enlightening than
presenting theories.

In one incident, a large financial institution had many Nokia/Check
Point firewalls, configured in active-passive failover. In one of
their many two-firewall sites, they use dynamic routing (OSPF) to
direct traffic through a primary firewall, while a secondary
firewall sits idle, waiting to take over for the first, should it
fail. Check Point's state synchronization is commonly used in these
scenarios to share connection table information between two or more
firewalls, so if one fails, no users experience dropped
connections. At 1 AM one morning, the state sync between one such
pair of firewalls failed, causing the firewall daemon's log files on
the active firewall to be flooded with errors. The logging pushed
the firewall's CPU over the edge, and it was running at 90-99% CPU,
with 1-3 _second_ packet latencies. Needless to say, the client was
not happy, as it was impacting their business.

Still nothing too unusual there, except that this company, like many
others, pays quite a bit of money to keep their hardware and
software support "up-to-date". In this particular case, the Check
Point version had hit "end-of-life" during the time they were trying
to resolve the incident. Despite the support ending in June, and
this event occurring in May, the company was told that they would
not be able to get code support, should it rise to that level. So,
if a bug was found, Check Point _refused_ to even entertain the idea
of providing a patch for their software.

The solution, of course, as presented by the vendor, was to upgrade
to the newest, supported version of Check Point, while any bug found
would not even be guaranteed to be fixed. The client has to accept
the software vendor's word that the latest version of their software
is "much more reliable", and wouldn't cause the firewall to fall
over. This particular company had _no choice_, they had to accept
these terms, regardless of the dire straits they found themselves
in. Sure, legal action could be possible, but they need help _now_,
and any such action would take months or years. Besides, we all know
how EULAs are written - certainly suing for defective software is
not an option. Now, why is it significant that this is all
proprietary software? Because if it were truly FOSS, the client
would have several options (outside of an upgrade):

* They could fix the bug themselves, assuming they had skilled
  people on staff.

* They could hire someone to fix the bug, and do so in a
cost-effective manner, since everyone has access to the application
source, and presumably more than one vendor could provide a fix.
(Note that this applies to using software provided by large vendors
of FOSS, like Red Hat, even though they might follow similar
"end-of-life" practices. The source code is still free to obtain,
modify and distribute).

The second point is important, as the availability of source code,
along with the rights to change and distribute it (for money, or
not), actually results in competition. In this scenario, the vendors
providing the best support make the most money. Sounds pretty
sensible, right?

In today's current software market, software vendors who write and
sell proprietary software hinge their entire business model on the
software itself, the marketing hype that surrounds it, and the
assumption that people will pay to upgrade when the time
comes. Support is always an afterthought, a "necessary evil", and
getting good support for such software is almost impossible. There
is absolutely no incentive to provide good support once the software
is sold and the customer is locked-in to one vendor for support.

It is important to note at this point that I'm referring to_
software_ support, which is different from _product_ support. The
latter can be quite competitive, unlike the former. As an example,
plenty of people provide Check Point product support, some better
than others, but only one company can provide Check Point software
support (i.e. bug-fixes).

## Comments

**[Kyle Gonzales](#134 "2007-07-11 23:05:00"):** Spot on. This is an
excellent commentary. I see this all the time, and becomes even more
problematic in situations were people are using multiple proprietary
binaries (say, a kernel driver and a database) who have differing
kernel requirements, and neither vendor will budge. One such way to
work around the problem is to get rid of one of the proprietary
binaries. One way to ensure they don't come up again is to get rid
of ALL the proprietary binaries. Not realistic in many enterprise
settings, but the closer you get to pure open source, the fewer
interoperability issues you face.