A Review of ``The Day the Phones Stopped'' by Leonard Lee

At 14:25, 1990-01-15, exactly thirty-four years ago as of today, the AT&T national telephone network
started failing to route roughly half of all calls correctly for a little over nine hours.  The book
under review describes this failure in some detail and, to my surprise and delight, this book mostly
details many other failures caused by programming errors the author found to be worthy of inclusion.
Amusingly and horrifyingly, this book tells of the ongoing software crisis, that is software growing
beyond man's ability to tame it, in 1991.  In some ways, it's funny to see how things have worsened.

Poorly, this is yet another book which I'd to read on loan from my local library, as I couldn't find
it any other way.  I can easily read any old new garbage, but a nice book like this is hard to read.

This book was written at a time when computer terms had to be explained to the reader, for computers
weren't yet widespread and well-understood; newer books are poorer for the lack of such explanation.

The first chapter, ``DANGEROUS MEDICINE'', starts with something more severe than telephone failure,
by explaining the Therac-25, a controlled dosage radiation therapy machine well-known for killing or
maiming several unfortunate patients.  I've long known of this deadly machine, and its fatal flaw in
hardware misconfiguration, but only learned the details in this chapter; where I'd imagined an array
of mirrors or whatnot, I instead found a simple tungsten rod not in-place.  The machine had two ways
to deliver radiation: by a low-powered electron beam, or a high-powered X-ray striking the tungsten.
The chapter elaborates on how the flaw was repeatedly uncovered in operation and explained away, for
the technicians insisted the computer makes no mistakes; after each maiming, the units were taken in
for analysis, to find no errors.  The flaw was what is known as a race condition, due to the removed
hardware mechanisms from previous models.  Further, the emergency shutdown mechanism was too slow at
three tenths of a second.  After the flaw was known, the Therac-25 software was rewritten and a one-
pulse shutdown mechanism, working at one three hundredth of a second, was installed onto all models.
The chapter gives some room to other faulty medical devices too, such as pacemakers that don't work.

The second chapter, ``THE UNFRIENDLY SKIES'', details airplanes and businesses operating airlines to
some degree.  The attention airplanes received in this book is outsized compared to all other topics
covered and very conspicuously so.  For some reason, this chapter is easily the longest in the book,
at twice or thrice the length of most others, and more than four times larger than the smallest; two
later chapters cover very similar or related topics even further; this chapter should've been split.
It begins with an amusing remark that not all software errors work against the common man, using the
example of airline rewards being improperly credited, but honoured nonetheless; follows is a case in
which rewards weren't being properly credited, but this was also corrected.  Following yet is a case
of an automatic baggage handling system that couldn't properly track luggage when deployed; the book
remarks that the software's testing environment was unrealistic, a common theme with later chapters.

In July 1983, Air Canada Flight 143 ended abruptly after the plane exhausted its fuel supply in mid-
flight; the fuel reporting subsystem could handle total failure of one of its two data channels, but
not partial failure, and so failed to work; unit confusion later led to the plane being underfueled.
When fuel ran out, the emergency battery systems allowed the fully-computerized cockpit to work, and
an air turbine provided more power.  The plane, without full power, provided a worse experience than
older planes lacking computers entirely.  That situation was worsened by newer radar systems failing
to recognize the plane, because it no longer sent out coded signals with a transponder; fortunately,
the particular nearby air control tower had an old system that would recognize anything.  The flight
ended without death with the plane landing as safely as it could on a racetrack around many viewers.

Interestingly, shortly before starting the book I read this alternative report of the same incident:
https://www.damninteresting.com/the-gimli-glider

The chapter goes on to describe the Airbus A320, and taught me the meaning of fly-by-wire.  On 1988-
06-26, an A320 crashed at a French air show, and was deemed pilot error.  On 1990-02-14, Valentine's
Day, an Indian flight crashed at an airport, and India was just as incompetent then as now, with the
emergency response both pathetic and entirely unplanned, thwarted by things like a locked gate; this
quote from the book properly captures the sorry state thereof, and the situation is no better today:

 ``One rescue truck tragicomically attempted to spray foam from the other side of the fence, but
   could barely reach any of the wreckage.''

The chapter contains warning about computers causing recklessness, such as with the French incident,
by making pilots believe themselves to have more leeway than is present; I found some concerns to be
similar to those for the current self-driving car nonsense.  Another concern is planes programmed to
disobey pilots when a maneuvre be deemed dangerous, but such things can be necessary in emergencies.

Military failures are also noted, like a Swedish crash caused by delayed controls.  On 1985-03-13, a
Blackhawk helicopter crashed.  In April 1987, a similar failure occurred.  It was found to be caused
by radio tower interference commanding the helicopter's fly-by-wire systems.  The given solution was
in the form of special maps which show strong sources of electromagnetic interference to be avoided.

On 1983-04-31, Korean Airlines Flight 007 flew into protected airspace and was shot down by a Soviet
fighter jet.  The Inertial Navigation System (INS) tracks the position of a plane by movement deltas
from an origin, and may have been slightly misconfigured.  It's reportedly very easy to misconfigure
and ignore any warnings.  On 1987-07-08, an INS error was noted to have, nearly, caused a collision.

The second chapter ends with notes on autopilot errors and grieving over how pilots were once taught
about the entire plane and its failure modes, but no longer.  It goes on to explain how they perhaps
could be taught about the plane at a higher level, despite that a total overview is now impractical.

The third chapter, ``THE BIGGER THEY ARE...'', finally explains that titular accident causing AT&T's
telephone network failure that lasted nine hours.  Unlike in the first chapter, I learned more about
the technical nature of this failure from elsewhere, and following is a pleasant article about that:
http://users.csc.calpoly.edu/~jdalbey/SWE/Papers/att_collapse

The chapter begins by describing AT&T's Network Operations Center (NOC) and how it had displayed the
status of the 114 switching centers, so-called 4ESS units, across the United States; the display was
updated every five minutes.  At 14:25, 1990-01-15, the trouble began in New York and spread out from
there.  The chapter mostly details the affected businesses, such as telemarketing, and also contains
the first mention of the Sabre airline reservation system.  The New York Stock Exchange was designed
in a way leaving it unaffected, and other long-distance telephone systems still worked without flaw.

The flaw, introduced in December 1989, was triggered by a very rare set of circumstances.  The self-
healing network hadn't been designed to cope with all nodes failing by exactly the same way and time
like this.  AT&T employees after getting more specific information, so-called ``machine-discretes'',
revealed the error to be in Signalling System 7 (common channel signalling) nodes, but not the older
Signalling System 6.  Signalling System 7 was out-of-band, faster, and capable of more features, but
also more complex.  AT&T maintained a model switching network in a laboratory for diagnosing failure
of this kind, which is how the flaw was eventually found.  That first patch prepared for the problem
failed to work, and the second patch worked by preventing a signal from being sent; it took days for
a final solution patch to be made.  Ironically, news reporting on the event was difficult due to the
communications issues for CBS News in New York.  Similar telephone network issues happened later on,
in Minneapolis and California.  Oddly, I believe the C language wasn't mentioned at all in the book,
despite being responsible for many of its failures, including this AT&T failure.  One AT&T executive
had this to say, and it may have been included to reinforce the false notion about software quality:

 ``Our goal is to achieve no errors certainly, but we're also not going to delude anyone by saying
   it'll ever be one hundred percent perfect.''

The chapter also discusses other minor failures, such as a flaw in a corporate mail system, to which
one executive commented so, amusingly: ``How many computer errors are they going to get away with?''

The fourth chapter serves as a lamentation on software development more than covering any particular
problems, although it also features examples of misguided space probes and stock exchange downtimes.

Many times in this book is repeated a saying, a factoid, which I know to be false and which follows:

 ``In truth, software can never be made completely reliable and trustworthy.''

The book continually remarks on how no amount of testing will suffice, which is true, but also tells
a persistent lie repeatedly, that all software must have flaws.  The fourth chapter even begins with
``The bottom line is that all software contains errors.'' and later in the next paragraph ``It can't
be done.  Software can never be made completely error-free.'' and how false.  Software can be tamed.

The fourth chapter also features a cameo from Frederick Brooks, who tells of how software may be the
most complex constructs by size ever created by man.  Of perhaps basic interest is the mentioning of
European Computer-Aided Software Engineering (CASE), but I chose to look no further into it for now.

The fifth chapter, ``MONEY, MONEY, MONEY'', regards the woes of banking and other businesses, but is
mostly concerned with banking.  It covers how businesses waste money on systems that don't work, the
improperly sent checks and other problems such systems cause, amongst other things.  The Bank of New
York had unbalanced books and was forced to take a loan from the federal reserve, with all assets as
collateral, due to a programming error; something similar occurred with the Clearing House Automated
Payment System (CHAPS) in the UK, with money sent improperly but returned voluntarily.  Anyone who's
read about banking software woes has read such stories before.  This chapter also discusses problems
with domain mismatch from programmers who know not about the very domain for which such programs are
written, an example being a programmer writing banking software who knows none of the terms thereof.
The chapter also contains some interesting paragraphs on throwaway software, so-called ``cardboard''
programs, that avoid some of the outlined issues by being disposable as requirements change or fall.

I'd previously known not too much about the Sabre airline reservation system, and this chapter gives
details about the secure and well-supplied compound housing it, alongside some amusing software woes
causing failures, despite such extreme measures.  Perhaps Sabre should've been given more attention.

The sixth chapter, ``COMPUTERS AND UNCLE SAM'', is similar to the fifth, but about the government of
the United States.  It gives a very detailed description of the tax return procedure of the Internal
Revenue Service (IRS) and its dysfunctions.  Amusingly, a long text about the IRS wanting to upgrade
its aging infrastructure could be written just as easily today.  In November 1984 and anticipating a
great amount of work, its unfinished and upgraded system was used, to ghastly results.  This chapter
details great amounts of redundant work required of the employees in order to cope with flaws in the
software, and also things such as cases where disgruntled workers discarded tax returns in the trash
to avoid dealing with them.  This is yet another old work looking to the future with the hopes of an
electronic filing system to avoid such horrible drudgery; unfortunately, such hopes have been dashed
by parasitic businesses.  The book is one so old that the senator's full name is given, Albert Gore.

The seventh chapter, ``UP IN THE AIR'', is another aviation chapter, this time regarding air traffic
control systems.  It starts with the AUTOMATED RADAR TERMINAL SYSTEM (ARTS IIIA) and the story given
about it in this chapter is one that sounds believable to the ignorant, too silly to the novice, and
only all-too-believable to the wise.  The machine tracking the flight plans would become overloaded,
and crash for thirty seconds or so at a time, leaving the display useless.  A patch was made to cope
with a particularly large event in Dallas, but it left the display garbled and therefore unreliable;
air traffic controllers were forced to use their memory, a pen, and paper of all things to track the
flights, leading to near collisions even after new leaving flights were stopped.  While the incident
resulted in a new system, other places weren't so fortunate.  This chapter includes a list of flight
delays caused by computer problems.  Interestingly, this chapter notes the anticipated increases for
air traffic in the next millenium, and I wondered if they'd come true only to realize the attacks on
2001-09-11 probably put a damper on those increases, coincidentally.  Also noted are developments of
the FAA's National Airspace System (NAS) and, upon delays, its Interim Support Plan (ISP).  Finally,
the chapter gives information about a preventable plane crash involving one in bad visibility, which
got in the way of another taking off, leading to deaths, and an emergency escape system failed also.

The eighth chapter, ``THE GOD OF WAR'', covers the United States military and its software problems.
It begins with examples of expensive and redundant military installations that have issues such as a
radar system interfering with fly-by-wire controls.  A third of the chapter is dedicated to the B-1B
bomber project, then-new, and is an example of failures caused not just by software but hardware and
general design problems with its goals; an example of a failure was something like the radio jamming
ability jamming itself.  The chapter includes an interesting paragraph about the future of war, with
soldiers carrying handheld computers, wearing helmets with computer displays in the visors, and with
computer identification chips in their teeth; as far as I know, only the final prediction has yet to
come.  I was pleased to see Ada mentioned in this chapter with the usual story of over three hundred
languages being in-use and a replacement being sought, but, unfortunately, the book was written well
before Ada 1995 made the language much more pleasant to use, and so the mention isn't very positive;
instead, things like issues with the compilers available are noted, which made the software too slow
to use in certain systems for which it was intended.  Also listed is the ``Star Wars'' defense plan,
which would be the most complicated piece of software ever written, like an even bigger version of a
Patriot defense system.  The latter pages of the chapter cover military accounting issues, including
the 131 different and incompatible accounting programs in-use at the Air Force.  While reading about
it, I couldn't help but wonder how much of it were incompetence, and how much were cover for malice.

The ninth chapter, ``A CASE OF MISTAKEN IDENTITY'', has a title which would've been better suited to
the next chapter.  It tells the harrowing story of the Aegis, named after Zeus's armor, installed on
the USS Vincennes; the story slowly builds up over the chapter, starting with a ``fatal flaw'' later
deemed ``a seemingly trivial design decision'' but seemingly forgotten as the story continues.  As I
was reading, dread hit me when I realized what had been happening, and that design flaw came back in
full force to the fore of my mind.  The chapter gives an extremely-detailed telling of the events on
the ship, emphasized by an ominous foreshadowing with which some paragraphs begin, counting down the
minutes until the disaster had hit.  It was an excellent read which left me with the dissatisfaction
of a bad situation where there was arguably no one to blame.  A particularly good quote now follows:

 ``Incredibly, surrounded by $600 million worth of computers and electronics, Rogers has to order a
   crewman to start frantically thumbing through an ordinary paperback airline flight guide.''

This was a case in which the software worked correctly, but had a design that could've been improved
to avoid misuse and misinterpretation.  I found it very interesting to learn a tape containing a log
of everything done was both available and analyzed.  Crewman were found to have been panicking, such
as with pressing the wrong buttons five and twenty-three times before they corrected their mistakes.

The tenth chapter, ``OF CRIME AND PUNISHMENT'', details wrongful arrests caused by computer criminal
records, and poor recordkeeping practices.  It reads rather like a modern article decrying computers
used in criminal identifications and arrests, but the state-of-the-practice was much worse then than
nowadays.  There were many different databases across the country tracking incompatible traits, poor
logging on how they were accessed, and varying policy on how they could be cleared and updated.  One
police deputy spoke of how he ignores slight mismatches, for he expects the computers to be slightly
wrong.  The chapter goes on to explain the FBI's National Crime Information Center (NCIC) in detail,
and how it stores a great amount of incorrect, outdated, and incomplete information on its subjects.

A man demanded to have his entry updated after being held at gunpoint by police on several occasions
due to a case of mistaken identity.  All of this was worsened by such records having no photographs.
The story of a retarded man who languished in prison for forty days, because his name wasn't entered
correctly and thus couldn't be found by machine, was another example of such computer mismanagement.
I thought it very interesting, how Janlori Goldman of the ACLU recommended a new governmental office
to be established for data quality.  Lastly, the passages in this chapter on the dangers of improper
usage of all of this information for blackmail, employment checks, and whatnot feel quaint nowadays.

The eleventh chapter, ``THE HUMAN ELEMENT'', is regarding human interface design and the like.  This
chapter correctly points out many differences between a computer and a human brain, something people
still get wrong.  Humans are overwhelmed by the sheer amount of data available, microscopic compared
to the present, but machines can handle it with ease, albeit unintelligently.  It notes many studies
and research results that should be noted by interface designers, such as how the first suggestion a
machine makes will be valued over others.  Since machines are trusted to be correct, they must be if
they're to be of use.  The chapter laments that decision-making research has been made but not used.
It also contains remarks from an Apple Computers employee, and notes on RSI and posture when using a
computer.  Repeated too is the idea that people must know the discipline for which they program, and
programmers must also know how the programs will be used.  Thus, this chapter is unlike many others.

The twelfth and final chapter, ``NO MIRACLES'', contains a great chunk of information about remedies
to the problems explained throughout the book, and a solemn warning that software is vital to nation
and industry, which is already falling behind other countries on hardware.  I was disappointed about
some of the ignorance held in this chapter, as the author was apparently unaware of Edsger Dijkstra.

At one point in the chapter, a Dr. William Wulf makes the following claim both ridiculous and false:

 ``knowing how to do it is still a deep intellectual problem.  The proper
   foundation, even the appropriate mathematics, does not exist.''

I found it amusing that a programmer shortage was bemoaned even so long ago, the suggested solutions
similar to the present day.  Those solutions were using ``minorities'' in college programs, women in
particular, to offset the supposed shortage.  The chapter notes how programming has no education, no
standards, and no practices, in general, and discusses the notion of professional certifications for
software developers at some length, including how some programmers would rather move to other fields
than accept such a state of affairs.  AI research is also discussed a tad, in ways reflecting modern
concerns about the unpredictability thereof.  Those concerns of the US losing dominance in computing
are well-justified in hindsight, and that situation has only worsened.  The bills S.272 and H.R. 656
are the ``High-Performance Computing Act of 1991'' and mentioned in the final pages of the book, but
not decided at the time of publishing; sponsored by Albert Gore, it ultimately passed, and I suppose
improved the dire situation presented in the book, but I find it obvious that more needs to be done.

Following is a list of topics mentioned in the final chapter that may make for interesting research:
Japanese Daisys
Japanese program to translate telephone calls in real-time
Japanese Software Industrialized Generator and Maintenance Aids (SIGMA)
European European Strategic Program for Research and Development in Information Technology (ESPRIT)
European EUREKA
DARPA prototyping

The book was a wonderful read, albeit not without flaws, but my viewpoint is unconventional compared
to most other programmers.  I'm forced to wonder how he got so much detail for this book; it must've
required months of research, poring over newspapers, interviews, and official reports; I'm impressed
with it, and can easily and highly recommend others to read it; despite this, I'm still unaccustomed
to reading such a book with such detail, and it forces me to question it to some degree, but I don't
have particular issue believing what I read therein, despite issues I've noted throughout my review.

The book ends with Wulf's lamentation, true: ``We are not going to find a quick and easy solution.''