Nobody Writes Perfect Software -- get over it!

Have you ever written perfect software?  Really?  (And if you did, how exactly do you know that?)  If you're imperfect like the rest of us, how do you take that into account in your software architecture?

As far as I can tell nobody knows how to write perfect software. Nearly perfect is as good as it gets, and that comes at exponentially increasing costs as you approach perfection. While imperfect may be good enough for many cases, the bigger issue is that we all seem to act as if our software is perfect, even when it's not.

I first ran into this issue when I was doing software robustness testing (the Ballista project -- many moons ago). A short version of some of the conversations we'd have went like this.  Me: "If someone passes a null pointer into your routine, things will crash." Most people responded: "well they shouldn't do that" or "passing a null pointer is a bug, and should be fixed."  Or "nobody makes that kind of stupid mistake" (really??).  We even found one-liner programs that provoked kernel panics in commercial desktop operating systems.  Some folks just didn't care. But the folks who concentrated on highly available systems said: "thanks -- we're going to fix everything you find." Because they know that problems happen all the time, and the only way to improve dependability in the presence of buggy software is to make things resilient to bugs.

Now ask yourself about your embedded system. You know there are software bugs in there somewhere. Do you pin all your hopes on debugging finding every last bug?  (Good luck with that.) Or do you plan for the reality that software is imperfect and act accordingly to increase your product's resilience?

Here are some of the techniques that can help.

• Watchdog timer in case your system wedges (is it turned on? is it kicked properly?)
• Input parameter sanity checks (check for null pointers, values out of range, other problems)
• Defaults on switch statements that invoke an error handler (what if you forgot a case and it maps into whatever case you picked as the default?)
• Run-time assertions (the value if "i" should be positive -- oops it's negative right now)
• Error return codes (what happens if the subroutine call didn't work?)
• Robustness testing (some folks call this fuzz testing although this just one approach -- toss bad values at your software and see if things fall apart)
• Error logging (so you can track down problems in units returned for service)

Many of them you may have heard of, and you probably have heard the umbrella term "defensive coding" somewhere. Do you have any favorite techniques I've missed?

But the real question is do you actually use them?  Or is your software so perfect you needn't bother?

ESC SV Conference Handouts

Thanks to everyone who attended my talk at ESC SV!  There were about 80 attendees, and I especially appreciated the many questions and insightful comments from the audience.

The organizers ran out of handouts, so I'm posting the handout Acrobat file so everyone can get a copy. This copy has been updated to include some last-minute changes I made right before the talk. It summarizes a lot of the areas covered by my book:

Avoiding the Top 43 Embedded Software Risks (handouts), Phil Koopman, ESC SV, May 2011
http://www.ece.cmu.edu/~koopman/pubs/koopman11_escsv_handouts.pdf     (5.7 MB, Acrobat format)

You can also see the accompanying paper at this link.
Enjoy!

Book availability on Amazon

I've received a couple notes saying that my book is out of stock on Amazon. What's really going on is that Amazon itself doesn't sell the book, but rather the publisher sells it as Fulfilled by Amazon (FBA).  So what you are looking for is the "Available from these sellers" link and order from Geos Fulfillment, which is then shipped direct from Amazon's warehouse.

Article on Software Failures

Jack Ganssle has a nice design article that details a long history of design errors, from bridge building to safety critical software problems. Much of it is about NASA mission failures, but he also checks in on the topics of radiation therapy, pacemakers, and nuclear experiments.  Good one-stop shopping for horror stories and a discussion of high level patterns behind these sorts of problems.


On discussing the Therac 25:
"The FDA found the usual four horsemen of the software apocalypse at fault: inadequate testing, poor requirements, no code inspections, and no use of a defined software process."


Quote of the day from the article:
"Globals are responsible for all of the evil in the universe, from male pattern baldness to ozone depletion."

Source: Mars Ate My Spacecraft

Peer Reviews are Cheap

In previous posts I've talked about how useful peer reviews can be. But some many folks say they are just too expensive. That just isn't so.  Here's the math.

You can write good embedded software at the rate of 1-2 lines of code per hour. (That is all source lines of code divided by all hours for a project, including requirements, test, etc.   Really, that is the number.)

Formal, rigorous, thorough peer reviews target 100-200 lines of code reviewed per hour (we're talking Fagan style inspections, which are quite formal, but still the most cost effective review method from all the data I've seen). Let's say you have 4 people in an inspection. That is 25-50 lines of code per person-hour.  If you include an hour of prep time for each person for a two-hour review, that is still 17-33 lines of code per person-hour.

Think you write code faster?  OK, let's say you're Agile and do 3 lines of code/hr (what I've seen in industry, and comes with increased risk we can discuss elsewhere).  That's still nowhere near the review productivity rate.

We're talking maybe 5%-10% of project effort to do heavy-weight reviews.  And it generally finds half the bugs. More importantly, it finds them early, so you don't have a lot of rework.

So why does everyone say it is too expensive?

Good Article on User Interface Design Rules

I found a well-written article about user interface design rules. "Understanding user interface design rules," by Jeff Johnson, EE Times. (Two part article links:  Part 1  | Part 2 )

The take-aways are:

• Make the operation task-focused, simple, and consistent. Bring the operation itself as close to possible as what the user wants to accomplish (the outcome), not what the device knows how to do (the mechanisms available).
• Keep things simplicity and consistent.
• Make the terminology familiar (and task-focused, and simple). Use the meaning that people expect to see, not jargon. Use different terms whenever there is a concept that differs in an important way. Don't use different terms if the difference doesn't matter to the user.
• Make mistakes low cost (e.g., provide undo) so people aren't afraid to explore the interface.

The concept of an objects/actions matrix looks interesting. The idea is you put objects as the rows of a matrix and actions on those objects as a column.  Whenever an action is permissible for an object it gets a check mark in the matrix.  Good designs have square, densely filled in matrices (simple, consistent). Bad designs have large, sparsely filled in matrices (complex, inconsistent).

This article is well worth the time for anyone who designs embedded systems.

Using a Risk Analysis Table to Categorize Bug Priority

Perhaps the single most difficult thing to get right in a bug report is assigning a priority. It is tempting to assign a priority based on how spectacular the result is. But you also need to take into account how likely the bug is to manifest in a deployed system. For example, a bug that causes a system to crash and automatically reboot may be a lot more dramatic than a confusing screen message, but if that confusing screen message results in thousands of tech support calls, it could be a disaster for your company.  The best approach is one that combines severity and probability.

It is tempting to try to use fancy math to combine severity and probability. Usually that doesn't work out so well in practice. Instead, I recommend borrowing a technique from the safety critical system community. They use a Risk Table to assign a "criticality" to a particular adverse event as part of their Preliminary Hazard Analysis (PHA). You can use the same table in a different way and just say you are assigning bug "priority" instead of "risk."  Below is an example Risk Table:

Probability is your best estimate as to how often the bug will be seen in use. Consequence is how big a problem it will cause. The Risk (indicated by each box in the grid) is how big the risk is to product reputation -- which ought to be the same as the bug priority.  It helps to have clearly defined statements to guide assigning any particular bug to a row and to a column. Once you assign probability and consequence, the table tells you the priority of that particular bug.


You can modify this table to have 3 to 6 rows and 3 to 6 columns depending upon your needs (the table can be a rectangle rather than a square).  You can also modify the asymmetry of assigning risks as has been done in this example (consequence is weighted a bit higher than probability for this table by putting extra "Very High" boxes on the top row and so on). The point is not the table itself, but rather that binning things in this way makes assigning bug priority a lot easier for people to do in practice.