Embedded Software Risk Areas -- Architecture

Series Intro: this is one of a series of posts summarizing the different red flag areas I've encountered in more than a decade of doing design reviews of industry embedded system software projects. You can read more about the study here. If one of these bullets applies to your project, you should consider whether that presents undue risk to project success (whether it does or not depends upon your specific project and goals). The results of this study inspired the chapters in my book.

Here are the Architecture red flags:

• No defined software architecture

There is no picture showing the system’s software architecture. (Many such pictures might be useful depending upon the context – but often there is no picture at all.) Ill defined architectures often lead to poor designs and poor quality code.

• No message dictionary for embedded network

There is no list of the messages, payloads, timing, and other aspects of messages being sent on an embedded real time network such as CAN. As a result, there is no basis for analysis of real time network performance and optimization of message traffic.

• Poor modularity of code

The design has poorly chosen interfaces and poorly decomposed functionality, resulting in high coupling, poor cohesion, and overly long modules. In particular, interrupt service routines are often too big and mask interrupts for too long. The result is often increased risk of software defects due to increased complexity.

Embedded Software Risk Areas -- Design

Series Intro: this is one of a series of posts summarizing the different red flag areas I've encountered in more than a decade of doing design reviews of industry embedded system software projects. You can read more about the study here. If one of these bullets applies to your project, you should consider whether that presents undue risk to project success (whether it does or not depends upon your specific project and goals). The results of this study inspired the chapters in my book.

Here are the Design red flags:

• Design is skipped or is created after code is written

Developers create the design (usually in their heads) as they are writing the code instead of designing each module before that module is implemented. The design might be written down after code is written, but usually there is no written design. As a result, the structure of the implementation is messier than it ought to be.

• Flowcharts are used when statecharts would be more appropriate

Flowcharts are used to represent designs for functions that are inherently state-based or modal and would be better represented using a state machine design abstraction. Associated code usually has deeply nested, repetitive “if” condition clauses to determine what state the system is in, rather than having an explicit state variable used to control a case statement structure in the implementation. The result is code that is significantly more bug prone code and difficult to understand than code based on a state-machine based design.

• No real time schedule analysis

There is no methodical approach to real time scheduling. Typically an ad hoc approach to real time scheduling is used, frequently featuring conditional execution of some tasks depending upon system load. Testing rather than an analytic approach is used to ensure real time deadlines will be met. Often there is no sure way to know if worst case timing has been experienced during such testing, and there is risk that deadlines will be missed during system operation.

• No methodical approach to user interface design

The user interface does not follow established principles (e.g., [5]), making use of the product difficult or error-prone. The interface might not take into account the needs of users in different demographic groups (e.g., users who are colorblind, hearing impaired, or who have trouble with fine motor control).

Embedded Project Risk Areas -- Development Process Part 2

Series Intro: this is one of a series of posts summarizing the different red flag areas I've encountered in more than a decade of doing design reviews of industry embedded system software projects. You can read more about the study here. If one of these bullets applies to your project, you should consider whether that presents undue risk to project success (whether it does or not depends upon your specific project and goals). The results of this study inspired the chapters in my book.

Here are some of the Development Process red flags (part 2 of 2):

• High requirements churn

Functionality required of the product changes so fast the software developers can’t keep up. This is likely to lead to missed deadlines and can result in developer burnout.

• No SQA function

Nobody is formally assigned to perform an SQA function, so there is a risk that processes (however light or heavy they might be) aren’t being followed effectively regardless of the good intentions of the development team. Software Quality Assurance (SQA) is, in essence, ensuring that the developers are following the development process they are supposed to be following. If SQA is ineffective, it is possible (and in my experience likely) that some time spent on testing, design reviews, and other techniques to improve quality is also ineffective.

• No mechanism to capture technical and non-technical project lessons learned

There is no methodical effort to identify technical, process, and management problems encountered during the course of the project so that the causes of these problems can be corrected. As a result, mistakes are repeated in future projects.

Embedded Project Risk Areas -- Development Process Part 1

Series Intro: this is one of a series of posts summarizing the different red flag areas I've encountered in more than a decade of doing design reviews of industry embedded system software projects. You can read more about the study here. If one of these bullets applies to your project, you should consider whether that presents undue risk to project success (whether it does or not depends upon your specific project and goals). The results of this study inspired the chapters in my book.

Here are some of the Development Process red flags (part 1 of 2):

•  Informal development process

The process used to create embedded software is ad hoc, and not written down. The steps vary from project to project and developer to developer. This can result in uneven quality.

• Not enough paper

Too few steps of development result in a paper trail. For example, test results may not be written down. Among other things, this can require re-doing tasks such as testing to make sure they were fully and correctly performed.

• No written requirements

Software requirements are not written down or are too informal. They may only address changes for a new product version without any written document stating old version requirements. This can lead to misunderstandings about intended product functions and difficulty in designing adequate tests.

• Requirements with poor measurability

Software requirements can’t be tested due to missing or subjective measurement criteria. As a result, it is difficult to know whether a requirement such as “product shall be user friendly” has been met.

• Requirements omit extra-functional aspects

Product requirements may state hardware processing speed and hardware reliability, but omit software response times, software reliability, and other non-functional requirements. Implementing and testing these undefined aspects is left at the discretion of developers and might not meet market needs.

Embedded Software Costs $15-$40 per line of code

Have you ever wondered how much software costs?  Most techies vastly underestimate the cost of the software they are writing. Partly this is because engineers and especially software developers are an optimistic lot. Partly it is because our intuition for such things was formed back in the days when we hacked out 1000 lines of code in an all-night session. (I wouldn't trust my life to that code!  But that is what our brains remember as being how long it takes to write that much code.)

I've had the opportunity to see how much it really costs embedded system companies to produce code.

The answer is $15 to $40 per line of code.  At the $40 end you can get relatively robust, well designed code suitable for industry applications. The $15 end tends to be code with skimpy design packages and skimpy testing. (In other words, some people spend only $15/line, but their code is of doubtful quality.) Use of all-US developers tends to be more expensive than joint development with both US and overseas team members, but the difference is less than you might have imagined.

The methodology used to get those numbers is simply divide COST by code SIZE. Ask a project manager how much a project cost (they almost always know this) and how many lines of code were in the project.  Divide, and you have your answer.

COST (in dollars) includes:
  - All developer labor plus overhead, fringe benefits, etc.
  - Test, SQA, technical writers, etc.
  - Expenses (e.g., tools), capital equipment if not in overhead, etc.
  - First-level managers for the above
  - All activities, including meetings, design reviews, acceptance test, and holiday parties
  - Does NOT include support after initial release (which can be huge, but I just don't have data on it)

In other words, you have to account for the whole cost of developing, not just the part where you actually write the code.

SIZE (in source lines of code SLOC) includes:
  - All the non-comment lines of code written or substatially rewritten for the project

This metric is far from perfect. For example, tracking developers split among multiple projects gets tricky. So does accounting for reuse of big chunks of code. But if you get a manager to give a reasonable estimate following the above guidelines, the answer always seems to come in at the $15-$40 range for industrial and transportation embedded software.

Sure, you can spend as much or as little on software as you like. But it is hard to find an organization who can field embedded software and have a viable product at less than $15/line. And it is usually just regulated, safety critical applications in which it is worthwhile to spend more than about $40/line. When all is said and done, the results I've seen make sense, and are generally in line with what few other data points I've heard on this topic.

UPDATE, October 2015.   It's probably more like $25-$50 per line of code now.  Costs for projects outsourced to Asia have done up dramatically as wages and competition for scarce coders have increased.

Testing a Watchdog Timer

I recently got a query about how to test a watchdog timer. This is a special case of the more general question of how you test for a fault that is never supposed to happen. This is a tricky topic in general, but the simple answer (if you can figure out how to do it easily) is to insert the fault intentionally into your system and see what happens.

Here are some ideas that I've seen or thought up that you may find helpful:

(1) Set the watchdog timer period to shorter and shorter periods until it trips in normal operation. This will give you an idea how close to the edge you are. But, it is easy to make a mistake changing the watchdog period back to the normal value. So it isn't a good test for nearly-final code.

(2) Add a timeout loop (for example, a do-nothing loop that you have made sure won't be removed by your optimizer). Increase the timeout value until the watchdog timer trips. This similarly gives you an idea how close to the edge you are in terms of timing, but with a nicer level of detail. It also has the advantage of testing operation without modifying the watchdog code itself.

(3) Use a jumper that, when inserted, activates a time-wasting task (similar to idea #2). The idea is when a jumper is installed it enables the running of a task that wastes so much time it is guaranteed to trip the watchdog. When the jumper is removed, that task doesn't run and the system operates properly.  You can insert the jumper during system test to make sure the watchdog function works properly.  (So just jumping to the watchdog handling code isn't the idea -- you have to simulate a situation of CPU overload for this to be a realistic test.)  When you ship the system, you make sure the jumper has been removed. Just to avoid problems, put the watchdog test first on the outgoing test plan instead of last. That way the jumper is sure to be removed before shipment. If you want to be really clever, that same jumper hardware could be used to disable the watchdog in early testing, and code could be changed to have it trip the watchdog as the system nears completion.  But whether you want to be that tricky is a matter of taste and the type of system you are building.

Using Car Remote Entry Key Fobs For Payments?

There is a news story that suggests people will be using the remote keyless entry systems for their cars as payment systems.  (So now your car key transmitter might have "lock", "unlock", and "pay" buttons.)  Do you think this is a good idea?

I did some design work on a previous generation of this technology. Cryptographic algorithms I developed jointly with Alan Finn were in a lot of model year 1994-2004 cars. What struck me about this market was the extreme cost sensitivity of things. No way could we afford industrial strength crypto.  The competition all had laughable crypto or made what I consider rookie mistakes (for example, using the same manufacturer key in all devices).  I never heard that my algorithm was broken during its designed 10-year life (but I'll bet the NSA had a nice chuckle when they saw it).  But other algorithms have proved to be insecure. For example, the Keeloq system has become the target of numerous attacks.  While the first published attacks are relatively recent, people likely knew about the weaknesses and could have attacked them a lot earlier if they had wanted to do so. While technology has changed over the years, these important lessons probably are the same:

• You have to have real security experts work on these products. You can't just put something together without really knowing your stuff or you will make rookie mistakes.
• You have to have some appreciation for security by the customers buying the systems (often it is a car manufacturer deciding how good is good enough). Too often decisions are made on "cheap and not obviously bad" rather than "is actually secure." One of the best things that happened to me in my experience was that the customer had someone (his name is Tom) who understood crypto and was willing to back us when I said we couldn't go below a certain cost threshold without compromising security beyond the required level.
• You have to use real crypto now, not cheesy crypto. Attackers have gained in sophistication and if someone has time to attack an old Keeloq system they'll attack your system.
• Once you let a key fob control money, it's a lot more attractive to attack. So it is sure to be attacked.
• Cars have a 10-15 year life, multi-year production runs, and easily a 3-5 year lead time. So you have to plan for your approach to be secure 20-25 years from now. That's hard to do in any system, much less one that is supposed to be inexpensive.

I predict next generation devices will have vulnerabilities due to failure to appreciate the above bullets. It has happened in the past, and it will happen in the future. I don't plan to use my key fob transmitter to make payments any time soon, and I used to design these things for a living. But then again, as long as you read about problems in the news before you're the victim of an attack, maybe you'll be OK.