Embedded Software Risk Areas -- Architecture

Series Intro: this is one of a series of posts summarizing the different red flag areas I've encountered in more than a decade of doing design reviews of industry embedded system software projects. You can read more about the study here. If one of these bullets applies to your project, you should consider whether that presents undue risk to project success (whether it does or not depends upon your specific project and goals). The results of this study inspired the chapters in my book.

Here are the Architecture red flags:

• No defined software architecture

There is no picture showing the system’s software architecture. (Many such pictures might be useful depending upon the context – but often there is no picture at all.) Ill defined architectures often lead to poor designs and poor quality code.

• No message dictionary for embedded network

There is no list of the messages, payloads, timing, and other aspects of messages being sent on an embedded real time network such as CAN. As a result, there is no basis for analysis of real time network performance and optimization of message traffic.

• Poor modularity of code

The design has poorly chosen interfaces and poorly decomposed functionality, resulting in high coupling, poor cohesion, and overly long modules. In particular, interrupt service routines are often too big and mask interrupts for too long. The result is often increased risk of software defects due to increased complexity.

Embedded Software Risk Areas -- Design

Series Intro: this is one of a series of posts summarizing the different red flag areas I've encountered in more than a decade of doing design reviews of industry embedded system software projects. You can read more about the study here. If one of these bullets applies to your project, you should consider whether that presents undue risk to project success (whether it does or not depends upon your specific project and goals). The results of this study inspired the chapters in my book.

Here are the Design red flags:

• Design is skipped or is created after code is written

Developers create the design (usually in their heads) as they are writing the code instead of designing each module before that module is implemented. The design might be written down after code is written, but usually there is no written design. As a result, the structure of the implementation is messier than it ought to be.

• Flowcharts are used when statecharts would be more appropriate

Flowcharts are used to represent designs for functions that are inherently state-based or modal and would be better represented using a state machine design abstraction. Associated code usually has deeply nested, repetitive “if” condition clauses to determine what state the system is in, rather than having an explicit state variable used to control a case statement structure in the implementation. The result is code that is significantly more bug prone code and difficult to understand than code based on a state-machine based design.

• No real time schedule analysis

There is no methodical approach to real time scheduling. Typically an ad hoc approach to real time scheduling is used, frequently featuring conditional execution of some tasks depending upon system load. Testing rather than an analytic approach is used to ensure real time deadlines will be met. Often there is no sure way to know if worst case timing has been experienced during such testing, and there is risk that deadlines will be missed during system operation.

• No methodical approach to user interface design

The user interface does not follow established principles (e.g., [5]), making use of the product difficult or error-prone. The interface might not take into account the needs of users in different demographic groups (e.g., users who are colorblind, hearing impaired, or who have trouble with fine motor control).

Embedded Project Risk Areas -- Development Process Part 2

Series Intro: this is one of a series of posts summarizing the different red flag areas I've encountered in more than a decade of doing design reviews of industry embedded system software projects. You can read more about the study here. If one of these bullets applies to your project, you should consider whether that presents undue risk to project success (whether it does or not depends upon your specific project and goals). The results of this study inspired the chapters in my book.

Here are some of the Development Process red flags (part 2 of 2):

• High requirements churn

Functionality required of the product changes so fast the software developers can’t keep up. This is likely to lead to missed deadlines and can result in developer burnout.

• No SQA function

Nobody is formally assigned to perform an SQA function, so there is a risk that processes (however light or heavy they might be) aren’t being followed effectively regardless of the good intentions of the development team. Software Quality Assurance (SQA) is, in essence, ensuring that the developers are following the development process they are supposed to be following. If SQA is ineffective, it is possible (and in my experience likely) that some time spent on testing, design reviews, and other techniques to improve quality is also ineffective.

• No mechanism to capture technical and non-technical project lessons learned

There is no methodical effort to identify technical, process, and management problems encountered during the course of the project so that the causes of these problems can be corrected. As a result, mistakes are repeated in future projects.

Embedded Project Risk Areas -- Development Process Part 1

Series Intro: this is one of a series of posts summarizing the different red flag areas I've encountered in more than a decade of doing design reviews of industry embedded system software projects. You can read more about the study here. If one of these bullets applies to your project, you should consider whether that presents undue risk to project success (whether it does or not depends upon your specific project and goals). The results of this study inspired the chapters in my book.

Here are some of the Development Process red flags (part 1 of 2):

•  Informal development process

The process used to create embedded software is ad hoc, and not written down. The steps vary from project to project and developer to developer. This can result in uneven quality.

• Not enough paper

Too few steps of development result in a paper trail. For example, test results may not be written down. Among other things, this can require re-doing tasks such as testing to make sure they were fully and correctly performed.

• No written requirements

Software requirements are not written down or are too informal. They may only address changes for a new product version without any written document stating old version requirements. This can lead to misunderstandings about intended product functions and difficulty in designing adequate tests.

• Requirements with poor measurability

Software requirements can’t be tested due to missing or subjective measurement criteria. As a result, it is difficult to know whether a requirement such as “product shall be user friendly” has been met.

• Requirements omit extra-functional aspects

Product requirements may state hardware processing speed and hardware reliability, but omit software response times, software reliability, and other non-functional requirements. Implementing and testing these undefined aspects is left at the discretion of developers and might not meet market needs.

Embedded Software Costs $15-$40 per line of code

Have you ever wondered how much software costs?  Most techies vastly underestimate the cost of the software they are writing. Partly this is because engineers and especially software developers are an optimistic lot. Partly it is because our intuition for such things was formed back in the days when we hacked out 1000 lines of code in an all-night session. (I wouldn't trust my life to that code!  But that is what our brains remember as being how long it takes to write that much code.)

I've had the opportunity to see how much it really costs embedded system companies to produce code.

The answer is $15 to $40 per line of code.  At the $40 end you can get relatively robust, well designed code suitable for industry applications. The $15 end tends to be code with skimpy design packages and skimpy testing. (In other words, some people spend only $15/line, but their code is of doubtful quality.) Use of all-US developers tends to be more expensive than joint development with both US and overseas team members, but the difference is less than you might have imagined.

The methodology used to get those numbers is simply divide COST by code SIZE. Ask a project manager how much a project cost (they almost always know this) and how many lines of code were in the project.  Divide, and you have your answer.

COST (in dollars) includes:
  - All developer labor plus overhead, fringe benefits, etc.
  - Test, SQA, technical writers, etc.
  - Expenses (e.g., tools), capital equipment if not in overhead, etc.
  - First-level managers for the above
  - All activities, including meetings, design reviews, acceptance test, and holiday parties
  - Does NOT include support after initial release (which can be huge, but I just don't have data on it)

In other words, you have to account for the whole cost of developing, not just the part where you actually write the code.

SIZE (in source lines of code SLOC) includes:
  - All the non-comment lines of code written or substatially rewritten for the project

This metric is far from perfect. For example, tracking developers split among multiple projects gets tricky. So does accounting for reuse of big chunks of code. But if you get a manager to give a reasonable estimate following the above guidelines, the answer always seems to come in at the $15-$40 range for industrial and transportation embedded software.

Sure, you can spend as much or as little on software as you like. But it is hard to find an organization who can field embedded software and have a viable product at less than $15/line. And it is usually just regulated, safety critical applications in which it is worthwhile to spend more than about $40/line. When all is said and done, the results I've seen make sense, and are generally in line with what few other data points I've heard on this topic.

UPDATE, October 2015.   It's probably more like $25-$50 per line of code now.  Costs for projects outsourced to Asia have done up dramatically as wages and competition for scarce coders have increased.