Tuesday, October 2, 2018

Cost of highly safety critical software

It's always interesting to see data on industry software costs. I recently came across a report on software costs for the aviation industry. The context was flight-critical radio communications, but the safety standards discussed were DO-178B and DO-254, which apply to flight controls as well.

Here's the most interesting picture from the report for my purposes:

(Source: Page 28 https://www.eurocontrol.int/sites/default/files/content/documents/communications/29012009-certification-cost-estimation-for-fci-platform.pdf.pdf )

Translating from DO-178B terminology, this means:

  • DAL A  (failure would be "catastrophic"):  3 - 12 SLOC/day
  • DAL B  (failure would be "hazardous"): 8 - 20 SLOC/day
  • DAL C (failure would be "major"): 15 - 40 SLOC/day
  • DAL D (failure would be "minor"): 25 - 64 SLOC/day
Worth noting is that, in my experience, really solid mission critical but NOT life-critical embedded software can be done at up to 16 SLOC per day for well-run experienced teams, so it tends to line up with DAL B costs.

For interpretation, "DAL" expresses a criticality level (a "Development Assurance Level"), with more critical software requiring more rigorous processes.  The document has quite a lot to say about how the engineering process works, and is worth a read if you want to see how the aviation folks do business.  (I'm aware that DO-178C is out, but this paper talks about the older "B" version.)    Note that there are other cost models in the paper that are less pessimistic in that report, but this is the one that says "industry experience."

Have you found other cost of software data for embedded or mission critical systems?

Monday, October 1, 2018

A Million Page Views

Blogger analytics say that I just hit 1 million page views.

For fun, the top 10 countries reading this blog in rank order are:
  • USA (about half the total views)
  • Germany (almost tied with India at more than 10%)
  • India (almost tied with Germany at more than 10%)
  • Russia
  • United Kingdom
  • Canada
  • France
  • Ukraine
  • Poland
  • Brazil
I look forward to continuing to post articles about embedded system software, as well as on my Safe Autonomy Blog

Thanks to all the readers who made this happen!

Tuesday, September 25, 2018

Potentially deadly automotive software defects

Here's a list of potentially deadly automotive software defects, mostly from NHTSA Recall notices.

There is still a lot of resistance to the idea that car software can have fatal defects that result in deaths not due to driver error. In fact such defects do exist, and for many of them we've just gotten lucky that few or no people have died as a result. Recently we've been seeing more deadly software defects being reported. This posting is intended to give a taste of what's been going on in automotive software quality. This is a very partial list of bad software that was deployed on production vehicles in the US.

This list includes a variety of subsystems including unintended acceleration, steering failures, brake assist failures, headlights going out while driving, and quite a lot of air bag failures. There are software defects, configuration management errors, leaving the module in "factory mode" when shipped, and even EEPROM wearout. Overall this paints a picture of an industry that is shipping a lot of safety critical software defects.  In fairness, yes, these are all ones that are being fixed, and there are certainly other causes of fatal accidents. (Presumably there are others not yet being fixed, if for no other reason than that the cars are still new on the road. But at least some of these recalls sure look like mistakes that simply should not be happening in life critical software.)

The list is almost certainly much, much longer, and I simply ran out of time trying to go through the full NHTSA database.  And even that doesn't include everything that happens. The list is heavy in 2013-2015 mostly because that was the most convenient source material I found. There is no reason whatsoever to believe things have gotten dramatically better since then.

The purpose of this list is not to call out any particular company or software defect. Rather, the point is that safety critical software defects are both pervasive and persistent across the automotive industry.  Yes, we can have discussions about how many vehicles vs. how many defects. But it still does not instill confidence about life critical software in a self-certifying industry that in the US is not required to follow international software safety standards.
  • "Automatic braking systems in some Nissan Rogues are going rogue, safety group says" / Mar 2019
  • "Alfa Romeo recalling 60,000 vehicles to repair cruise management fault" / Mar 2019
  • "Ford recalls 1.5 million Ford Focus cars that could stall with fuel tank problem" / Oct 2018
  • "Toyota recalls trucks, SUVs and cars to fix air bag problem" / Oct 2018
    • "Toyota says the air bag control computer can erroneously detect a fault when the vehicles are started. With a fault, the air bags may not deploy in a crash. The company wouldn't say if the problem has caused any injuries."
    • https://www.abc57.com/news/toyota-recalls-trucks-suvs-and-cars-to-fix-air-bag-problem
  • "Toyota isssues second prius recall in a month on crash risk" / Oct 2018
  • "Safety systems may be disabled when in use" (Mitsubishi) / Sept. 2018
    • "Inappropriate" software in the hydraulic ECU causes the pump to generate electrical noise that resets the ECU. That reset can cause: automatic braking to be cancelled, wheels lock momentarily, stability control to be momentarily cancelled, release break of brake auto-hold is active.
    • NHTSA recall 18V-621
  • "GM recalls more than 1M pickups, SUVs for power steering problem" / Sept. 2018
    • 30 crashes; two injuries, no deaths attributed
    • Voltage drop and return causes momentary power steering failure; fixed via software update
    • https://www.freep.com/story/money/cars/general-motors/2018/09/13/gm-recall-pickups-suvs-power-steering/1287911002/
  • "Expert investigation says BMW software to blame" / Aug 2018
  • "Fiat Chrysler recalls 5.3 million vehicles for cruise control defect" / May 2018
  • Incorrect Speed Limitation Software (Mercedes-Benz) / 2018
    •  These vehicles may be equipped with the incorrect reverse speed limitation software. While in reverse, any abrupt changes in steering while exceeding 16 MPH may cause the vehicle to become unstable.
    • NHTSA recall 18V-457
  • Cruise control may not disengage (Mercedes-Benz) / 2017
    • ESP software malfunction may cause engine not to reduce power regardless of speed, driving situation, or brake application.
    • NHTSA recall 17V-713
  • "Fiat Chrysler recalls 1.25 million trucks over software error" / 2017
  • Unintended vehicle movement (Ford) / 2017
    • Quick movement of gear shift can cause up to 1 second selection of reverse gear when shifting into intended drive (forward) gear.
    • NHTSA recall 17V-669
  • Air bags may not deploy in a crash (Mitsubishi) / 2017
    • SRS ECU misinterprets vibrations, disabling air bags from deploying in a crash
    • NHTSA recall 17V-686
  • Unintended acceleration failsafes "missing" (Dodge) / 2016
  • Inadvertent Side Air Bag Deployment (Chrysler) / 2015
    • Unexpected side airbags may unexpectedly deploy due to incorrect software calibration; may result in crash or injury
    • NHTSA Recall 15V-460 and 15V-467
  • Radio Software Security Vulnerabilities (Chrysler) / 2015
    • Exploitation of the software vulnerability may result in unauthorized remote modification and control of certain vehicle systems, increasing the risk of a crash.
    • NHTSA Recall 15V-461, 15V-508
  • "Toyota recalls 625,000 hybrids: Software bug kills engines dead with thermal overload" / July 2015
    • Software settings for motor/generator ECU cause thermal damage, then propulsion shutdown
    • https://www.theregister.co.uk/2015/07/15/toyota_recalls_625000_hybrids_over_enginekilling_software_glitch/
    • Note previous recall 14V-053 for similar sounding problem
  • Tire pressure monitoring system message (Ferrari) / 2015
    • TPMS displays 50 mph speed limit warning instead of "do not proceed" warning due to software defect. Driving on punctured tire would cause loss of vehicle control and crash.
    • NHTSA Recall 15V-306
  • Airbag Incorrect Deployment Timing (BMW) / 2015
    • Driver front air bag timing incorrect / fails to meet FMVSS 208 due to programming error
    • NHTSA Recall 15V-148 
  • Passenger Air Bag may be disabled (Jaguar) / 2015
    • Light weight adult may be misclassified, disabling air bag
    • NHTSA Recall 15V-093
  • Unintended side air bag deployment (Chrysler) / 2015
    • Unintended side curtain and seat air bag deployment during operation / software reflash
    • NHTSA Recall 15V-041
  • Brake controller might not activate trailer brakes (Ford) / 2015
    • Trailer brakes not activated when towing, lengthening stopping distance, increasing risk of crash. Fixed via powertrain control module reflash.
    • NHTSA Recall 15V-710
  • On but unattended vehicle may cause CO poisoning (GM) / 2015
    • Vehicle may turn on gasoline engine to recharge hybrid battery, causing carbon monoxide poisoning (e.g., if car is in garage)
    • NHTSA Recall 15V-145
  • Incorrect electric power steering software setting (Jaguar) / 2015
    • Power steering set in factory operating mode. Vehicle can experience additional steering inputs from EPS causing driver to lose ability to control the vehicle.
    • NHTSA Recall 15V-569
  • Air bag may not detect passenger in seat (Nissan) / 2015
    • Configuration management error: incorrect occupant classification software version installed, resulting in no air bag deployment
    • NHTSA Recall 15V-681
  • "Honda admits software problem, recalls 175,000 hybrids" / July 2014
  • Transmission calibration error (Ford) / 2014
    • Due to software calibration error vehicle may be in and display "drive" but engage "reverse" for 1.5 seconds.
    • NHTSA Recall 14V-204
  • Headlights may unintentionally turn off (Motor Coach Industries) / 2014
    • A mux controller may unintentionally turn off headlights while vehicle is in gear
    • NHTSA Recall 14V-370
  • Brake vacuum pump may stop functioning (Mitsubishi) / 2014
    • Software defect causes false detection of stuck relay, disabling brake power assist
    • NHTSA Recall 14V-522
  • Loss of brake vacuum assist (GM) / 2014
    • Loss of power brake assist; fixed with software reflash
    • NHTSA Recall 14V-247
  • Reprogram sensing and diagnostics module (GM) / 2014
    • Module left in "manufacturing mode" when shipped, disabling airbags
    • NHTSA Recall 14V-247
  • Passenger airbag may be disabled (Jaguar) / 2014
    • EEPROM wearout (which is due to a software defect) causes airbag to be partially or totally disabled
    • NHTSA Recall 14V-395
  • Hybrid transmission software (Champion Bus) / 2014
    • Software may improperly raise vehicle's engine speed during downshifts without the driver's input. The increase in speed may result in unintended acceleration.
    • NHTSA Recall 14V-303  (See also 14V-043; 14V-043 Navistar; 14V-026 Kenworth)
  • Cruise control unintended continued acceleration (Chrysler) / 2014
    • Unintended continued acceleration after releasing accelerator due to adaptive cruise control software; may increase risk of crash
    • NHTSA Recall 14V-293
  • Side-curtain rollover airbag deployment delay (Ford) / 2014
    • Errors in the programming software which may result in delayed deployment of side-curtain rollover airbag
    • NHTSA Recall 14V-237
  • Improper seat belt restraint software (Toyota) / 2014
    • Improper software can use insufficient force in crash (e.g., 110 pound passenger force for larger passenter)
    • NHTSA Recall 14V-272
  • Air bag may not detect passenger in seat (Nissan) / 2014
    • Software may incorrectly classify passenger seat as empty; airbag will not deploy
    • NHTSA Recall 14V-138
  • Vehicle may gradually accelerate unexpectedly (Nissan) / 2014
    • If lost signal from throttle position sensor is regained (intermittent fault) fail-safe mode is deactiveted, opening throttle resulting in "gradual" acceleration due to software error.
    • NHTSA Recall 14V-583
  • Inadvertent Air Bag deployment (Ram) / 2014
    • Side air bags deploy when hitting potholes; fixed via software update
    • NHTSA Recall 14V-528
  • Side airbags may deploy on the incorrect side (Chrysler) / 2013
    • Airbag on the wrong side of the vehicle could deploy, leaving occupants with no airbag protection at point of impact due to a software defect
    • NHTSA Recall 13V-283
  • Delayed deployment or non-deployment of airbags (Chrysler/Jeep) / 2013
    • Airbag deployment delayed or no airbag deployment in rollover due to software defect
    • NHTSA Recall 13V-233
  • Airbag deployment software (Chrysler) / 2013
    • Incorrect software installed; air bags may not deploy or might deploy improperly
    • NHTSA Recall 13V-291
  • Improper occupant classification / 2012
    • Incorrect software installed that misclassifies passengers; airbag might not deploy when it should, deploys incorrectly, or deploys when it should not
    • NHTSA Recall 12V-198
  • Occupant classification system (Hyundai) / 2012
    • Software might miss small stature adults and not deploy airbag.
    • NHTSA Recall 12V-354 
  • Cruise Control System/Brake Switch Failure (Mercedes-Benz) / 2011
    • Brake pedal may not automatically disengage cruise control as expected. (Other methods still work.)  If driver pumps brakes it will take unusually high force to stop vehicle.
    • NHTSA Recall 11V-208
  • Engine stall prevention assist software (Honda) / 2011
    • Unexpected vehicle movement from ECU software providing hybrid electric power and unexpectedly moving vehicle in reverse direction if the engine stalls.
    • NHTSA Recall 11V-458
  • Loss of steering power assist (Toyota) / 2010
  • "Toyota: software to blame for Prius brake problems" / 2010
  • ABS ECU Programming (Toyota) / 2010
    • Inconsistent brake feel; increased stopping distances for a given pedal force due to ABS programming, raising the possibility of a crash.
    • NHTSA Recall 10V-039
  • Restraint control module (Land Rover) / 2009
    • Passenger airbag disabled as a result of temporary loss of CAN network messages and a software defect
    • NHTSA Recall 09V-467
  • Double Clutch Gearbox (BMW) / 2008
    • Engine stall increasing risk of a crash due to software multistage downshift defect
    • NHTSA Recall 08V-595
  • Passenger sensing system (GM) / 2008
    • Software condition within passenger sensing system may disable passenger air bag (or enable when it should be disabled).
    • NHTSA Recall 08V-582
  • Passenger air bag fail to deploy (Nissan) / 2008
    • Passenger air bag might not deploy due to low battery voltage combined with software defect
    • NHTSA Recall 08V-066
  • Engine Control Module Software Update (VW) / 2008
    • Software defect can cause unexpected engine surge that can "result in a crash without warning."
    • NHTSA Recall 08V-235
  • SRS Electronic control unit software (Maserati) / 2007
    • Passenger air bag might not deploy if car battery is not fully charged due to software defect
    • NHTSA Recall 07V-550
  • SRS control unit software (Volvo) / 2007
    • Two software errors result in late deployment of side airbags
    • NHTSA Recall 07V-500
  • Passenger side airbag does not deploy (Volkswagen) / 2006
    • A weak battery could cause air bag control unit to deactivate due to a software defect; airbag will not deploy in a crash
    • NHTSA Recall 06V-454
  • Electronic Throttle Control (GM) / 2006
    • ETC torque monitoring failsafe disabled, permitting throttle opening greater than commanded (i.e., UA) due to a software defect
    • NHTSA Recall 06V-007
  • Powertrain control module (DaimlerChrysler) / 2006
    • Software can cause momentary lock up of drive wheels at speeds over 40 mph if operator shifts from drive to neutral and back.
    • NHTSA Recall 06V-341
  • BMW/Driver's seat occupant detection system / 2004
    • Software can't reliably determine if driver seat is occupied; airbag may not deploy.
    • NHTSA Recall 04V-379
  • Jaguar/Forward drive gear / 2004
    • Selecting forward drive gear could select reverse while in forward motion, without indication. (Apparent limp home mode logic defect.)
    • NHTSA Recall 04-024
  • BMW/ENgine Idle Speed/DME Idle Control / 2003
    • Increase of idle speed up to 1,300 RPM. If a gear is selected, the driver may feel as if the vehicle is being pushed.
    • NHTSA Recall 03V124
  • KIA/ABS Electronic Control Module / 2003
    • A programming error in ABS cases reduced braking force at speeds below 25 mph, extending stopping distances
    • NHTSA Recall 03V-158
  • "GM Admits Brake Flaws After Inquiry" / July 1999
  • Chrysler/Interior systems: air bag / 1996
    • Air bag software error which can delay air bag deployment
    • NHTSA Recall 96V-060

Noteworthy: These are software-related problems with cars that are worth knowing about, but less black and white because, for example, there has been no general recall issued.
  • To access NHTSA recalls you need to visit https://www.nhtsa.gov/recalls then select Vehicle then select "search by NHTSA ID" which can take a few mouse clicks to find on the indicated NHTSA web site.  (It might be the interface has changed since I posted this; you might need to poke around to find the lookup function.)
  • This is a work in progress and a VERY incomplete list.  I thought this would be a one-day exercise, but, well, no. If you know of something really important I've missed, please let me know!  More importantly, if you know of someone who is interested in maintaining a list like this, especially as a more rigorous academic study, I'd be happy to collaborate.  I simply don't have the time to keep up with this.
  • Reasonable people can perhaps disagree about the inclusion or exclusion of some items. But the point is really more about the volume rather than any individual item. By definition each recall is a defect that should not have been shipped, because it resulted in a recall.  I've paraphrased the recall reports. If you want to know more be sure to look at the supporting documents on the NHTSA web site, which often have more details than the summaries.
  • To be "deadly" these defects have to be software faults that either have caused, could reasonably cause, or should have reasonably prevented significant injury or death. (This includes defects in failsafes, for example) A partial list includes: un-commanded acceleration (UA), stalling at speed (dangerous when merging onto a highway), failure to deactivate cruise control, extended braking distances, airbag disablement, and incorrect airbag deployment.  What happens in practice depends upon the circumstances.
  • This should not be construed to be an expert opinion of root cause of any particular mishap. I am summarizing publicly available information and have not independently verified the technical facts in each case. Those public sources might be incorrect, or I might not have fully understood the implications of the statements in those sources. Again, this is more about the overall trend and not any particular incident report.
  • There are plenty of commenters who say things for unintended acceleration like "just apply the brakes, because brakes always overcome the engine." First, this is simply not true in many situations due to loss of vacuum assist, drivers with weak leg strength etc. A single point fault or sufficiently likely multi-point fault should not be trying to kill the occupants in the first place, so it's still a defect.
  • The air bag software problems were found in: https://www.autosafety.org/staging/wp-content/uploads/import/Historical%20Airbag%20Recalls_1.pdf  I independently verified them on the NHTSA database.
  • I independently verified on the NHTSA database some drivetrain recalls found here: https://www.autosafety.org/sites/default/files/imce_staff_uploads/Exemplary%20Vehicle%20Software%20Recalls.pdf
    and here: https://www.autosafety.org/wp-content/uploads/2016/04/2014-15-Software-Recalls.pdf
  • If you want to go exploring, you can download a copy of the raw database here that I used for some of the other defects: https://www-odi.nhtsa.dot.gov/downloads/

Saturday, September 8, 2018

Different types of risk analysis: ALARP, GAMAB, MEMS and more

When we talk about how much risk is enough, it is common to do things like compare the risk to current systems, or argue about whether something is more (or less) likely than events such as being killed by lightning. There are established ways to think about this topic, each with tradeoffs.

Tightrope Walker

The next time you need to think about how much risk is appropriate in a safety-critical system, try these existing approaches on for size instead of making up something on your own:

ALARP: "As Low As Reasonably Practicable"  Some risks are acceptable. Some are unacceptable. Some are worth taking in exchange for benefit, but if that is done the risk must be reduced to be ALARP.

GAMAB: "Globalement Au Moins Aussi Bon"  Offer a level of risk at least as good as the risk offered by an equivalent existing system. (i.e., no more dangerous than what we have already for a similar function)

MEM: "Minimum Endogenous Mortality"  The technical system must not create a significant risk compared to globally existing risks. For example, this should cause a minimal increase in overall death rates compared to the existing population death rates.

MGS: "Mindestens Gleiche Sicherheit"   (At least the same level of safety) Deviations from accepted practices must be supported by an explicit safety argument showing at least the same level of safety. This is more about waivers than whole-system evaluation.

NMAU: "Nicht Mehr Als Unvermeidbar"  (Not more than unavoidable)  Assuming there is a public benefit to the operation of the system, hazards should be avoided by reasonable safety measures implemented with reasonable cost.

Each of these approaches has pros and cons.  The above terms were paraphrased from this nice discussion:
Kron, On the evaluation of risk acceptance principles,

There is an interesting set of slides that covers similar ground here, and works some examples. In particular the graphs involving whether risks are taken voluntarily for different scenarios is thought provoking:

In general, if you want to dig deeper into this area, a search on
    gamab mem alarp 
will bring up a number of hits

Also note that legal and other types of considerations exist, especially regarding product liability.

Monday, March 12, 2018

Embedded Code Quality and Best Practices Training Videos full length

I've posted the full series of my available embedded system code quality and related best practices videos on YouTube.  These are full-length narrated slides of the core set of safety topics from my new course.  They concentrate on getting the big picture about code quality and good programming practices.
Each of the videos is posted to YouTube as a playlist, with each video covering a slide or two. The full lecture consists of playing the entire play list, with most lectures being 5-7 videos in sequence. (The slide download has been updated for my CMU grad course, so in general has a little more material than the original video. They'll get synchronized eventually, but for now this is what I have.)

Obviously there is more to code quality and safety than just these topics. Additional topics are available slides-only.  You can see the full set of course slides including for those lectures and others here:

Sunday, February 25, 2018

New Blog on Self-Driving Car Safety

I'm doing a lot more work on self-driving car (autonomous vehicle) safety, so I've decided to split my blogging for that activity.  I'll still post more general embedded system topics here, perhaps with reduced frequency.

You can see my new blog on self-driving car safety here:

Just to keep perspective, self-driving cars are still very complex embedded systems. You need to get the basics right (this blog) if you want them to be safe!

Friday, February 16, 2018

Robustness Testing of Autonomy Software (ASTAA Paper Published)

I'm very pleased that our research team will present a paper on Robustness Testing of Autonomy Software at the ICSE Software Engineering in Practice session in a late May. You can see a preprint of the paper here:  https://goo.gl/Pkqxy6

The work summarizes what we've learned across several years of research stress testing many robots, including self-driving cars.

As robotic and autonomy systems become progressively more present in industrial and human-interactive applications, it is increasingly critical for them to behave safely in the presence of unexpected inputs. While robustness testing for traditional software systems is long-studied, robustness testing for autonomy systems is relatively uncharted territory. In our role as engineers, testers, and researchers we have observed that autonomy systems are importantly different from traditional systems, requiring novel approaches to effectively test them. We present Automated Stress Testing for Autonomy Architectures (ASTAA), a system that effectively, automatically robustness tests autonomy systems by building on classic principles, with important innovations to support this new domain. Over five years, we have used ASTAA to test 17 real-world autonomy systems, robots, and robotics-oriented libraries, across commercial and academic applications, discovering hundreds of bugs. We outline the ASTAA approach and analyze more than 150 bugs we found in real systems. We discuss what we discovered about testing autonomy systems, specifically focusing on how doing so differs from and is similar to traditional software robustness testing and other high-level lessons.

Casidhe Hutchison
Milda Zizyte
Patrick Lanigan
David Guttendorf
Mike Wagner
Claire Le Guoes
Philip Koopman

Monday, January 29, 2018

New Peer Review Checklist for Embedded C Code

Here's a new peer review checklist to help improve the quality of your embedded C code.

To use the checklist, you should do a sit-down meeting with, ideally, three reviewers not including the code author. Divide the checklist up into three portions as indicated.  Be sure to run decent static analysis before the review to safe reviewer time -- let the tools find the easy stuff before spending human time on the review.

After an initial orientation to what the code is supposed to do and relevant background, the review process is:
  1. The review leader picks the next few lines of code to be reviewed and makes sure everyone is ONLY focused on those few lines.  Usually this is 5-10 lines encompassing a conditional structure, a basic block, or other generally unified small chunk within the code.
  2. Reviewers identify any code problems relevant to their part of the checklist.  It's OK if they notice others, but they should focus on individually considering each item in their part of the checklist and ask "do I see a violation of this item" in just the small chunk of code being considered.
  3. Reviewer comments should be recorded in the form: "Line X seems to violate Checklist Item Y for the following reason." Do NOT suggest a fix -- just record the issue.
  4. When all comments have been recorded, go back to step 1.  Continue to review up to a maximum of 2 hours. You should be covering about 100-200 lines of code per hour. Too fast and too slow are both a problem.
A text version of the checklist is below. You can also download an acrobat version here.  Additional pointers to support materials are after the checklist. If you have a static analysis tool that automates any of the checklist item, feel free to replace that item with something else that's important to you.

Peer Review Checklist: Embedded C Code
Before Review:
0    _____    Code compiles clean with extensive warning checks (e.g. MISRA C rules)
Reviewer #1:       
1    _____    Commenting:  top of file, start of function, code that needs an explanation
2    _____    Style is consistent and follows style guidelines
3    _____    Proper modularity, module size, use of .h files and #includes
4    _____    No orphans (redundant, dead, commented out, unused code & variables)
5    _____    Conditional expressions evaluate to a boolean value; no assignments
6    _____    Parentheses used to avoid operator precedence confusion
7    _____    All switch statements have a default clause; preferably an error trap
Reviewer #2:       
8    _____    Single point of exit from each function
9    _____    Loop entry and exit conditions correct; minimum continue/break complexity
10    _____    Conditionals should be minimally nested (generally only one or two deep)
11    _____    All functions can be unit tested; SCC or SF complexity less than 10 to 15
12    _____    Use const and inline instead of #define; minimize conditional compilation
13    _____    Avoid use of magic numbers (constant values embedded in code)
14    _____    Use strong typing (includes: sized types, structs for coupled data, const)
15    _____    Variables have well chosen names and are initialized at definition
Reviewer #3:       
16    _____    Minimum scope for all functions and variables; essentially no globals
17    _____    Concurrency issues? (locking, volatile keyword, minimize blocking time)
18    _____    Input parameter checking is done (style, completeness)
19    _____    Error handling for function returns is appropriate
20    _____    Null pointers, division by zero, null strings, boundary conditions handled
21    _____    Floating point use is OK (equality, NaN, INF, roundoff); use of fixed point
22    _____    Buffer overflow safety (bound checking, avoid unsafe string operations)
All Reviewers:     
23    _____    Does the code match the detailed design (correct functionality)?
24    _____    Is the code as simple, obvious, and easy to review as possible?
        For TWO Reviewers assign items:   Reviewer#1:  1-11; 23-24    Reviewer#2: 12-24
        Items that are covered with static analysis can be removed from checklist
        Template 1/28/2018:  Copyright CC BY 4.0, 2018, Philip Koopman

Additional material to help you with successful peer reviews: