Static Analysis Ranked Defect List

 Crazy idea of the day: Static Analysis Ranked Defect List.

Here is a software analysis tool feature request/product idea: So many times we see the problem that a static analysis tool or other way of automatically finding bugs inundates developers with so many possible bugs they turn it off in frustration. Or maybe they have a requirement to ship only "clean" code so they don't run the tool because then there is nothing to clean up. Put your favorite head-in-the-sand organizational dysfunction here.

I'd love to be able to recommend a static analysis or other tool that has a feature that reports that top 10 defects currently in the code base, ranked by likely risk. Go ahead and use machine learning for the ranking; fine with me. Regardless of methodology, fixing only some is a guess after all.

Ultimately, all the static analysis warnings should be cleaned up. Having a "top K warnings" feature would allow a team to make progress over time instead of simply sweeping the entire mess under the carpet and ignoring what is often very valuable information that predicts defect escapes to deployment.

For tools with many warnings an approximation can be to gradually turn on a series of warning flags over time and/or just run the warnings on a subset of modules. But a global "these are the 10 scariest warnings" would be a nice feature.

Thoughts? Does some tool already do this?

Job and Career Advice

I sometimes get requests from LinkedIn contacts about help deciding between job offers. I can't provide personalize advice, but here are my thoughts in general.

You must accept personal ownership for choosing what you want to do with at least the next few years of your life. Nobody can do this for you. Some luck is always involved, but fortune favors the prepared. It is up to you to set your own course.

Years of your working career are a scarce, non-renewable resource, so spend care in deciding. On the other hand, it is difficult to make a choice because, as they say, it is difficult to make predictions -- especially if they're about the future. It's even harder to foresee consequences, especially on your first couple jobs when you are learning how everything works. And there are always surprises, even for the most experienced of us.  But if you end up making a choice that works out poorly, then figure out the lesson to learn and switch to something else.

Take a look at how the job fulfills or supports your needs on Maslow's Hierarchy (https://en.wikipedia.org/wiki/Maslow%27s_hierarchy_of_needs).  Some of us live to work; some work to live.  There is no one right answer, but at least this gives you a simple checklist of what you want to be provided by the job -- and what you don't.  If you're signing up for a 12 hour x 7 day type job, it had better go a long way to filling up that pyramid in an acceptable, non-destructive way while you're at work, because you'll always be at work. (And that might be OK for some people in some phases of their career. But not for others.)

Maslow's Hierarchy of Needs [https://en.wikipedia.org/wiki/Maslow%27s_hierarchy_of_needs]

All organizations have dysfunction. Figure out if this organization's dysfunctions are going to be irritating to you or get in the way of satisfying your hierarchy of needs.  Most people can stand an irritating environment less well than they think they can over long stretches of time.

When you're young, there is a lot to be said for working in a structured, mature organization. You can learn a lot by watching how experienced folks work rather than by making rookie mistakes (if you are paying attention). Later on you might want less structure.  Skipping right to an unstructured job at an immature company will teach you a lot of bad habits that it can take a lifetime to unlearn, and leave many holes in your practical education. (Some jump right to a startup company with no "graybeards." Some skip college. Some get rich by winning the lottery. Some don't. I can only tell you how to stack the odds in your favor.) Consider the availability of mentors in your new position. Even the smallest, newest of startups can work for this if the mentorship is there.

Ask if the level of responsibility & authority is a fit both in terms of scope and structure.  My experience has ranged from military officer (highly structured) to consultant (freedom but few safety nets unless you've already built up a big cushion toward retirement).  Where you want to be will likely change as your career progresses.

Read the general job hunting advice books/web sites for things such as the realities of accepting a low paying first job and trying to get raises later.  The classic book is "what color is your parachute," but no doubt there are others, keeping in mind that highly skilled workers are a bit different than the general work force.  It helps to have a realistic understanding of what you are worth, and to get some objective advice from someone you trust on whether you're getting taken advantage of in a job offer.

2023 update: the flood of cheap venture money is drying up along with rising interest rates. It is easier to justify sky-high salaries when investor emphasis is on subsidized growth rather than profit margin.  Don't be surprised if there is downward pressure on salaries. Many will find themselves losing to inflation even if they maintain their salary. Junior folks and those with demands easier to fill with new grads will probably be hurt the most here. It's hard to predict how this will play out, but expect a fundamental change from the days of near-0% interest rates driving leveraged speculative investments.

After you've considered the above, IMHO only then should you worry about the more common philosophical areas you see mentioned on this topic.  (And really, most of them end up on the upper levels of Maslow's Hierarchy.)  My personal preferences are:

• Surround yourself with the smartest, most capable people you can. But stop short of jerks. Note that any company with a "no jerks" policy is making a relative statement to their existing staff, rather than an absolute measurement. Pay attention during the interview to this.
• Work for good leaders that support and empower those who work with them.
• Take advantage of any opportunity you can get to improve your communication skills and soft skills.
• If you're taking a job purely for the money, go into that situation with an exit plan and target exit date.  Make sure that is really how you want to spend a part of your life -- but in some circumstances this can be the right move.
• If you're stressed out, it's time to find a new job.  (Or re-invent your job from within.)
• If you're stressed out about your career, it's time to reinvent yourself and find a new career.
• I personally think the whole LeetCode interview process is nothing more than ritualized hazing.  More than a two-round interview process is another caution sign unless you are signing up for an extremely senior position. Anyone with a strong GRE score (or top-tier SAT score) has already sacrificed at the altar of preparing for a required hoop sufficiently, and that probably taught them more generalized skills. But apparently others disagree on this point.
• If you strive to be the absolute best at what you do, opportunities will find you. (Being visible helps: speak at conferences, write something that people might see somewhere.)
• If most days you wake up and are eager to get to work, reflect on how fortunate you are to have that.

Don't forget the part that you must own your choice. Your preferences will probably differ.

Please do not contact me with questions about your particular individual situation. The hours in my day are already too few to accomplish what I'd like for my personal goals.  I took some time to write this to help as many people as I can (one of my goals), but I lack the time to provide individual responses.  So if I don't respond to a personal query, please understand (and better yet, send the personal query to a trusted friend instead).

I hope this is helpful, and wish you the best of luck in your job choices and your career!

What compiler warnings should you enable?

 Here is a brief piece I wrote that Jack Ganssle just ran in The Embedded Muse 460.

The context was a previous discussion about enabling compiler warnings.

Better Embedded System Software e-Book & Paperback

There are only a handful of hardcover books left of the first edition, so I spend some time converting things over to an eBook & Paperback edition.

Amazon Kindle: https://amazon.com/gp/product/B08TZ9LYXC

Smashwords (epub): https://www.smashwords.com/books/view/1264918

Barnes & Noble (ebook): https://www.barnesandnoble.com/s/philip%20koopman

This is not a 2nd edition, but more like version 1.1.  The changes are:

• Some minor rewording and cleanup.
• A few small sections rewritten to reflect lessons I've learned about how to better explain things from teaching courses.  However, scope remains the same and the hardcover book is still serviceable if you already have that.
• A new summary list of high-level takeaways in the conclusions chapter.
• Publication support for everywhere KDP reaches, with local distribution in all supported markets.

The paperback is probably what you'd expect given the above, and is Print-On-Demand with production handled directly by Amazon.  There is no index due to publication platform issues. However, the table of contents is pretty well structured and in most cases that will get you where you need to go.  The price is significantly lower than the hardcover, and non-US readers can get it printed and shipped from someplace much closer to home since KDP has local POD for markets in Europe and Asia.

Amazon indicates they will print-on-demand for the following markets: US, DE, ES, FR, IT, UK, JP and CA.  Some readers report availability in other markets as well (for example, AU). So try your usual Amazon marketplace first, then one of the others close to you to minimize shipping cost.

The eBook is reflowable text for the body text (authored in EPUB format, but Amazon changes formats I believe).  Bitmaps are used for figures and equations, so it should look fine on most viewing devices without symbol font issues.  The price is significantly lower than the paperback since production and distribution costs are much lower.

Amazon has worldwide distribution rights for the eBook, but availability varies based on which country your device is set up for.  If your kindle is set up as being in the US market then you should have no troubles purchasing.  Many other markets (especially in Europe) should be fine as well.  Amazon promises e-boook availability in these specific markets: US (.com), IN, UK, DE, FR, ES, IT, NL, JP, BR, CA, MZ, AU.

The book is published via KDP, but Digital Rights Management (DRM) is OFF.  That should help folks with non-Amazon viewers (but I'm not able to provide support for how to side-load onto whatever platform).  If you have Kindle or a machine that runs the Kindle App then it should be seamless as with any other Kindle book.

I really appreciate the support of the thousands of readers of the hardcover edition over the past years. I hope that this makes the material more broadly available!

Link to free preview (from the hardcover edition)

The Y2038 Problem. Sooner than you think.

In the coming years, there will be other time rollovers beyond Y2K. The next big one isn't all that far away.

Contrary to what you might have heard, the reason more computers didn't break on Jan 1st 2000 wasn't because it was a false alarm. It was because massive resources were poured into avoiding many of the problems.  And many things did in fact break, but backup plans were in place.  (I recall not getting financial reports for most of 2000 for my spending accounts at work.  So I had to keep my own books and hope I didn't overspend -- because the old accounting system expired at the end of 1999 and the new one wasn't on-line until Fall 2000.)

In January 2021 we saw some aftershocks when a 2-year time digit window hack ran out of steam from Y2K patches.  But the world didn't come to an end.

The next potentially huge time problem will be January 2038 when the 32-bit signed Unix time in seconds rolls over.  

Plenty of embedded systems last 20+ years (already we are closer than that to 2038).  Plenty of embedded systems are using 32-bit Unix, since 64-bit CPUs just cost too much for the proverbial toaster oven.  An increasing number of systems are updatable, but many require manual intervention.   Updating your DVD player (if we still have them in 2038) won't be so bad.  Updating a natural gas pipeline valve in the middle of nowhere -- not as fun.   Updating all your smart light bulbs will range from tedious to buying all new lightbulbs. And so on.

This is a good time for embedded system designers to decide what their game plan is for Y2038.  As your expected product life starts overlapping with that (as I write this, it's only 17 years away), you're accumulating technical debt that will come due in a big chunk that year.  Better to have a plan now than a panic later.  Later has a way of sneaking up on you when you're not looking.

For a more detailed list of timer rollover issues, see:

http://www.lieberbiber.de/2017/03/14/a-look-at-the-year-20362038-problems-and-time-proofness-in-various-systems/

62 Software Experience Lessons by Karl Weigers

Karl Weigers has an essay about lessons he's learned from a long career in software development. You should benefit from his experience. The essay covers requirements, project management, quality, process improvement, and other insights.

https://medium.com/swlh/62-lessons-from-50-years-of-software-experience-2db0f400f706

A good example from the article is:

"You don’t have time to make every mistake that every software practitioner before you has already made. Read and respect the literature. Learn from your colleagues. Share your knowledge freely with others." 

LINT does not do peer reviews

https://pixabay.com/vectors/code-programming-head-computer-2858768/

Once in a while I run into developers who think that peer review can be completely automated by using a good static analysis (generically "lint" or compiler warnings).  In other words, run PC-LINT (or whatever), and when you have no warnings peer review is done.

Nope.

But the reality has some nuance, so here's how I see it.

There are two critical aspects to style:
  (1) coding style for compilers  (will the compiler generate the code you're expecting)
  (2) coding style for humans   (can a human read the code)

A good static analysis tool is good at #1.  Should you run a static analysis tool?  Absolutely.  Pick a good tool.  Or at least do better than -Wall for Gcc (hint, "all" doesn't mean what you think it means (*see note below)).  When your code compiles clean with all relevant warnings turned on, only then is it time for a human peer review.

For #2, capabilities vary widely, and no automated tool can evaluate many aspects of good human-centric coding style.  (Can they use heuristics to help with #1?  Sure.  Can they replace a human?  Not anytime soon.)

My peer review checklist template has a number of items that fall into the #1 bin. The reason is that it is common for embedded software teams to not use static analysis at all, or to use inadequate settings. So the basics are there.  As they become more sophisticated at static analysis, they should delete the automated checks (subsuming them into item #0 -- has static analysis been done?).  Then they should add additional items they've found from experience are relevant to them to re-fill the list to a couple dozen total items.

Summarizing: static analysis tools don't automate peer reviews. They automate a useful piece of them if you are warning-free, but they are no substitute for human judgement about whether your code is understandable and likely to meet its requirements.

Pointers:

• Coding style for Compilers   
• Coding style for Humans
• Peer review tutorial
• Peer review checklist template

* Note: in teaching I require these gcc flags for student projects:
-Werror -Wextra -Wall -Wfloat-equal -Wconversion -Wparentheses -pedantic -Wunused-parameter -Wunused-variable -Wreturn-type -Wunused-function -Wredundant-decls -Wreturn-type -Wunused-value -Wswitch-default -Wuninitialized -Winit-self