Sunday, April 23, 2017

SCAV 2017 Keynote: Challenges in Autonomous Vehicle Validation


Challenges in Autonomous Vehicle Testing and Validation from Philip Koopman


Challenges in Autonomous Vehicle Validation
Keynote Presentation Abstract
Philip Koopman
Carnegie Mellon University; Edge Case Research LLC
ECE Dept. HH A-308, 5000 Forbes Ave., Pittsburgh, PA, USA
koopman@cmu.edu

Developers of autonomous systems face distinct challenges in conforming to established methods of validating safety. It is well known that testing alone is insufficient to assure safety, because testing long enough to establish ultra-dependability is generally impractical. That’s why software safety standards emphasize high quality development processes. Testing then validates process execution rather than directly validating dependability.

Two significant challenges arise in applying traditional safety processes to autonomous vehicles. First, simply gathering a complete set of system requirements is difficult because of the sheer number of combinations of possible scenarios and faults. Second, autonomy systems commonly use machine learning (ML) in a way that makes the requirements and design of the system opaque. After training, usually we know what an ML component will do for an input it has seen, but generally not what it will do for at least some other inputs until we try them. Both of these issues make it difficult to trace requirements and designs to testing as is required for executing a safety validation process. In other words, we’re building systems that can’t be validated due to incomplete or even unknown requirements and designs.

Adaptation makes the problem even worse by making the system that must be validated a moving target. In the general case, it is impractical to validate all the possible adaptation states of an autonomy system using traditional safety design processes.

An approach that can help with the requirements, design, and adaptation problems is basing a safety argument not on correctness of the autonomy functionality itself, but rather on conformance to a set of safety envelopes. Each safety envelope describes a boundary within the operational state space of the autonomy system.

A system operating within a “safe” envelope knows that it’s safe and can operate with full autonomy. A system operating within an “unsafe” envelope knows that it’s unsafe, and must invoke a failsafe action. Multiple partial specifications can be used as an envelope set, with the intersection of safe envelopes permitting full autonomy, and the union of unsafe envelopes provoking validated, and potentially complex, failsafe responses.

Envelope mechanisms can be implemented using traditional software engineering techniques, reducing the problems with requirements, design, and adaptation that would otherwise impede safety validation. Rather than attempting to prove that autonomy will always work correctly (which is still a valuable goal to improve availability), the envelope approach measures the behavior of one or more autonomous components to determine if the result is safe. While this is not necessarily an easy thing to do, there is reason to believe that checking autonomy behaviors for safety is easier than implementing perfect, optimized autonomy actions. This envelope approach might be used to detect faults during development and to trigger failsafes in fleet vehicles.

Inevitably there will be tension between simplicity of the envelope definitions and permissiveness, with more permissive envelope definitions likely being more complex. Operating in the gap areas between “safe” and “unsafe” requires human supervision, because the autonomy system can’t be sure it is safe.

One way to look at the progression from partial to full autonomy is that, over time, systems can increase permissiveness by defining and growing “safe” envelopes, shrinking “unsafe” envelopes, and eliminating any gap areas.

ACM Reference format:
P. Koopman, 2017. Challenges in Autonomous Vehicle Validation. In
Proceedings of 1st International Workshop on Safe Control of Connected
and Autonomous Vehicles, Pittsburgh, Pennsylvania, USA, April 2017
(SCAV 2017), 1 page.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is  granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.
Copyright is held by the owner/author(s).
SCAV'17, April 21-21 2017, Pittsburgh, PA, USA
ACM 978-1-4503-4976-5/17/04.
http://dx.doi.org/10.1145/3055378.3055379

Monday, April 10, 2017

Challenges & solutions for Embedded Software Security, Safety & Quality (Full Tutorial Video)

This is a full-length video that talks about embedded software security, safety and quality: why it matters. What to do about it.


Embedded Software Quality Safety and Security [ECR]

The purpose of this video is to help you understand why safety and security are such a big deal for embedded systems, tell some war stories, and explain the general ways available to reduce risk when you're creating embedded and IoT products.

Topics covered include:
  • Case studies of safety and security problems
  • How to design for safety
  • How to design for security
  • Top 10 embedded software warning signs
  • How to create high quality embedded software
(27 Slides / 45 minutes)

Monday, March 27, 2017

Safety Architectural Patterns (Preview)

Here's a summary video on Safety Architectural Patterns:


Safety Architecture Patterns Preview [ECR]


Other pointers on this topic (my blog posts unless otherwise noted):
For more about Edge Case Research and how to subscribe to our video training channel, please see this Blog posting.

Monday, March 20, 2017

Critical System Isolation (Preview)

Here's a summary video on Critical System Isolation:


Critical System Isolation Preview [ECR]

Other pointers on this topic (my blog posts unless otherwise noted):


For more about Edge Case Research and how to subscribe to our video training channel, please see this Blog posting.

Monday, March 13, 2017

Redundancy Management for System Safety (Preview)

Here's a summary video on Redundancy Management:


Redundancy Management for Critical Systems Preview [ECR]

Other pointers on this topic (my blog posts unless otherwise noted):

For more about Edge Case Research and how to subscribe to our video training channel, please see this Blog posting.

Monday, February 27, 2017

Critical System Design (Preview)

Here's a summary video on Critical System Design techniques.


Critical Systems Preview [ECR]

Monday, February 20, 2017

Embedded System Dependability (Preview)

Here's a summary video on Embedded System Dependability.


Dependability Tutorial Preview [ECR]

Other pointers on this topic (my blog posts unless otherwise noted):
Other pointers
For more about Edge Case Research and how to subscribe to our video training channel, please see this Blog posting.

Monday, February 13, 2017

Safety Requirements for Embedded Systems (Preview)

Here's a summary video on Embedded System Safety Requirements.


Safety Requirements Preview [ECR]

Other pointers on this topic (my blog posts unless otherwise noted):
For more about Edge Case Research and how to subscribe to our video training channel, please see this Blog posting.

Monday, February 6, 2017

Embedded Software Safety Plan (Preview)

Here's a summary video on creating an embedded Software Safety Plan.   (See additional pointers below.)


Safety Plan Preview [ECR]

Other pointers on this topic (my blog posts unless otherwise noted):
Other pointers:
For more about Edge Case Research and how to subscribe to our video training channel, please see this Blog posting.

Monday, January 30, 2017

Autonomous Vehicle Safety: An Interdisciplinary Challenge



Autonomous Vehicle Safety: An Interdisciplinary Challenge

By Phil Koopman & Mike Wagner

Abstract:
Ensuring the safety of fully autonomous vehicles requires a multi-disciplinary approach across all the levels of functional hierarchy, from hardware fault tolerance, to resilient machine learning, to cooperating with humans driving conventional vehicles, to validating systems for operation in highly unstructured environments, to appropriate regulatory approaches. Significant open technical challenges include validating inductive learning in the face of novel environmental inputs and achieving the very high levels of dependability required for full-scale fleet deployment. However, the biggest challenge may be in creating an end-to-end design and deployment process that integrates the safety concerns of a myriad of technical specialties into a unified approach.

Read the preprint version here for free (link / .pdf)

Official IEEE version (subscription required):
http://ieeexplore.ieee.org/document/7823109/  
DOI: 10.1109/MITS.2016.2583491

IEEE Intelligent Transportation Systems Magazine (Volume: 9, Issue: 1, Spring 2017, pp. 90-96)

Correction:
"This would require a safety level of about 1 billion operating hours per catastrophic event. (FAA 1988)" should be
"This would require a safety level of about 1 billion operating hours per catastrophic event due to the failure of a particular function. (FAA 1988)"  (Note that in this context a "function" is something quite high level such as the ability to provide sufficient thrust from the set of jet engines mounted on the airframe.)

Monday, January 23, 2017

Embedded System Safety Overview (Preview)

Here's a summary overview video on Embedded System Safety.  (See additional pointers below.)


Embedded Software Safety Preview [ECR]

Other pointers on this topic (my blog posts unless otherwise noted):
On-line resources:
John Knight's book: Fundamentals of Dependable Computing for Software Engineers (2012) is an excellent current book on software dependability and safety.

Nancy Leveson has some great publications in the area of software safety, and is credited for developing this as an academic field. Anyone doing software safety should read at least these:
For more about Edge Case Research and how to subscribe to our video training channel, please see this Blog posting.

Thursday, January 12, 2017

Guest on Embedded.fm Podcast

Elecia & Chris invited me to chat with them on this week's Embedded.fm podcast and it was a lot of fun.

You can check out my episode here:

http://embedded.fm/episodes/183

Also, I highly recommend listening to Jack Ganssle's excellent episode 53: "Being a grownup engineer"

http://embedded.fm/episodes/53

Scroll through the episode list.  I'm episode 183 so you can tell they've been at this quite a while. There's a lot of great stuff to listen to.

Note added Tue. 1/17:  books are back in stock in Amazon.

Meanwhile, if you are ordering from the US, the best deal on the book is via paypal here: http://koopman.us/

Monday, January 9, 2017

Language Use (Coding Style for Compilers) Overview Video

Here's a summary video on Language Use (Coding Style for Compilers) which is half of the topic of coding style.

Other pointers on this topic (my blog posts unless otherwise noted):
For more about Edge Case Research and how to subscribe to our video training channel, please see this Blog posting.

Monday, January 2, 2017

Avoiding Embedded System Stack Overflow (Preview)

Here's a summary video on avoiding Stack Overflow. 

Stack Overflow Preview [ECR]


Other pointers on this topic (my blog posts unless otherwise noted):

Other useful pointers:
For more about Edge Case Research and how to subscribe to our video training channel, please see this Blog posting.