Course Highlights:
This Masterclass examines the activities involved in testing and debugging errors, flaws, faults and failures in
real-time and embedded systems software.

Testing and debugging are two distinct activities with the shared goal of extracting defects from software and systems.
Testing uses systematic, engineering-style methods, while debugging uses investigative, research-style methods.

The class begins with a presentation of the main concepts and principles for systematic testing of embedded
systems software. Emphasis is placed on uniquely embedded issues such as flaws in interfacing, multitasking and
timing, rather than on general data processing issues.

The class continues with an examination of  approaches important in embedded software testing.  High-level testing
approaches are discussed for advanced stages of system development and integration, including security testing,
stress testing and independent verification and validation.  Techniques and tools are presented to support these
approaches.

The debugging portion of the course starts with a survey of the wide range of categories of bugs from which
embedded and real-time software can suffer .  It then focuses on a detailed discussion of the "top 5" kinds of bugs
that can be "project killers" or "product killers", including stack overflows, race conditions, deadlocks, timing problems,
and re-entrancy conditions .  Emphasis is placed on multitasking and timing behaviors.

The class continues with an examination of debugging tools that are helpful in real-time software development.
'Liveness' issues including deadly embrace, lockout, memory starvation and CPU starvation are discussed in depth.  
As are the particular sorts of bugs that tend to create problems in multi-tasking software when using real-time
operating systems.  Emphasis then focuses on the unique sorts of bugs that plague Interrupt Service Routines
("ISRs"), such as interrupt overflow.

Participants are asked to do detailed exercises on many of the techniques presented, so that the concepts and
methods taught are reinforced and absorbed into the participant's arsenal of testing and debugging skills.

This course is not a general course about software testing and debugging, but rather it is highly focused on the
testing and debugging of embedded, time-constrained, resource-constrained multitasking software that may run
under the control of a modern real-time operating system ("RTOS").

Objective of the course:
The primary goal of this course is to give the participant the skills necessary to systematically identify and correct
defects in software for real-time computer systems developed with or without an RTOS.  This is a very practical,
results-oriented course that will provide knowledge and skills that can be applied immediately.


Who Should Attend:
This course is intended for practicing real-time and embedded systems software designers, software developers,
project managers, testers and quality engineers who have responsibility for planning, implementing, testing and
debugging the software for real-time and embedded computer systems.

Course participants are expected to have practical knowledge of coding for real-time and embedded systems, plus
some familiarity with at least one RTOS.  [This knowledge can also be gained by attending one of our prerequisite
introductory courses "Introduction to Embedded Systems and Software" or "Introduction to Real-Time Operating Systems".]


Course Outline:

Embedded Testing Fundamentals

8 Principles of Testing Embedded Software
Black Box Approaches: Equivalence Partitioning, Boundary Condition Testing
White Box Approaches: Stubs and Drivers
Code Coverage Levels
High Payoff White, Gray, and Black Box Tests
Unique Issues in Embedded Testing
Exercise: DO-178B "Modified Condition / Decision Coverage"
Real-Time Correctness: 'Liveness', Race Conditions

Embedded Testing Techniques

Input Data Testing
Exercise: Backus-Naur Form of a Protocol
State Testing: State Transition Diagrams and Harel StateCharts
Exercise: State Transitions of High-Availability Switchover
Pairwise Testing and Orthogonal Arrays
Exercise: Pairwise Test Design
Fault Injection Testing

Life Cycle for Embedded Software Testing

Top-Down vs. Bottom-Up Integration: Embedded Testbeds
System & Acceptance Testing
Security Testing
Stress Testing: Case Studies of NASA Spacecraft
Cause-Effect Graphing & Decision Tables
Exercise: Cause-Effect Test Design for a Temperature Display
Independent Verification and Validation
When to Stop Testing ?

Concepts for Debugging of Embedded / Real-Time Software

What is a Bug ?
The Debugging "Cookbook"
9 Golden Guidelines for Debugging
A Taxonomy of Real-Time / Embedded Bugs
Exercise: Bugs when Porting to New Hardware

David's "Top 5"  Real-Time Bugs

Stack Overflows
Race Conditions
Deadlocks
Timing Problems
Re-entrancy Conditions

Real-Time Debugging Tools and their Limitations

Where Traditional Debugging Techniques are Insufficient
The Correctness Killers: Violations of Liveness
Deadlock: 4 Ingredients, and 8 'Fixes'
Task Lockouts and Starvations
The 3 Classes of Real-Time Software Debuggers
Cutting Edge Debuggers: Time-Reversal, SensorPoints
Logic Analyzers
Post-Mortem Debugging
Exercise: Bug in a Real-Time Controller

Mindsets for Debugging Real-Time Software
Cause and Effect: The Scientific Method
Assertions and Invariants
Memory Allocation/De-allocation Bugs
Proactive Debugging Techniques

Bugs in Multi-tasking and ISRs: The Mis-use of RTOS's

Task Decomposition and Communication Defects
Multi-tasking Debugging Techniques
Interrupt Service Routine Design Defects
RTOS Usage Defects
Exercise: Bug in Interrupt Handling

Programming Bugs in Real-Time Software

C-Language Ambiguities and Coding Disciplines
C-Language Code Analysis and Tools
Static vs. Dynamic Code Analysis
Exercise: C-Language Shenanigans
Metrics Predict Bugs
Exercise: Cyclomatic Complexity
Taxonomy of Embedded Testing Tools

Optional Sections:
Hard Real-Time Performance Bugs and Calculations
Bugs in Multi-Core and Symmetric Multi-Processing Software

Final Examination.

INSTRUCTOR:  Dr. David Kalinsky
Dr David Kalinsky has more than thirty years of experience in the design and construction of real-time and embedded computer systems software.  He is a popular lecturer and seminar leader on technologies for embedded software development, appearing before audiences of professional engineers in North America, Europe and Israel.  David regularly presents classes at the Embedded Systems Conferences on topics such as "Architectural Design of Device Drivers" and "Principles of High Availability Embedded Systems Design".

He has built and managed high-tech training programs on aspects of software engineering for the development of real-time and embedded systems for a number of Silicon Valley companies.  He has also been involved in the design of many embedded medical and aerospace systems.  In addition, he has in the past developed and taught training courses on a number of major real-time operating systems (RTOSs), including VRTX, pSOS, VxWorks, OSEK / VDX, Nucleus, OSE and others.  With his broad experience, he has trained thousands of embedded systems software engineers and architectural designers throughout the world.