An Introduction to Verification and Validation Testing for ADAS

An Introduction to Verification and Validation Testing for ADAS

Advanced Driver Assistance Systems (ADAS) have become popular features in automobiles. In addition to providing driver convenience, ADAS features are having a positive effect on safety by reducing head-on, rear-end, and cross-traffic collisions.

Before the time of automation, automotive manufacturers could easily test a vehicle and its components to ensure they operated correctly under driver control. Flip the switches, check the displays, run the vehicle around the test track, slam the doors, etc. If it all checked out, testing was complete.

Then, different functions started to appear to aid the driver. Cruise control and anti-lock brakes were fairly basic, yet these still triggered more scrutiny since the driver was now relying on a system to manage vehicle speed or minimize the loss of control under hard braking.

Today’s ADAS design such as adaptive cruise control, lane-keeping, and collision-avoidance braking greatly extend the systems’ role in operating the vehicle. When placing such systems in a vehicle, manufacturers must demonstrate not only that the systems meet all specified requirements but also that the systems will do what is truly intended, even under changing conditions. Without this assurance, safety is at risk.

The challenge then, as levels of automation and system complexity both increase, is to ensure that design and testing programs are sufficiently robust.

Functional Safety – The Starting Point

Before performing any testing, automotive engineers need to know the conditions against which the systems will be tested. The best starting point is Standard ISO 26262, Functional Safety – Road Vehicles, from the International Organization for Standardization (ISO). The standard promotes a safety culture that should become deeply rooted within an organization. It establishes measures to reduce risks to acceptable levels and applies to both hardware and software across the vehicle’s entire lifecycle.

The objective of functional safety is that systems and components either operate and respond to inputs correctly or fail predictably and acceptably. The processes focus primarily on the avoidance of malfunctions.

Under ISO 26262, a hazard analysis and risk assessment (HARA) is performed in the concept phase. Based on the severity, exposure, and controllability of events identified in the HARA, Automotive Safety Integrity Levels (ASILs) are determined which, in turn, establish the level of rigor required in each step of development, testing, and production. The quality of the HARA and ASILs will be directly affected by the extent of sincere effort expended in identifying as many hazards as possible.

For each HARA event that poses an unacceptable safety risk, safety goals are assigned that will bring those risks within an acceptable range. These goals are broken down into lower-level safety requirements and allocated to individual components within the ADAS. Engineers then design and test the components in a manner that ensures the safety requirements are fulfilled in the delivered product.

This is not necessarily a straight-line process. During ADAS verification and validation (V&V), it may be found that residual risks (those which have not yet been designed out of the system) may be greater than what is deemed acceptable. In this case, the process starts over in a circular fashion.

For lower levels of autonomy, ISO 26262 may be sufficient. However, as ADAS become more complex and are allowed to assume more control from the driver, there is an increasing risk of unintended behaviors from systems that are operating “correctly.” Therefore, functional safety alone is not enough.

Levels of Autonomy

The Society of Automotive Engineers (SAE) defines autonomy across multiple levels from Level 0, where warnings and momentary assistance may be provided, to Level 5, where the vehicle can perform all driving functions under all conditions.

Many ADAS features today fall within L0-L2 where the driver must maintain command of the vehicle even in the presence of combined automated controls. At this level, hazards are more easily identified.

But, ADAS at L3 are becoming more widespread. For L3-L5, system behaviors are more dynamic, many hazards may be unknown, and the risk of unintended behaviors rises dramatically.

Planning for Unintended Behaviors

Consider this scenario: A vehicle is equipped to L3 with lane-keeping, adaptive cruise control, and automatic emergency braking. On the freeway, the vehicle is capable of driving itself. As examined under functional safety, all sensors and actuators work as designed.

But how might these systems react to “odd” situations? Could lane-keeping be confused by missing lane markers or changes in road surface? Can sensors be blinded by fog or the setting sun? What happens if another vehicle is in an adjacent lane but, on a tight curve in the road, the braking system sees that vehicle as “ahead” instead of “beside” and considers it to be an obstacle? For L3-L5 vehicles, the sensor and control systems must be able to appropriately recognize, diagnose, and react to diverse situations.

Safety of the Intended Functionality (SOTIF) is the next step beyond functional safety. SOTIF examines behaviors that may occur under known or predictable circumstances (including reasonably foreseeable misuse) and in unknown situations. Standard ISO/PAS 21448 (under development) describes SOTIF practices and guidelines.

Under ISO/PAS 21448, the overall approach is similar to ISO 26262 (HARA; examination of severity, exposure, and controllability; safety goals), but the focus is now on potential unintended system behavior rather than malfunctions.

Verification and Validation

ISO 26262 and ISO/PAS 21448 guide automotive engineers to identify hazards, risks, and resulting safety goals. The check on how well the design meets those goals comes through verification and validation.

In verification, systems and supporting software undergo quality audits, testing, and expert reviews to confirm that all are designed to specification. During validation, product tests – including full operation of systems and components – ensure the end product will function as intended and confirm the adequacy of the safety goals.

For L0-L2 autonomous features, testing and validation strategies provided by ISO 26262 are enough to validate the system design. However, with ADAS transitioning quickly into higher autonomy levels, more robust processes are necessary.

ADAS-supported vehicles must operate in varying conditions and, especially, combinations of conditions (e.g., low visibility in fog AND high traffic density; high speed AND broken pavement). This can raise a significant possibility of confusion regarding mode and responsibilities between systems operating together or in parallel. Therefore, not only must systems be tested in ways that mimic all possible conditions and misuse cases imagined in the HARA process, it is imperative that the systems also be exposed to corner cases – circumstances outside of “normal” operating parameters. To help understand system behavior for corner case scenarios, ISO/PAS 21448 has a theoretical approach and guidelines for testing strategies.

SOTIF guides engineers to test product behaviors in both known and unknown situations. Perhaps the unknown scenarios cannot be tested directly; however, a testing plan that incorporates a broad mix of simulated scenarios can uncover hidden faults and increase confidence in system performance under real-time driving conditions. Historically, validation plans have included controlled, on-road testing (e.g., proving grounds). Considering the number of test variations that may be needed for truly robust validation of L3-L5 vehicles, lab simulations such as Hardware in the Loop, Software in the Loop, and Driver in the Loop can provide the means to cover a high number of situations cost-effectively.

Sending a product to market with 100% reliability and zero faults is unlikely due to uncertainties that cannot be avoided. Post-deployment long-term field monitoring is then an important step to enable safety engineers to identify necessary changes. Such changes must conform to the same design and test rigor as the original product to make sure that updates do not introduce any new risks or alter the system functions in a way that would affect safety.

Challenges

Developing sufficiently robust validation processes comes with challenges. For L0-L2 vehicles, the vehicle is constantly under driver control and all the worst-case scenarios can be captured to ensure system safety. But in L3-L5, the vehicle is in control and the driver does not need to be fully engaged. Safe system operation, where the vehicle pilots itself under demanding and potentially unforeseen conditions, requires a unique approach that has substantial rigor.

The following represent some of the V&V challenges vehicle developers face.

Today’s ADAS are the initial steps toward fully autonomous vehicles. Complex maneuvers such as overtaking, lane changing, and tunnel driving will someday be expected. Though most manufacturers are focused at the moment on systems that do not exceed L3, it would be prudent to develop validation processes now that contemplate the future at L4 and L5.

Expanded use of autonomous features, to the point they become common and customary, will tend to convert drivers into complacent occupants. This lack of attention will affect the operator’s ability to react to emergencies. Though the true and full effect is unknown, this concern should be factored into design and testing.

Sensor interaction is a normative procedure for L3-L5 systems to perform critical driving tasks. Sensors must capture all the essential information and respond instantaneously to allow actuation of the necessary control functions. To aid response, control units can be pre-supplied certain information such as preloaded route geography, environmental conditions, and expected speed range. But, driving patterns and interactions with other actors (neighboring vehicles, pedestrians, animals, and the like) are always shifting. These factors become inputs to validation.

During its lifetime, a vehicle may require software updates or hardware changes. Due to the complex nature of ADAS/autonomous system configurations, it is critical to address how the updates and modifications would be handled and to model these activities in the V&V process.

L3-L5 vehicles rely on complex algorithms that may include extensive machine learning or artificial intelligence. This may require special consideration for V&V efforts: How do you test a system that will modify itself over time? How do you confirm that the systems have learned or will learn proper driving techniques and rules of the road?

Conclusion

Increased user demand for ADAS features in automobiles requires manufacturers to tackle new challenges to ensure that vehicles are safe. Traditional testing programs are no longer sufficient when considering the automated functions now being installed. Though ISO 26262, the functional safety standard, provides an excellent foundation, more is needed.

ADAS operate primarily in a dynamic environment; testing programs must represent the range of scenarios in which those systems must function. ISO PAS 21448, Safety of the Intended Functionality, provides recommendations in this direction.

Vehicle autonomy will surely increase. Organizations in this industry are well-advised to prepare today for the depth of validation testing necessary to protect vehicles and their occupants as this environment evolves.

LHP can play a crucial role in supporting your organization’s technical and process implementation needs!

Further reading and references

• The Guidelines for Developing Test Solutions for Advanced Driver Assistance Systems (ADAS).