AUTOSAR Acceptance Tests

AUTOSAR Acceptance Tests

Acceptance tests, as a quality control mechanism, are found in many facets of manufacturing. Within the field of automotive software development, acceptance tests are used to ensure quality of work for code built both with, and without, the AUTomotive Open Standard ARchitecture (AUTOSAR) framework we are currently examining. Acceptance testing is a universal mechanism to measure whether a product meets the requirements of a feature’s specification or a particular stipulation in a contract, such as between a supplier and an Original Equipment Manufacturer (OEM).

In the case of an AUTOSAR acceptance test, a specific segment or unit of code for an automotive electronic control unit (ECU) is evaluated to ensure that it meets the requirements set forth in the AUTOSAR standard. Hence, before an ECU’s function can be subject to customer-specific acceptance testing, it should first undergo AUTOSAR standard acceptance testing. This “additional layer” testing event can be part of the requirements for a supplier to show compliance with the AUTOSAR standard. While considered “optional” by AUTOSAR.org, this testing is, in most cases, highly desirable from the OEMs’ perspective.

A broad definition of AUTOSAR acceptance testing

An AUTOSAR Classic platform’s implementation acceptance test evaluates the execution of the ECU’s basic software and runtime environment (RTE), regarding application and bus compatibility. An AUTOSAR acceptance test is written at the specification level. This is a behavior-validation test for the inter-operability of specific software components.

The RTE and basic software are two of the three main layers found in AUTOSAR architecture. The basic software layer is an abstraction of multiple facets of the lower-level software. It not only contains hardware functions (I/O hardware, for example), and drivers like memory and microcontrollers, but also, both simple and complex services, including wireless communication and crypto for cybersecurity. The RTE manages communication between layers and software components, by initiating the proper standardized interfaces through the Virtual Function Bus, and is the gateway to the basic software layer.

AUTOSAR acceptance testing is also considered “black box” testing by AUTOSAR.org, meaning the function of the software components is assessed without considering the underlying code. The behavior of the “software under test” (SUT), or software component being tested, must be entirely assessed by the observed operation of inputs and outputs at its interfaces.

The “black box” test method is very common in software engineering, though some software, as we will see, can be tested under “white box” or even “grey box” conditions. These terms refer to the opacity of abstraction, or the degree to which the operation of the function can be observed via watching the code run.

Automotive ECU programs built on AUTOSAR architecture are by definition heavily abstracted via multiple layers. Black box testing could almost be considered built-in to AUTOSAR-compliant ECU code.

Major differences between UI test, acceptance test, and AUTOSAR acceptance test

To help understand AUTOSAR’s significance in automotive software development, and the importance of AUTOSAR acceptance testing for programmers, suppliers, and OEMs, it may be useful to look at the differences between specific AUTOSAR acceptance testing, and other types of acceptance test.

Whether in automotive software or any other discipline, if we consider the idea of “test” in general enough terms, almost anything is an acceptance test. It’s probably safe to say that every test for the purpose of quality control must have acceptance and rejection criteria. However, this very broad approach to test definition is not useful when comparing different test types to one another. We need a narrower focus between three types of tests which evaluate software componentry, function, or architecture. In many cases, the type of test is dictated by the device or its intended end use.

• A User Interface (UI) test measures, records, and differentiates the performance of a device’s interface that an end user will encounter. What is a user interface? Often, what is said is that if a human will touch it, it’s a UI device. In the past, automotive UI has taken the form of levers, knobs, dials, switches, and handles to control various features and functions which now are actuated via ECU and a graphical UI on a flat screen.

User interfaces are likely the ultimate expression of abstraction. All underlying code must be hidden, not only for security and architectural standardization purposes, but because of user demand, as this is the final finished or near-finished product the end customer will see. This type of acceptance testing is absolutely black box testing, and is performed near the end of all development. UI acceptance tests can and often are automated for repetitive and regression type tests, but sometimes are still performed manually as well.

• Non-UI acceptance tests that are also not AUTOSAR specific may be used to determine adherence to a requirement specified by a customer, as of a function or a specification that the software show a certain behavior, or of another standard. These tests may be required to be performed as black box (where underlying code is entirely abstracted and only function is tested for acceptance) or white/clear box, where the underlying code is visible because its structure or architecture may be part of the acceptance test. General acceptance tests may also be performed as a “grey box” test where part of the underlying function of the interface may be visible to the test environment or to other ECU programs. This is generally seen in a cybersecurity function acceptance test¹.
• AUTOSAR acceptance testing is specification-level testing to determine how well certain modular ECU software components adhere to the AUTOSAR standard in terms of their function only. AUTOSAR acceptance testing is strictly black box. This type of acceptance testing can be performed at the application level to observe the runtime environment’s behavior or to test basic software services and libraries, or at the communication bus level to assess bus adherence to AUTOSAR protocols².

What are the primary objectives of AUTOSAR acceptance testing?

AUTOSAR acceptance testing primarily aims to minimize the test effort and test costs. One avenue toward this goal is the development and maintenance of common test protocols and methods, and the exchange of trusted test results³. To pursue this goal in a particular configured platform, we can confirm program code behavior and performance in relation to benchmarks set for it by the AUTOSAR standard.

In the AUTOSAR spirit of modularity, the primary acceptance testing goal can be split into simpler component steps. Just as the AUTOSAR method of programming encourages a modular approach (breaking down large amounts of code into smaller and more manageable chunks), we can break the primary testing goal into subordinate goals, as well. As these subordinate goals become more granular, they also become more specific. Their scope becomes more focused compared to the larger, generalized primary goal.

• The acceptance test should assess how well a specific basic software stack interacts with AUTOSAR application software components.
• The test should also assess how the basic stack collaborates with the vehicle-level communication network.

AUTOSAR’s standardized acceptance tests prompt, record, and examine the software’s responses to requests or other stimuli momentarily inserted into the code via a diagnostic tool, usually centered around the way the RTE reacts via sending outputs over standardized interfaces as defined in the AUTOSAR standard.

Who writes and performs AUTOSAR acceptance tests?

Typically, as opposed to end-user acceptance testing, AUTOSAR acceptance testing on ECU code is performed by programmers or software engineers on the supplier side. The tests may be written by the same engineers or programmers who developed the code originally, depending on the size and organization of the supplier.

The tests should be written and assembled according to AUTOSAR standards and guidelines, including formal description of the System under Test (SUT). The SUT in AUTOSAR acceptance tests comprises the runtime environment (RTE) and basic software, and are assumed operating under black box ICC1 (Implementation Conformance Class one). ICC1 is one of three Implementation Conformance Classes, a term in software engineering that differentiates between groups or clusters of entities and their interface and functionality compliance. (ICC2 and ICC3 are beyond the scope of this AUTOSAR acceptance test article.)

Summary

AUTOSAR acceptance tests examine the operation of certain modular software components within an AUTOSAR-compliant electronic control unit’s code, without consideration of the underlying code structure. In this way, they are similar to a user acceptance test, except that the AUTOSAR acceptance test occurs much earlier in development, and is not performed by the end user. Rather, this acceptance test is often used to prove that a particular software component stack complies with the AUTOSAR standard.

1. Jake Frankenfield. “Gray Box.” 19. March 2023. https://www.investopedia.com/terms/g/gray-box.asp
2. AUTOSAR.org. “Overview of Acceptance Tests.”https://www.autosar.org/fileadmin/standards/tests/1-2/AUTOSAR_EXP_AcceptanceTestsOverview.pdf
3. Ibid.