Acceptance tests, as a quality control mechanism, are found in many facets of manufacturing. Within the field of automotive software development, acceptance tests are used to ensure quality of work for code built both with, and without, the AUTomotive Open Standard ARchitecture (AUTOSAR) framework we are currently examining. Acceptance testing is a universal mechanism to measure whether a product meets the requirements of a feature’s specification or a particular stipulation in a contract, such as between a supplier and an Original Equipment Manufacturer (OEM).
In the case of an AUTOSAR acceptance test, a specific segment or unit of code for an automotive electronic control unit (ECU) is evaluated to ensure that it meets the requirements set forth in the AUTOSAR standard. Hence, before an ECU’s function can be subject to customer-specific acceptance testing, it should first undergo AUTOSAR standard acceptance testing. This “additional layer” testing event can be part of the requirements for a supplier to show compliance with the AUTOSAR standard. While considered “optional” by AUTOSAR.org, this testing is, in most cases, highly desirable from the OEMs’ perspective.
An AUTOSAR Classic platform’s implementation acceptance test evaluates the execution of the ECU’s basic software and runtime environment (RTE), regarding application and bus compatibility. An AUTOSAR acceptance test is written at the specification level. This is a behavior-validation test for the inter-operability of specific software components.
The RTE and basic software are two of the three main layers found in AUTOSAR architecture. The basic software layer is an abstraction of multiple facets of the lower-level software. It not only contains hardware functions (I/O hardware, for example), and drivers like memory and microcontrollers, but also, both simple and complex services, including wireless communication and crypto for cybersecurity. The RTE manages communication between layers and software components, by initiating the proper standardized interfaces through the Virtual Function Bus, and is the gateway to the basic software layer.
AUTOSAR acceptance testing is also considered “black box” testing by AUTOSAR.org, meaning the function of the software components is assessed without considering the underlying code. The behavior of the “software under test” (SUT), or software component being tested, must be entirely assessed by the observed operation of inputs and outputs at its interfaces.
The “black box” test method is very common in software engineering, though some software, as we will see, can be tested under “white box” or even “grey box” conditions. These terms refer to the opacity of abstraction, or the degree to which the operation of the function can be observed via watching the code run.
Automotive ECU programs built on AUTOSAR architecture are by definition heavily abstracted via multiple layers. Black box testing could almost be considered built-in to AUTOSAR-compliant ECU code.
To help understand AUTOSAR’s significance in automotive software development, and the importance of AUTOSAR acceptance testing for programmers, suppliers, and OEMs, it may be useful to look at the differences between specific AUTOSAR acceptance testing, and other types of acceptance test.
Whether in automotive software or any other discipline, if we consider the idea of “test” in general enough terms, almost anything is an acceptance test. It’s probably safe to say that every test for the purpose of quality control must have acceptance and rejection criteria. However, this very broad approach to test definition is not useful when comparing different test types to one another. We need a narrower focus between three types of tests which evaluate software componentry, function, or architecture. In many cases, the type of test is dictated by the device or its intended end use.
User interfaces are likely the ultimate expression of abstraction. All underlying code must be hidden, not only for security and architectural standardization purposes, but because of user demand, as this is the final finished or near-finished product the end customer will see. This type of acceptance testing is absolutely black box testing, and is performed near the end of all development. UI acceptance tests can and often are automated for repetitive and regression type tests, but sometimes are still performed manually as well.
AUTOSAR acceptance testing primarily aims to minimize the test effort and test costs. One avenue toward this goal is the development and maintenance of common test protocols and methods, and the exchange of trusted test results3. To pursue this goal in a particular configured platform, we can confirm program code behavior and performance in relation to benchmarks set for it by the AUTOSAR standard.
In the AUTOSAR spirit of modularity, the primary acceptance testing goal can be split into simpler component steps. Just as the AUTOSAR method of programming encourages a modular approach (breaking down large amounts of code into smaller and more manageable chunks), we can break the primary testing goal into subordinate goals, as well. As these subordinate goals become more granular, they also become more specific. Their scope becomes more focused compared to the larger, generalized primary goal.
AUTOSAR’s standardized acceptance tests prompt, record, and examine the software’s responses to requests or other stimuli momentarily inserted into the code via a diagnostic tool, usually centered around the way the RTE reacts via sending outputs over standardized interfaces as defined in the AUTOSAR standard.
Typically, as opposed to end-user acceptance testing, AUTOSAR acceptance testing on ECU code is performed by programmers or software engineers on the supplier side. The tests may be written by the same engineers or programmers who developed the code originally, depending on the size and organization of the supplier.
The tests should be written and assembled according to AUTOSAR standards and guidelines, including formal description of the System under Test (SUT). The SUT in AUTOSAR acceptance tests comprises the runtime environment (RTE) and basic software, and are assumed operating under black box ICC1 (Implementation Conformance Class one). ICC1 is one of three Implementation Conformance Classes, a term in software engineering that differentiates between groups or clusters of entities and their interface and functionality compliance. (ICC2 and ICC3 are beyond the scope of this AUTOSAR acceptance test article.)
AUTOSAR acceptance tests examine the operation of certain modular software components within an AUTOSAR-compliant electronic control unit’s code, without consideration of the underlying code structure. In this way, they are similar to a user acceptance test, except that the AUTOSAR acceptance test occurs much earlier in development, and is not performed by the end user. Rather, this acceptance test is often used to prove that a particular software component stack complies with the AUTOSAR standard.