Steinbeis experts conduct testing of complex driving scenarios
Vehicles offer more and more features, with everything now from straightforward assistance systems to allowing the vehicle to drive itself. Despite the obvious benefits, these features also bring risks, which need to be tested in development by conducting simulations. Vehicle simulations work through a variety of driving scenarios, which are captured in description languages. Depending on the scenario and driving maneuver, this results in high levels of complexity. The more complex a scenario, the greater the risk that it doesn’t match desired testing requirements and this might give a false impression that a test was successful. Key performance indicators (KPIs) can be used to assess the quality and complexity of scenarios. The experts at Steinbeis Interagierende Systeme GmbH (IAS) have been using a variety of KPIs to compare two different description languages for test scenarios.
In addition to open standards such as OpenSCENARIO 1 and 2, a variety of languages are now available for describing simulation environments and testing scenarios, analogous to those used in the development of driver assistance systems. The experts at IAS have been helping with the development and application of such domain-specific languages (DSLs) for several SAE levels of driving automation.
KPIs – crucial numbers in simulations
KPIs are needed to assess the efficiency of DSLs in different simulation environments and testing scenarios, particularly when it comes to development, application, and scalability. The Steinbeis experts compared the KPIs of two examples based on identical scenarios. Each was described using two different DSLs. OpenSCENARIO 1 and the EBTB language co-developed by IAS were used as the description languages. Two modeling notations were used: XML and a shorthand notation (SN) transformed from and to XML. Both OpenSCENARIO and EBTB described the runtime vehicle behavior within simulations.
Two reference scenarios were used to make initial comparisons, using the two languages as examples. A simple scenario was used to demonstrate a simple lane change, and a more complex scenario extended this situation to include stopping and acceleration. The idea of using two scenarios was to reveal dependencies and identify whether the results of one scenario were random or could be replicated in other scenarios.
The IAS experts compared the following KPIs in the simulation:
- Number of lines of code
Number of events and actions: events and actions needed for the language to model the required scenario - Number of values assigned: the number of lines with a fixed value, such as vehicle speed or acceleration time
- Number of characters (with and without spaces/tabs)
- Nesting/indentation depth (maximum depth): the number of tabs or nesting depth
A direct comparison between two scenarios
The first table (p. 44) shows measured KPIs for a simple lane-change scenario. The low values show a more simple description of the scenarios. Relative values for the SN are given in parentheses. EBTB and OpenSCENARIO 2 have the lowest values.
The second table shows the metrics of the more complex lane-change-stop-and-go scenario. The figures show an increase in the number of lines of code and an even greater increase in the number of characters in the respective test cases.
The third table makes this even clearer, showing expansion in the languages from the simple to the more complex scenario. OpenSCENARIO 2 works with the fewest lines of code, enabling scenario description with minimal code. The short notation in EBTB comes in second place, but is less expanded. OpenSCENARIO 1 requires the most lines of code and characters to describe the respective scenarios in the experimental setup. A comparison of the lane-change and lane-change-stop-and-go scenarios showed an enormous increase in lines and characters.
EBTB scenarios are more compact and less complex in the SN. The IAS experts attribute this to the XML notation, since OpenSCENARIO 1 has a very high number of structural elements. These are reduced to a minimum in the shorthand notation of EBTB. With more complex scenarios involving a large number of commands, actions, and events, this difference keeps increasing.
Simulation makes complexity visible
The example scenarios clearly show that complexity increases as the number of commands increases. It’s useful for people managing tests to be aware of this complexity. A quantitative evaluation using KPIs makes it possible to make the right decisions with an influence on future test scenarios. Testing involving high levels of complexity can be investigated with an eye to making them more precise and, as far as possible, avoid disappointing outcomes. Given the increasing complexity of vehicle functions, it’s important to address the exact testing requirements without being negatively influenced by other factors.