Automotive OTA Automated Testing Solutions

With the rapid development of wireless communication technologies such as 5G, the automotive industry and wireless communication technology are further integrated, which is rapidly promoting the development process of automobile intelligence. OTA technology is one of the symbols of automobile intelligence, and OTA upgrades continue to provide customers with New features continuously enhance user experience. At the same time, it can also repair software problems reported by users and enhance the manufacturer’s after-sales service experience.

At present, OTA remote upgrade technology has become a standard feature of many automobile manufacturers. Since OTA upgrades may bring potential risks (especially FOTA upgrades that include ECU firmware updates), the testing process for OTA systems is particularly important.

Human-computer interaction test (OTA functional test)

• OTA background management operations (interaction with TSP)

OTA backend management operations are mainly responsible for uploading OTA packages and formulating OTA upgrade strategies, including backend login, vehicle model selection, vehicle VIN number confirmation, OTA release, etc.

• Resource preset before OTA upgrade

Before the OTA upgrade, the car needs to be flashed to the initial version (the version before the OTA upgrade), and then resources will be preset (such as installing commonly used apps, copying audio and video files, and entering chat records) in order to check the OTA upgrade. Whether the scene will change later (whether installed apps or chat history will be lost).

• Perform OTA upgrade operation

The test engineer will perform the download task in three ways (central control screen download, mobile phone download, scheduled download operation), and download the OTA upgrade package from the server to the vehicle. After the download is completed, the system prompts the test engineer to upgrade the installation package. If If the upgrade conditions are met, the upgrade package installation process will be entered. After the installation is completed, the test engineer will check the version number information to determine whether to upgrade to the latest version, and also check whether the previously installed App and chat records are lost to determine whether this time Whether the upgrade was successful.

• Relevant function testing under abnormal scenarios

Abnormal scenarios during the OTA process may cause the system to crash. The OTA test needs to simulate these scenarios and verify whether the system has done evasive processing in this scenario. These abnormal scenarios include resumption of downloads (downloads caused by power outages, communication interruptions, or network deterioration). Interruption), download failure/retry, upgrade failure and rollback to the previous system version, etc.

Software reliability and stress testing

• OTA upgrade precondition test

Safety is the primary consideration in automobile design. OTA upgrade updates each controller system, which will inevitably affect the vehicle condition. At the same time, failure of the upgrade will cause the vehicle system to be unable to operate. Ensuring the safety of the upgrade is the top priority in the design of each vehicle system. Therefore, the system will check the current vehicle status before executing the OTA package installation. OTA upgrades are not allowed under certain vehicle conditions, such as the car being charged or the car's battery is low. Test engineers need to simulate the above preconditions and verify whether the system will refuse to install the OTA package under the above preconditions to ensure the safety of the upgrade process to the greatest extent.

• OTA upgrade success rate test

OTA upgrades are intended for a large number of car users. In order to ensure safety, it is necessary to test the success rate of upgrades on multiple vehicles. Before the official release of OTA, repeated tests of the OTA download and installation process for multiple vehicles need to be carried out to verify this OTA upgrade. failure probability, discover some random potential risks in advance, and avoid problems during large-scale OTA upgrades.

• Stress test of multi-terminal OTA download

There are more and more car models that support OTA functions, and more vehicles may perform OTA download operations at the same time. At the same time, we also consider the OTA download scenario in silent upgrade mode and examine the impact on the server in the scenario where multiple terminals download OTA packages at the same time to avoid major incidents. An abnormality occurred during scale upgrading.

Function and performance testing of each controller

OTA upgrade is to update each controller system, which may include new functions, system performance optimization, system bug modification, etc. In addition to verifying the modified points, the test engineer must also check whether it causes new problems, especially if the changes are relatively large. For large modules, test engineers need to formulate test strategies for the corresponding controller based on the content of the OTA upgrade.

• Pain points of manual OTA testing

1. Some special working conditions for OTA upgrade testing are difficult to implement. These working conditions involve preconditions that do not allow upgrades, different network environments, etc. Many manufacturers often lack testing procedures in this area, which brings some potential risks.

2. The manual OTA test process is cumbersome and involves multi-module collaboration and multi-link switching. It involves not only TSP side operations, but also operations between mobile phones and vehicles; it involves not only wireless cellular network environments, but also wireless WiFi environments and LAN links. wait. Manual switching efficiency is very low in the above environment.

3. There are many models and versions, and manual testing requires heavy workload, high cost, and low efficiency, which greatly affects the progress of OTA release.

• How to deal with the challenges of OTA testing

1. Simulate different upgrade environments through the simulation system to achieve simulation of complex environments and simulate various business scenarios of real vehicles in the test environment.

2. The overall test plan is fully controlled through the automated testing system, without manual participation. Each module is automatically called through the program to complete the entire testing process.

Dongzhou OTA test solution consists of human-computer interaction system, simulation system, assertion system, automation software system, TSP interaction system (OTA background automation management) and the system under test (composed of each domain controller and interactive screen).

1. Human-computer interaction system (execution system)

Through the HMI manipulator and ADB, the interactive operation with the car, machine and mobile phone is realized, simulating the manual operation of the test process, realizing single click, double click, sliding, etc., 24 hours of uninterrupted testing, improving test efficiency and shortening the test cycle.

2. Simulation system

The simulation system can simulate various test environments and working conditions, especially for environments where manual testing is difficult to achieve (such as prerequisites for OTA installation and various network environments, etc.).

3. Assertion system

Provides a variety of assertion methods such as images, text, sounds, and messages. The self-developed image algorithm is not affected by light and angle. Various assertion methods cover all test scenarios to ensure the accuracy of test result assertions.

4. System under test

It consists of a car machine, a screen and various controllers, which are integrated into the bench and connected through CAN bus, ETH bus, LIN bus, etc.

5. TSP automated interaction system

The TSP automatic interaction system implements operation management of the OTA backend, including automatic login, upgrade strategy confirmation, vehicle selection, vehicle VIN number confirmation, etc.

6. Automated software testing system (host computer)

Dongzhou's self-developed automated testing system controls the entire testing process and realizes the management and execution of the testing process.

Advantages of Dongzhou OTA test solution

1. The OTA automated test system has a modular design and has various system forms. It can adapt to different automated test solutions according to user needs. Currently, there are real vehicle OTA projects, laboratory UI interaction projects and complete vehicle test OTA solutions.

2. Graphical use case writing, the writing process is simple and highly readable, and you can get started quickly without any programming knowledge.

3. Use a manipulator to simulate the human-computer interaction process, restore the interaction process between the user and the device under test to the greatest extent, and ensure the accuracy of the test.

4. Self-developed image algorithm is not affected by light and angle, ensuring the accuracy and stability of the image assertion method.

5. The automated software testing system distributes test tasks according to customer needs, automatically executes the testing process, and realizes complete automation of backend management, upgrade process, and post-upgrade testing of each module, greatly shortening the testing cycle.