How Do Vendors of Software Debugging and Test Tools Test Their Products?

Anbieter zum Thema

MicroConsult Academy GmbH

ISH Ingenieursozietät GmbH

Apacer Technology B.V.

Test Automation @ iSYSTEM

fitIDEA is an automated test tool that was initially developed for iSYSTEM internal purposes. It uses a generic API (application programming interface) of the iSYSTEM development environment named winIDEA (isystem.connect) and the public scripting language Python.

As a tool manufacturer iSYSTEM faces almost the same challenges as their customers: Lots of different hardware variants, historically grown code base plus legacy code, and extended functionality according to higher requirements of customers, increase the complexity of development and test tools as well and makes it evident to introduce test automation.

All critical aspects of the embedded software debugger and integrated unit test tool suite are tested. Amongst others, the following functions are checked:

• Standard debugging and IDE functions like download to flash memory, target system memory access (read, write, real-time access), breakpoint functions (execution and access)

• Specific time measurements to test advanced debugging, like trace and profiler functionality

• Software test with code coverage and unit test

• API functionality

The fitIDEA concept is based on the deterministic behavior of the test environment comprising the following components:

• Reference system (standard board of the semiconductor manufacturer, iSYSTEM board, customer-specific system)

• Reference application for such a board/system

• iSYSTEM debugging hardware (blue box)

• iSYSTEM software (winIDEA and testIDEA on PC side)

• Compiler

The reference application for the respective target system is the key element. It is programmed thus that a deterministic (defined and reproducible) behavior is exhibited:

Example:

The execution of this function obviously increments the variable g_c by one. fitIDEA now carries out the following steps:

• Execute the code until Increment() is requested

• Read the value of variable g_c

• Execute the function through to the end

• Read the value of variable g_c

The test was passed if the second read-out of variable g_c shows that the value was incremented by one. This test covers the following functions:

• Identification of code and data symbols from the download file (set breakpoints to symbolic values, read variable values through symbolic names)

• Interrupt of execution (a breakpoint was set on function entry and exit)

• Execution of code and CPU status verification (run until, wait until CPU stops)

• Read access to memory (read-out of variable values)

• C compiler: code and debug information was generated

Even if a component of the test environment changes (iSYSTEM tool, compiler, target, ...), the application has to maintain its behavior, and fitIDEA, as the test tool, still has to deliver the same result. The Python test scripts are created by dedicated test engineers. The iSYSTEM reference application uses reference commands (//rt) to save expected results of a test. These references are stored in the source code of the references application (see image above) and are thus independent of the compiler used.

So far, more than 60 hardware configurations in the tests benches are actively running tests, and each configuration has about 100 test cases. Each developer can access all configurations and can also run tests locally before taking over the modified program code. Automatic testing with fitIDEA is executed overnight, and test results are available the next morning, thus facilitating seamless integration. New test cases are continuously added based on new functionalities, customer applications or resolved support cases.

(ID:38231760)