Embedded Software Development Tracing and Visualization of Embedded Linux Systems

Tracing is an important method to detect program errors in embedded systems. Visualization tools lend support to a better understanding of the tracing results.

Anbieter zum Thema

MicroConsult Microelectronics Consulting & Training GmbH

Axivion GmbH

Apacer Technology B.V.

Embedded Tools GmbH

Debugging embedded software can be a challenging, time-consuming and unpredictable factor in development of embedded systems. Detecting errant program execution begs the question “How did the software reach this state?” What combination of inputs and timing resulted in the error, and why?

Tracing can often provide the answer.

Tracing entails recording software behaviour during runtime, allowing for later analysis of collected trace data. Tracing is most often a development bench activity, but tracing can also be enabled for production use, continuously active to record behaviours and catch errors post-deployment. Production tracing can be an effective technique for detecting rarely-manifested errors that are therefore are difficult to reproduce in a debugger. These can include situations where the system responds more slowly than expected, gives incorrect or suboptimal output, freezes up or crashes.

Tracing can be performed either in hardware (in the processor) or in software. Hardware-based tracing generates a detailed instruction-level execution history, while software-based tracing focuses on selected events, typically in the operating system and important application-level interfaces.

Hardware-generated trace provides details regarding control-flow and does not impact the execution of the traced system, but does require special equipment and a trace-enabled hardware platform.

Software-generated trace does not require any special hardware and can even be deployed in shipped products like a “black box” flight recorder used in aviation. Moreover, software trace allows for storing data, such as system call parameters, while hardware trace is typically limited to control-flow only. Software tracing does induce some CPU overhead, but typically less than 1 %. Software tracing relies on target system RAM for storing the trace data, at least initially, but the RAM buffers are usually configurable to allow for balancing RAM usage vs. execution history length.

Tracing is especially important for systems integrating an operating system. A central feature of OSes is multi-threading - the ability to run multiple programs (threads) on a single processor core by rapidly switching amongst execution contexts. Multi-threading, however, makes software behaviour more complex, and affords the developer less control over run-time behaviour as execution is pre-empted by the OS.

Tracing in Linux using LTTng

LTTng is an open source technology for software-based tracing in Linux. LTTng is very efficient, proven in use and supported by most Linux distributions. The LTTng kernel tracer can record scheduling events, system calls, IRQs, memory management, and other kernel-level activities. LTTng user-space tracer (LTTng-UST) tracks user-defined events in application code.

Although LTTng is based on software instrumentation, it does not require a rebuild of the Linux kernel. Rather, LTTng leverages tracepoints which already exist at strategic locations throughout the kernel. These tracepoints are placeholder function calls, dormant by default but activated by LTTng.

Using LTTng-UST, developers can trace custom events in user-space application and library code. It is even possible to trace selected function calls without modifying source code, by creating a shared object file with wrapper functions containing tracepoints, specified in LD_PRELOAD when launching the application.

LTTng-UST function wrapping is completely transparent to application code, with no need for recompilation. On the first call, the wrapper function looks up the address of the original function and stores it in a function pointer (as with malloc in the picture above)

Visualization of LTTng traces in Tracealyzer

LTTng outputs binary data files, so a tool is required for analysis. Babeltrace can convert the trace data to text files, but it is hard to “see the big picture” from vast amounts of trace data in text format. A visualization tool greatly facilitates analysis since the human brain is much better at spotting patterns in images than in text data.

Tracealyzer is a family of trace visualization tools developed by Percepio AB in Sweden. Tracealyzer for Linux is designed to visualize LTTng trace data and provide multiple graphical perspectives to facilitate analysis. Tracealyzer understands and highlights dependencies among related events in trace data, for instance sending and receiving of a semaphore signal. Highlighting such dependencies makes it easier to understand operating system behaviour, e.g., why some threads are blocked and others triggered.

The main trace view in Tracealyzer (see the main picture on the previous page) presents events (e.g., system calls) along a vertical time-line using colour-coded labels. Labels can be filtered in several ways and automatically adjust to avoid overlapping. Label background colour indicates status and type of operation, e.g., red labels show system calls that block the calling thread. Custom application events from LTTng-UST can be configured to appear either as service calls (e.g., malloc) or as user events (generic debug messages).

The Tracealyzer main view is complemented by 20+ additional views, showing CPU load, response times, kernel blocking, scheduling intensity, kernel calls and other traced events. User events can be shown separately in a text log view, and user event arguments can be visualized as signal plots. All views are interconnected, meaning that clicking on a task, event or data-point will open another relevant view with the clicked entity highlighted. This interconnection makes it easier to use multiple perspectives when studying a specific interval in the trace.

Summary

Tracing provides a powerful tool for analysing multi-threaded software systems. On Linux, tracing is enabled by LTTng, a mature and proven open source solution. Percepio Tracealyzer for Linux lets developers visualize LTTng trace data through multiple, interconnected graphical views.

Tracealyzer makes dense and voluminous trace data more accessible to software developers, giving them greater benefit from tracing. Tracealyzer helps developers make sense of complex trace data, find bugs and tune performance, and thereby produce better software.

* * Johan Kraft is the founder and CEO of the Swedish software company Percepio which specializes in the visualization of tracing results.

(ID:42726408)