Debugging Techniques for Embedded Systems using Real-Time

Software Trace1
Anthony Berent ARM2
Abstract
Until a few years ago real-time software trace was only available using a Logic Analyzer or In Circuit
Emulators. These tools were expensive and difficult to use, and as such were often only used for software
debugging when everything else has failed. Also, as processors became more complex and more deeply
embedded, it became less and less practical to make the signals required for such debugging accessible
to the tools.
In the last few years, however, processors, such as ARMs cores, have included real-time trace facilities.
The easy availability of such real-time trace facilities allows new debugging paradigms to be developed.
In this paper I describe some of these techniques for debugging embedded systems. I also show that
real-time trace can have advantages even for debugging systems that do not have hard real-time
requirements.

Introduction
Most software engineers spend a significant portion of their time looking for bugs in
software. Traditionally this is done using a start/stop debugger. Using such a debugger
the engineer can define breakpoints at which the system is to stop, and then examine
the state of the system when the program hits a breakpoint. The engineer can, from
this, see how the software is failing.
Unfortunately such techniques are almost useless for real-time systems, since stopping
the system changes its real-time behavior. This causes the problems one is trying to
debug to disappear as soon as a breakpoint is introduced. An alternative technique is
needed. One such technique is real-time software trace, in which the some hardware
captures the sequence of instructions, and possibly data accesses, executed by the
processor. The engineer then uses this historical record of the behavior of the processor
to investigate the bug.
1

ARM Ltd. 2002

Anthony Berent is a Principal Engineer at ARM Ltd, and is the lead engineer on ARM's Trace Debug
Tools. He has over 20 years experience as a software engineer, mainly working with embedded systems,
and has spent most of the last 10 years developing debugging tools and techniques for embedded
systems.

Debugging Techniques for Embedded Systems using Real-Time Software Trace

Page 1

Until a few years ago real-time software trace was only available using a Logic Analyzer
or In Circuit Emulators. These tools were expensive and difficult to use, and as such
were often only used for software debugging when everything else has failed. Also, as
processors became more complex and more deeply embedded, it became less and less
practical to make the signals required for such debugging accessible to the tools.
In the last few years, however, processors, such as ARMs cores, have included realtime trace facilities. The easy availability of such real-time trace facilities allows new
debugging paradigms to be developed. In this paper I describe some of these
techniques for debugging embedded systems. I also show that real-time trace can have
advantages even for debugging systems that do not have hard real-time requirements.
Why real-time trace is needed
In a traditional start/stop debugging environment the processor, or the application, has
to be stopped to find out anything about the state the system. There are two major
problems with this in real-time systems:
1. It may be impossible to safely stop the system. For example, in a disk drive, the
head may keep moving until told to stop by the processor. If the processor is
stopped, so the head is not told to stop in time, the disk will crash. This is an
expensive debugging technique.
2. Even if it safe to stop the processor doing so will change the state of the system. In
particular, in real-time systems many bugs may depend on the timing of the software
relative to external events (e.g. clock interrupts). Stopping the processor will change
the relative timing of events, hence causing the bugs to disappear or to
metamorphose into different problems.
The effect of this is that it is often very difficult to debug real-time embedded systems.
This can add considerably to the time taken to complete the development of such
systems, and as such adds considerably to the cost of developing such systems.
How real-time trace works
Real-time trace captures a trace of the instructions executed by a processor, running in
real-time, and stores these instructions in a buffer for later analysis. In addition to the
instructions executed the data used by those instructions may be captured. It is typically
possible to select which instructions or data is captured.
It is also normally possible to select a trigger condition, such as execution of a particular
instruction, or writing a particular value to a particular location in memory. Much more
complex trigger conditions are also sometimes available. When the trigger condition
occurs the buffer stops capturing the trace data, either immediately or some time later,
hence ensuring that the buffer retains a trace of the systems behavior around the time
the trigger condition occurred.

Debugging Techniques for Embedded Systems using Real-Time Software Trace

Page 2

Traditional real-time trace

Traditionally, to obtain real-time trace, one has needed a logic analyzer. The logic
analyzer is connected to the processors address and data busses, and traces all
accesses to memory (both instruction fetches, and data accesses). This is then used to
reconstruct the sequence of instructions executed by the processor.

Trace output
Logic Analyser

Processor

Address bus

Other
devices

Data bus

Target System

Figure 1 Traditional real-time trace

While this approach can work well it has a number of problems:
Cost. Logic analyzers are typically quite expensive, and it is not normally practical to
provide them to every software engineer.
Access to address and data busses. In modern systems the processor will often be
embedded in a chip with many other functions. The processors address and data
busses may never leave the chip. In these circumstances it is not possible to
connect a logic analyzer to the processors busses.
Complex processors. In a modern processor, with caches, an instruction pre-fetch
pipeline, and possibly out of order execution, the order in which instructions are
fetched may not reflect the order in which they are executed. It is, in fact, very
possible for a processor to execute a significant number of instructions (say in a
loop) without accessing external memory at all. As processors become more
complex it becomes progressively more difficult to reconstruct the instruction
sequence from watching the bus.

Debugging Techniques for Embedded Systems using Real-Time Software Trace

Page 3

Embedded real-time trace

Many modern processors, including most ARM processors, now include an embedded
trace macrocell (ETM). The ETM captures the sequence of instructions and data
accesses and send a record of them to a buffer, which may be either on or off chip.
ARMs ETM includes features to allow triggers to be set, and to allow the trace to be
filtered as it is captured. It also compresses the data so that the buffer memory is used
efficiently, and the bandwidth required to the buffer is small.

Microcontroller
ARM CPU Macrocell
Control

Trace Debug
Tools running
on host

BREAKPT

Multi-ICE

5 wire
JTAG

JTAG
Port

Address

Execution
Unit

ETM

Data

EmbeddedICE

TAP
Trace
Port

MultiTrace

Figure 2 ARM's Embedded Trace solution

The advantages of this approach are:

Cost. If the chip has an on chip trace buffer then trace can be used with no
additional hardware beyond that used for normal debugging. Even if the chip does
not have an on-chip buffer the cost of an external trace buffer is considerably less
than the cost of a full logic analyzer.
Access to the processor. Since the trace is captured within the chip the ETM can
easily access the processor.

Debugging Techniques for Embedded Systems using Real-Time Software Trace

Page 4

Complex processors. The ETM always monitors the data and address busses
between the processor and the cache. This means that caches are not a problem.
ARMs more complex processors contain private interfaces to the ETM that provide
the ETM with all the information it needs to fully trace the processor.

While this does add something to the cost of the processors, in terms of silicon area, the
incremental cost typically is small. For ARM processors the ETM is optional, but we
have been finding that most of our partners are now including it in their designs.

Other ways of debugging real-time systems.

Simulation/modeling This is a useful method, particularly in the earlier stages of
developing the software. It allows full use of traditional start/stop debugging on a
software model of the system. It can, however, be slow, and it is often difficult to
create a full simulation of the system.
Real-time debug monitor running on target. A real-time debug monitor is a piece of
software running as part of the target code that allows a debugger to communicate
with the system, and to debug its non time critical aspects, while allowing its time
critical aspects (e.g. critical interrupts) to keep running. This works well if the
problems are in the non time critical parts, but is useless for debugging time critical
code.
Using real-time trace to debug real-time systems
The fundamental technique for using trace is not very different from using breakpoints
for debugging. To debug a problem:
Find a point in the program, or a condition, that you expect will give you some
information about the bug.
Set a trigger on that point, and allow the program to run (if not already running).
When the trigger is reached the trace output is generated. Examine the trace output
to see what it tells you about the bug.
Repeat until the bug is understood.
There are, however, some major differences.
Trace gives information over a period. Even simple instruction trace will tell you how
the trigger point was reached. In effect, with real-time trace, you can look backwards
as well as forwards.
Because real-time trace is non-intrusive it can give accurate timing information.
Using this timing information you can investigate timing bugs that would be difficult
to investigate any other way.
You have to chose in advance what information to collect. In a start/stop debugging
system, once you reach a breakpoint you can look at the state of any part of the
system. Trace systems have limited capacity, and since the system does not stop at
the trigger point, you cannot get additional information about the system after
collecting the trace.

Debugging Techniques for Embedded Systems using Real-Time Software Trace

Page 5

The fact that real-time trace is not intrusive means that it is possible to debug parts of
the system that are typically not debuggable using breakpoints. For example, one can
usefully set trace triggers on interrupt routines, and hence, for example, discover the
timing of interrupt routines, and how interrupt routines are interacting with one another.
Figure 3 shows an example of trace output. The trigger point is defined to occur when a
value is stored to x, and occurs on index 0. As we can see the trace allows the user to
see how this point was reached, which is not something that is available using start/stop
debugging. The trace buffer will contain hundreds of thousands or millions of
instructions leading up to the trace point, giving extensive information about how the
trigger point was reached.
By filtering the trace to only capture selected instructions one can extend the time
period over which trace is captured to seconds or longer. Using the filtering facilities of
ARMs ETMs one can, for example, capture only the code executed when handling a
particular interrupt, or exclude all code in a particular source file.

Figure 3 Example of trace output - cycle 0 is the trigger cycle

Debugging Techniques for Embedded Systems using Real-Time Software Trace

Page 6

Advantages of real-time trace when debugging non-real-time systems

As we saw earlier, one of the advantages of real-time trace is that it can give
information about how a particular point in the code, or state, was reached. This makes
it particularly valuable for solving bugs that are either intermittent, or where it is not
obvious how the program can reach its final, faulty, state. This is true even when the
system is not a real-time system, or the bug is not related to its real-time aspects. I,
when I am debugging, often find myself asking How on earth did the program reach
that line?. With traditional start/stop debugging often all one can do is step through the
code until the program reaches the line in question. With real-time trace I can set a
trigger point on that line of code, and then work backwards through the trace to work out
how the program got there.
An even worse, but similar, example is data corruption. For example, if a global
variable, or an element of a data structure is corrupted it is often very difficult to work
out how it is being overwritten. Even once one has found the code that is writing to the
location all this typically tells one is that a pointer is incorrect. Using trace one can set a
trigger on writes to the corrupted location, and then work backwards to find out why that
location is being written to.
Another example where trace is valuable is in debugging problems that cause
processor exceptions (e.g. data aborts). Once again the ability to work backwards from
the trigger is the critical factor here. Using trace one can trigger on the execution of
exception vector instructions, and then work backwards through the trace to determine
how the exception was caused.

Debugging Techniques for Embedded Systems using Real-Time Software Trace

Page 7

Using real-time trace for performance analysis

A further use of real-time trace is for performance analysis. Traditional ways of
analyzing the performance of software include adding in special function calls to record
the time at particular points in the code (instrumenting the code), or using a timer
interrupt to sample its state at particular points. Both these techniques can give good
approximate measures, but because they require running additional software on the
target they, in themselves, change the behavior, and hence the performance, of the
target system. Using real-time trace one can get a complete trace of the instructions
executed, together with cycle counts for each instruction. One can then analyze this
information to give a detailed picture of the behavior of the system over the period in
which the trace was collected. Since real-time trace does not in any way modify the
behavior of the system, this information is accurate.

3%

1%
2% 1%1%

11%

Func_1

3%

Proc_8

4%

Proc_7
_memcpy_small
11%

5%

_strcmp_loop
Func_3
_strcmp_return

5%

_memcpy_aligned_loop
Proc_2
Proc_4

5%

11%

__rt_sdiv
Proc_3
Func_2

6%

Proc_1
Proc_5
9%
8%

strcmp
Proc_6

8%

8%

__rt_memcpy_w

Figure 4 Example performance analysis data from real-time trace

Debugging Techniques for Embedded Systems using Real-Time Software Trace

Page 8

Conclusion
The use of embedded real-time trace, as provided by ARMs ETMs, provides new ways
of debugging real-time and non-real-time embedded systems. In many cases the
availability of real-time trace allows the easy analysis of bugs that would otherwise be
difficult or impossible to analyze. As such using processors that include embedded realtime trace can significantly reduce the time taken to debug the resulting systems, and
hence produce significant cost savings.
In addition real-time trace is one of the few techniques that gives truly accurate
performance profiling information. This allows performance bottlenecks to be found and
resolved, and hence eases the development of performance critical software.
ARM, ARM Powered, StrongARM, Thumb, Multi-ICE, Integrator, PrimeCell and ARM7TDMI are registered trademarks of ARM
Limited. ARM7TDMI-S, ARM7EJ, ARM720T, ARM740T, ARM9TDMI, ARM920T, ARM922T, ARM940T, ARM9E, ARM926EJ-S,
ARM946E-S, ARM966E-S, ARM1020E, ARM1022E, EmbeddedICE, EmbeddedICE-RT, AMBA, MultiTrace, ModelGen, ARM
Developer Suite, RealView, ETM, ETM7, ETM9, ETM10, Embedded Trace Macrocell, Jazelle, PrimeXsys, MOVE and JTEK are
trademarks of ARM Limited. CodeWarrior is a registered trademark of Metrowerks Corporation. All other brand names or product
names are the property of their respective holders. "ARM" is used to represent ARM holdings plc (LSE: ARM and NASDAQ:
ARMHY); its operating company ARM Limited and the regional subsidiaries ARM, INC.; ARM KK; ARM Korea Ltd. Neither the whole
nor any part of the information contained in, or the product described in, this document may be adapted or reproduced in any
material form except with the prior written permission of the copyright holder. The product described in this document is subject to
continuous developments and improvements. All particulars of the product and its use contained in this document are given by ARM
in good faith. All warranties implied or expressed, including but not limited to implied warranties of satisfactory quality or fitness for
purpose are excluded. This document is intended only to provide information to the reader about the product. To the extent
permitted by local laws ARM shall not be liable for any loss or damage arising from the use of any information in this document or
any error or omission in such information.

Debugging Techniques for Embedded Systems using Real-Time Software Trace

Page 9