Scheduling sporadic events in real-time systems
By Lonnie VanZandt
Consultant
E-mail: lonniev@ieee.org
Most real-time operating sys-
tems (RTOSes) use a priority-
based, preemptive scheduler to
share a single CPU among mul-
tiple threads. Such a scheduler
is optimal for systems composed
entirely of constant duration
periodic threads, all of which
experience only bounded exter-
nal stimuli. However, reality is
far from the ideal: software sys-
tems are routinely subjected to
unbounded external stimuli and
are required to perform time-
varying computations. In these
situations, events can drive sys-
tems built with typical RTOS
schedulers into overload condi-
tions that fail to satisfy critical
deadlines.
The design team for one of
my previous projects spent sev-
eral months specifying the soft-
ware architecture for a system
controller in a digital switch.
Since a telecoms switch must
meet demanding time con-
straints, we selected an OS that
claimed to be ideal for real-time
applications and then turned
the system and its OS over to
the application engineers.
A few months later, engi-
neers on the product’s quality
assurance team began report-
ing intermittent system dead-
locks. During these lockups,
the user interface did not re-
spond to keyboard activity and
the processor activity LED re-
mained lit 100 percent of the
time. Yet a firmware-driven
sanity lamp was f lashing its
heartbeat, as if nothing were
wrong at all.
Since the processor LED in-
dicated continuous kernel ac-
tivity and the highest priority
heartbeat interrupt ser vice
routine was running, we in-
ferred that the deadlock was
due to a medium priority inter-
rupt service routine caught in
an infinite loop. And since the
fault was intermittent, finding
the cause was particularly chal-
lenging. At first, the problem
occurred on only one system;
soon thereafter, it surfaced on
another. Eventually, we discov-
ered a short-circuiting trace, on
a board quality assurance engi-
neers were shuffling between
systems, which was raising a
f lash-memor y controller’s
level-sensitive interrupt re-
quest signal. Although the sys-
tem executed the high-priority
heartbeat thread, it was unable
to respond to low-priority key-
board I/O because it was con-
stantly servicing the medium-
priority flash controller inter-
rupt thread.
Over the years, we encoun-
tered similar deadlocks: net-
work packet storms overload-
ing network interface control-
lers, unbounded and bursty
character I/O overloading se-
rial controllers and SCSI bus
jabbering overloading disk
controllers. We wondered why
external events could easily
compromise our system. It
turned out that reality simply
failed to conform to our simpli-
fying assumptions: our threads
were not strictly periodic, and
we had failed to architect our
system to manage jitter and
overload. We learned that a
RTOS is only the beginning of
a solution.
Prioritization
Static-priority preemptive
schedulers require that an engi-
neer assign unique priorities to
individual threads to inform the
r(t)
Command
A/D
y[k]
y(t)
Sensor
Figure 1: This is a simple SISO control system. More complex control systems manage
multiple plants at multiple rates and are MIMO controllers.
scheduler which thread to run
when multiple threads are ready.
There are several formal meth-
ods—and of course, numerous
ad-hoc methods—for determin-
ing the relative priority of spe-
cific threads. Methods that relate
a thread’s priority to its time con-
straints outperform those that
base priorities solely on subjec-
tive measures such as the impor-
tance of a thread’s role.
8
Execution budget
0
4 3
Figure 2: A sporadically scheduled thread that begins with an execution budget of eight
units and a 12-unit replenishment period.
Two optimal priority as-
signment methods are the
deadline monotonic (DM) and
rate monotonic (RM) algo-
rithms. These assign increas-
ing priority to threads with
shorter deadlines and periods,
respectively. The optimality of
these methods is such that if
one can feasibly schedule a sys-
tem with any static-priority al-
gorithm then one can schedule
it using a DM or an RM algo-
rithm. Furthermore, if these
optimal algorithms cannot
feasibly schedule the threads of
a particular application, then
no static-priority assignment
can.
A/D
r[k]
Control-law
computation D/A
u[k]
u(t)
Plant Actuator
Although static-priority pre-
emptive scheduling cannot
schedule all possible systems,
designers frequently prefer it
and commercial RTOSes pro-
vide it because it offers greater
stability and predictability than
dynamic priority scheduling.
Static-priority preemptive
schedulers are also relatively
easy to implement and impose
minimal OS overhead.
+4
+3
+1
Replenishment period
1
Digital control
Digital control systems are ideal
applications for static-priority
preemptive scheduling. These
systems are pervasive, appear-
ing in navigation devices, home
appliances, industrial manufac-
turing, chemical processing and
automobiles.
Figure 1 diagrams a simple
single-input/single-output
(SISO) control system. More
complex control systems man-
age multiple plants at multiple
rates and are multiple-input/
multiple-output (MIMO)
controllers.
Using a standard RTOS, a
programmer can easily imple-
ment a MIMO controller with
three parallel closed-loop con-
trol threads running at differ-
ent rates to control three dis-
tinct plants. Since they have a
well-defined, restricted, strict-
ly periodic thread set, such con-
trol systems conform to the as-
sumptions of classical RM
analysis and are ideal applica-
tions for static-priority preemp-
tive scheduling.
However, many practical
systems employed in real-time
scenarios also experience ape-
riodic or sporadic external
events. Such systems include
network and telecoms equip-
ment, multimedia players,
medical devices and industrial
process control systems; or,
more generally, any system that
connects to an environment
with potentially misbehaving
agents or agents acting at ran-
dom intervals. For example, a
plant manager can at any un-
foreseen moment switch an in-
dustrial process from produc-
tion mode to safety mode. The
threads that transition the sys-
tem between these two modes
are sporadic threads.
Likewise, many practical ap-
plications contain threads with
stochastic computation inter-
vals such as a data processor
that performs different compu-
tations depending on the infor-
mation within the data rather
than a single computation that
is dependent only on the quan-
tity of data.
Overload
A system goes into overload
when events’ interarrival times
are shorter than planned or
when threads’ execution inter-
vals are longer than planned.
Under these conditions, the sys-
tem must discard events or allow
one or more threads to miss their
deadlines to allow more critical
threads to complete on time.
Overload results when pe-
ripherals and external agents
conform to their true nature
rather than to designers’ ideal-
ized assumptions. Hardware de-
vices may fail, generating spuri-
ous signals greatly out of line
with proper behavior. Errant or
malicious code, improperly
trained or malicious users and
unusual number of external
agents can generate excessive
event traffic that violates de-
signers’ statistical assumptions.
Several practical examples
include Mother’s Day callers,
who are notorious for overload-
ing the phone network; net-
work packet broadcast storms;
focused denial of service at-
tacks; large files fed at high
speed into serial ports; mal-
functioning disk controllers;
faulty sensors; electromagnetic
disturbances; errant, dynami-
cally loaded device driver code
that forgets to clear interrupt
conditions; and the stuck active
interrupt request signals my
team encountered.
The choice of scheduling
policy makes a dramatic differ-
ence on how the system behaves
in overload, and determines
whether a designer can predict-
ably characterize the system
during overload. For example,
the dynamic priority earliest
deadline first (EDF) scheduling
policy performs poorly in over-
load. Since EDF is used to as-
sign thread priorities on the fly,
the designers of the system can-
not predict which specific
LISTING 1 Sporadic scheduling example
#define NIC_SPORADIC_BKGD_PRIORITY 5 // just above dirt
#define NIC_SPORADIC_PRIORITY 21 // above most other I/O
#define NIC_SPORADIC_MAXREPLENISHMENTS 3 // blocked only 3 times at most
#define NIC_SPORADIC_PERIOD 1024000 // 1024 microsecs
#define NIC_SPORADIC_BUDGET 400000 // 400 microsecs
// initialize an attributes structure and configure it for sporadic scheduling
if (pthread_attr_init(&pattr) != EOK) {...}
// override the default of inheriting policies from our parent
if (pthread_attr_setinheritsched(&pattr, PTHREAD_EXPLICIT_SCHED) != EOK) {...}
// configure the normal and background priorities, the budget interval, and
// the maximum number of replenishments
param.sched_ss_low_priority = NIC_SPORADIC_BKGD_PRIORITY;
param.sched_priority = NIC_SPORADIC_PRIORITY;
param.sched_ss_max_repl = NIC_SPORADIC_MAXREPLENISHMENTS;
nsec2timespec(&param.sched_ss_repl_period, NIC_SPORADIC_PERIOD);
nsec2timespec(&param.sched_ss_init_budget, NIC_SPORADIC_BUDGET);
if (pthread_attr_setschedparam(&pattr, &param) != EOK) {...}
// now set the scheduling policy to sporadic
if (pthread_attr_setschedpolicy(&pattr, SCHED_SPORADIC) != EOK) {...}
// create the thread/task and have it explicitly use sporadic parameters
if (pthread_create(&devData->threadId, &pattr, rtl_EventHandler, devData) != EOK) {...}
Listing 1: The POSIX C code used to configure a sporadic scheduling policy for the packet reception thread of the network driver is found here.
threads will miss their dead- suite describes a programming If a thread ever requires more
lines during overload condi- interface for portable operating processor time than its budget
tions. Furthermore, an over- systems. In 1999, the POSIX allows, the sporadic scheduling
running thread may cause Working Group introduced policy dynamically lowers the
other future threads to miss IEEE 1003.1d to specify addi- priority of the thread to its
their deadlines. tional real-time extensions for “background” priority. While
On the other hand, a designer the C language. Section 13.2.4 at either priority, the scheduler
using a static-priority policy can of that standard adds the spo- treats the thread identically to
perform schedulability analysis radic scheduler policy to the other threads with the conven-
to assure that the deadlines of round robin and FIFO policies tional FIFO scheduling policy.
critical threads will be satisfied previously standardized. The For example, priority inherit-
even in overload. Employing a POSIX Working Group speci- ance and positioning within
predictable scheduling policy is fied the sporadic scheduling ready queues remain the same—
especially important for safety- policy precisely for handling the important differences are
critical systems, which must re- aperiodic and sporadic events the enforcement of the thread’s
main predictable even in over- and threads running within the execution time and priority re-
load. For example, an overloaded context of a static-priority pre- assignment of the thread.
system that misses a few user in- emptive scheduler. By lowering the thread’s pri-
terface updates is preferable to In addition to the single, ority, the system allows other
one that fails to close control normal-priority level used for threads to execute that the spo-
valves. FIFO and round-robin schedul- radic thread would otherwise
While poorly designed sys- ing, the parameters of a spo- preempt at its normal priority.
tems often fail to satisfy the radic policy include a second, Dynamic adjustment of thread
constraints of critical and less lower background priority; a priorities is the key capability of
important threads in overload, replenishment interval; an ex- the policy that enables pro-
well-designed systems must se- ecution budget; and a maxi- grammers to build predictable
lectively process events to as- mum number of replenish- systems.
sure that the deadlines of criti- ments allowed within each re- A sporadic thread consumes
cal threads will always be met. plenishment inter val. To- execution time whenever it is
POSIX’s sporadic scheduling gether, the replenishment in- running. The scheduler de-
policy makes good design easy. terval and execution budget de- ducts consumed time from the
fine the maximum processor thread’s budget each time the
Sporadic scheduling utilization granted to a thread thread blocks or the scheduler
IEEE’s POSIX specification at its “normal” priority. preempts it.
 When a sporadic thread is
blocked (but not preempted),
the sporadic scheduler allocates
a new replenishment event one
replenishment period after the
most recent unblocking of the
thread. The scheduler sets the
amount of execution budget that
it will grant to the thread at that
replenishment to the amount of
time the thread ran since it un-
blocked.
A thread may contend for
multiple resources with lower
priority threads during a single
period and, therefore, might
have several replenishment
events outstanding. To mini-
mize scheduler overhead, the
POSIX specification enables a
Parameter
NIC interrupt request interarrival
Thread computation requirement
Effective maximum utilization
Table 1: For these requirements, we can create a periodic thread to handle packet reception and assign it a rate monotonically derived
priority based on its 1,024ms period.
designer to limit the number of
outstanding replenishment
events per period. If the maxi-
mum is reached, no additional
replenishment events are
scheduled within the replenish-
ment period.
If and when a sporadic
thread consumes its entire ex-
ecution budget, the scheduler
is informed so it can lower the
thread’s priority to its back-
ground level. The thread re-
mains at this priority until a re-
plenishment event occurs, at
which point the scheduler pro-
motes the thread back to its nor-
mal priority. If the thread is
able to run at its background
priority because no higher pri-
ority thread is ready to run,
then the scheduler allows the
sporadic thread to run and does
not deduct time from its execu-
tion budget.
These consumption and re-
plenishment policies assure that
within any interval of time equal
to the thread’s replenishment
interval, the thread is allowed to
execute only for as long as its
computation budget indicates.
The scheduler thereby forces the
thread’s processor utilization to
conform to the designers’ expec-
tations and to the assumptions
of optimal RM static-priority
preemptive scheduling. This
utilization enforcement is what
assures predictability of the sys-
tem even when in overload.
To clarify these concepts,
Figure 2 shows a sporadically
scheduled thread that begins
with an execution budget of
eight units and a 12-unit replen-
ishment period. Once released,
the thread runs for four units
before blocking. The scheduler
deducts four units from the
thread’s budget and schedules a
replenishment of four units to
occur 12 units after the thread
started. Unblocked, the thread
runs for three units before block-
ing a second time. The sched-
uler deducts three units from
the budget and schedules a
3-unit replenishment to occur
12 units after the thread un-
blocked. Unblocked the second
time, the thread executes for
one unit, whereupon the sched-
uler detects the exhaustion of
the execution budget and down-
grades the thread’s priority.
The thread is free to run at the
background priority if no
other higher priority threads
preempt it.
Note that the resolution of
the operating system’s timer
tick limits the accuracy of the
sporadic scheduler. A low-
resolution 10ms tick, wouldn’t
yield good results for threads
with periods and budgets mea-
sured in milliseconds.
An example
Let’s see how we can put a spo-
radic scheduling policy into
practice by examining the re-
ception thread of a network in-
terface controller’s (NIC) de-
vice driver.
Applications communicate
over packet networks via device
drivers attached to NICs. These
controllers implement the
physical and data link layers of
the physical medium’s commu-
nication protocol to provide
node-to-node connections
without errors, to arbitrate for
the medium, and to handle the
physical and electrical specifi-
cations necessary to transmit
data across the medium. A net-
work device driver contains at
least one thread for receiving
packets as part of interrupt ser-
vice, and another for transmit-
ting packets.
Designers make several as-
sumptions when employing a
multi-node communications
medium into their designs, in-
cluding expectations on the uti-
lization of the medium, the
bandwidth of the medium, the
average and worst-case packet
sizes and the expected interval
between packets destined for
the connecting system. With
these assumptions, the design-
ers determine an appropriate
period, computation interval
and priority for the packet re-
ception thread.
Duration
1,024µs
400µs
39 percent
My system needed to con-
nect to a 100Mbps network,
would receive constant sized
(64byte) packets, required
25ms of processing per packet,
and expected to receive packets
once every 64ms. I selected an
NIC that interrupted the host
processor for each packet re-
ceived, yet also included a
32-entry ring buffer. I planned
to protect against packet drops
by never letting the ring buffer
become more than 50 percent
full. Processing all packets
within the time constraints was
a goal, but never overloading
the system was the require-
ment. To ensure this, I allowed
the system to drop packets if
absolutely necessary.
Given the requirements in
Table 1, I created a periodic
thread to handle packet recep-
tion and assigned it a rate
monotonically derived priority
based on its 1,024ms period. To
keep the ring buffer utilization
under 50 percent the period is
16 x 64µs and the computation
interval is 16 x 25µs. (I also
used an OS that uses extremely
short interrupt exception han-
dlers and performs interrupt
service handling within a pri-
oritized thread.)
I realized that if the incom-
ing rate exceeded 15,625 pack-
ets/s, the system would go into
overload. To protect the sys-
tem against this, I switched the
reception thread’s default
FIFO policy to a sporadic
scheduling policy and so re-
stricted its high priority utili-
zation of the processor to the
desired worst case amount,
39 percent. I accomplished
this by initializing the fields of
a POSIX scheduling param-
eters attribute to my expected
period and execution budget
and passing those attributes at
POSIX’s thread creation.
The sporadic scheduling
policy ensured that the system
could withstand abnormal
packet bursts and would drop
excessive incoming traffic, but
would still satisfy the time con-
straints of critical threads lower
in priority than the NIC recep-
tion handler.
Listing 1 presents the
POSIX C code to configure a
sporadic scheduling policy for
the packet reception thread of
the network driver. A complete
implementation of a Realtek
NIC device driver for the QNX
OS is available for download
from www.embedded.com/
code.htm.
Predictably, sporadic sched-
uling manages aperiodic and
sporadic events and threads by
wrapping them within a peri-
odic framework. This is an ef-
fective technique for predict-
ably handling overload sce-
narios on static-priority pre-
emptive schedulers. Using spo-
radic scheduling, threads that
would otherwise introduce un-
desirable jitter or deny the pro-
cessor to lower-priority threads
during overload can be safely
included. With sporadic sched-
uling, we can predictably ana-
lyze unpredictable systems.
If you are interested in incor-
porating sporadic scheduling
into your applications, you can
find implementations of the
policy within the Ada95 pro-
gramming language, the QNX
RTOS, Java Virtual Machines
compliant with the Real-Time
Specification for Java (RTSJ),
and several real-time variants of
the Linux kernel. Note, how-
ever, that not all implementa-
tions are fully compliant with
the POSIX 1003.1d specifica-
tion. For example, RTSJ’s spo-
radic scheduling policy neither
uses the C bindings nor imple-
ments replenishments. Ada95’s
implementation uses POSIX’s
1003.5 Ada bindings.
[Embedded Systems Programming]