What is a real-time system?

A system that maintains a continuous and timely interaction with its environment
Cruise-Control System
Brake Accelerator Wheel Rotation Desired Speed Commands

Real-Time Systems

Throttle Value

Manas Saksena University of Pittsburgh

University of Pittsburgh

Manas Saksena

University of Pittsburgh

Manas Saksena

Real-Time Systems
Real-time computer system

corrects depends on when the results are produced (timeliness) in addition to what results are produced

Timing Constraints
Speed Control

sample sensor value (speed) every 100 ms, periodic activity inter-arrival time (period) output actuator command within 60 ms deadline for each arrival

Embedded System

The computer system is part of a larger system Examples: automobile control system(s) air-traffic control
Manas Saksena 3

Brake Press

stop cruise control within 50 ms arrival rate: irregular deadline

Manas Saksena 4

University of Pittsburgh

University of Pittsburgh

Hard vs Soft Real-Time

Hard Real-Time

Missing deadlines means system failure Missing deadlines can cause serious loss Deadlines may be missed occasionally

Value Functions

Safety Critical Soft Real-Time

value Soft real-time Hard real-time

Most real systems have a mix of hard and soft real -time components The distinction between hard and soft real -time is somewhat subjective Soft real-time is not non real-time

deadline

University of Pittsburgh

Manas Saksena

University of Pittsburgh

Manas Saksena

Traditional Real-Time Applications

Avionics Data Acquisition Digital Controllers Defense Systems Industrial Process Control

Emerging Embedded Real-Time Applications

Cell-phones, VoIP phone, PDAs MP3 players Set-top boxes, Game Consoles Automotive Systems Network Boxes On Demand Servers Embedded Devices

University of Pittsburgh

Manas Saksena

University of Pittsburgh

Manas Saksena

Real-time Control Systems Real-Time Software Design Basics

Controller

Reference input

A/D A/D

Control-raw computation

D/A

sensor

Plant

actuator

University of Pittsburgh

Manas Saksena

University of Pittsburgh

Manas Saksena

10

Software Structure: State Transition Diagrams

State Machines
initial state
Initialize Object

state machine state

initialization

external trigger? Wait for Event Handle Event

manual trigger
cruise/startControl()

Take actions ISR: to set/clear events Change system state

automatic
shutdown/

action expression final state

Terminate Object

timeout/doControl()

transition
University of Pittsburgh Manas Saksena 11 University of Pittsburgh Manas Saksena 12

State Transition Diagrams: Design

Schedulability Analysis
Are my timing requirements being met? Real-Time System Model

manual

automatic

A set of events Maximum arrival rate for each event A handler for each event Maximum execution time for each handler A deadline for each event Finish time (e) Arrival time (e) <= Deadline
Response Time

University of Pittsburgh

Manas Saksena

13

University of Pittsburgh

Manas Saksena

14

Software Structure: Cyclic Executives

Real-Time System Software

Concurrency

real-world events occur concurrently multi-tasking simplifies software structure

Replace Interrupts by Polling

More predictable behavior

Static Cyclic Schedule

Repeated at run-time

Timed Events

need timers to trigger timed events build timer support

Implemented by a single loop 

 
 
  !
 #"$
 %
University of Pittsburgh Manas Saksena 15

Timeliness of Response

Different events have different requirements How to structure software to satisfy all the requirements
Manas Saksena 16

University of Pittsburgh

Real-time System Development

Cross-Development for Embedded Systems

Use the host to edit, compile, and build application programs

Target system Application tasks

Development host Compiler, debugger, loader, simulator, shell monitor NT OS (Solaris) Pentium PC station SUN workstation

configure the target

At the target embedded system, use tools to

load, execute, debug, and monitor (performance and timing)

Real-time OS Input output

Iterative process
PC Host + development tools RS-232 (console) application tasks RTOS (vxWorks) Target (MBX860)

Hardware

RS-232 Ethernet

Ethernet (load, debug, monitor)

University of Pittsburgh

Manas Saksena

17

University of Pittsburgh

Manas Saksena

18

Traditional Real-Time OS Requirements

Real-Time Operating Systems and Real-Time Software Design

Multi-Tasking kernel

Fixed priority, preemptive scheduling Semaphores, Message Queues, etc. Priority Inheritance

Inter-task communication Timers Device Drivers Interrupt Management

Do this with low and predictable overheads.

Manas Saksena

University of Pittsburgh

Manas Saksena

19

University of Pittsburgh

Manas Saksena

20

Real-time OS
G Functions: &
task management, scheduling, dispatcher communication (pipe, queue) synchronization (semaphore, event)

Schedulable Entities
Tasks Thread of Control Event Queue

' ' ' '

memory management time management device driver interrupt service

Application Kernel

External interrupt

Interrupt dispatch

Interrupt service Scheduling & dispatcher Task execution

Timer interrupt

Scheduler

Task Queue

Time service & events

Signal Queue

System calls (trap)

Services (create thread, sleep, notify, send,)

kernel University of Pittsburgh Manas Saksena 21 University of Pittsburgh Manas Saksena 22

Real-Time OS: Examples

Traditional Real-Time OSes

VxWorks, pSOS, QNX, LynxOS ThreadX, Ercos, Nucleus Solaris, NT, Linux

Response Time Factors

Execution Time

Amount of time taken in the absence of resource contention

Minimal Real-Time OSes General Purpose OS Made Real -Time

Higher Priority Non Schedulable Entities

Interrupt Handlers Time taken by higher priority work Time taken by lower priority work Often, due to contention on some shared resource
Manas Saksena 24

Scheduling Shared Resources

There are many other real-time operating systems. Many developers roll out their own real-time operating system

University of Pittsburgh

Manas Saksena

23

University of Pittsburgh

Scheduling
Tasks

Primary scheduling entities Scheduled by the kernel Done within the task Usually makes sense with event-driven tasks

Real-Time Software Design

Task Centric Design

Tasks are primary units for scheduling Application = Set of Tasks

Task Activation by signals

Events or Messages

Signals

Interrupts to tasks Useful for activating tasks (e.g., timer expiry)

Event Centric Design

Events are primary units for scheduling Application = Set of Events and their Responses
Single Task or Multiple Tasks

University of Pittsburgh

Manas Saksena

25

University of Pittsburgh

Manas Saksena

26

Task Centric Design

Initialize Task Wait Control Read Inputs Perform Work Write Outputs

Event Centric Design - Single Task

Event Scheduling Initialize

Task Scheduling Preemptive or Non-Preemptive

Often, priority based mechanism

Non-Preemptive
Wait for Event Process Event Priority Based Event Priorities

Task Priorities
Application Controlled

Application Controlled

Terminate Terminate Task

University of Pittsburgh Manas Saksena 27 University of Pittsburgh Manas Saksena 28

Event Centric Design - Multiple Tasks

Initialize Wait for Event Process Event Initialize Wait for Event Process Event

Task Centric Design

Terminate

Terminate

Task Priorities and Scheduling Fixed, or Dynamic

Preemptive Scheduling
University of Pittsburgh Manas Saksena 29 University of Pittsburgh

Manas Saksena

Manas Saksena

30

Abstract Design Model

Identify Events

Interrupts Timers

Implementing Tasks
Two Common Ways

Interrupt Handlers Threads Need an RTOS that supports threads (also called tasks in many RTOSes)

Event Handlers

Code Segments call these actions

task

Interrupt Handlers

fast response times difficult to program correctly often IH will do some minimal work and then wake up a task to the real work
Manas Saksena 32

Typically, mix of both

event action

University of Pittsburgh

Manas Saksena

31

University of Pittsburgh

Multi-Tasking
Communications

Shared Memory Semaphores/Mutexes to Control Access Messages Message Queues

Key Concepts
Tasks

Encapsulate thread of control Schedulable Unit Each task implements one function

Scheduling

Priorities IH may be viewed as higher priority tasks Preemptive Priority Scheduling kernel is usually fully preemptive
University of Pittsburgh Manas Saksena 33

Protected Objects

Encapsulate Shared Data (Resource) Mutually exclusive operation of Object Methods

University of Pittsburgh

Manas Saksena

34

Response Time Components

Execution
Execution time of the task itself, running alone

Execution and Preemption

Preemption
Execution delayed by higher priority tasks Execution delayed by lower priority tasks

HIGHER PRIORITY TASK P MY TASK PREEMPTION TIME BLOCKING TIME B

LOWER PRIORITY TASK

priority

Blocking (Priority inversion)

time
University of Pittsburgh Manas Saksena 35 University of Pittsburgh Manas Saksena 36

Preemption Analysis
Each higher priority task has a preemption effect Depends on Number of Arrivals and Execution Time of the Higher Priority Task
I j ( R ) = NumArrival s j ( R ) C j
Number of Arrivals of Task j in a window of interval R Preemption Effect from Task j
University of Pittsburgh

Unbounded Priority Inversion

high

S1

low

High and low priority tasks share a resource This can lead to UNBOUNDED Blocking
s1

highest

Attempts to lock S1

high medium s1 s1 s1

low

Response Time
Manas Saksena 37

Normal execution

s1

Execution in Critical Section

Unbounded Priority Inversion

University of Pittsburgh Manas Saksena 38

Highest Locker Protocol

high

Priority Inheritance Protocol

high

S1

low

High and low priority tasks share a critical section Ceiling Priority of CS = Highest Priority Task that can use the CS Execute CS at Ceiling priority
Ready s1 Ready

S1

low

High and low priority tasks share a critical section Inherit priority of blocked task in critical section

highest

high medium s1 s1 s1

highest
Attempts to lock S1

high Ready medium s1 s1 s1

s1

low

low

Normal execution
University of Pittsburgh

Execution in Critical Section

39

Normal execution
University of Pittsburgh

s1

Execution in Critical Section

40

Manas Saksena

Manas Saksena

Example of Chained Blocking

1

Blocking Analysis
Common Property

A lower priority task can cause blocking only while it is in a critical section

S1 S2

Same task set as sample problem Assumes priority inheritance used May need to wait for all possible critical sections
Attempts to lock S2 S1 S2

Attempts to lock S1

1
S1 S1

Highest Locker

The critical section must have a ceiling priority that is higher (or equal) than task priority Only one such CS can cause blocking

2
S2

S2

Priority Inheritance Protocol

Multiple CS can cause blocking in a chained manner The effect is sum of the largest such chain
University of Pittsburgh Manas Saksena 42

Normal execution

S1 S2

Execution in Critical Section using S 1 Execution in Critical Section using S 2

41

University of Pittsburgh

Manas Saksena

Response Time Analysis

Ri = Ci + Bi +
Response Time Execution time of task

Design/Implementation Choices
Non-Preemptive Scheduling
Similar Analysis, but consider that No preemption effect once task is given CPU Blocking effect due to non-preemptive execution

jhp ( i )

I j ( Ri )

Resource Contention Protocols

Different protocols have different blocking effects
Preemption from higher priority tasks Assumptions No Self Interference Bounded Blocking

Priority Assignment
Preemption and Blocking Effects Depend on Task Priorities

Blocking time from lower priority tasks (priority inversion)

Ri Ti

) Task Response times depend on all these three factors. ) These represent design/implementation choices.
43 University of Pittsburgh Manas Saksena

Functional Correctness does (should) not depend on the choice.

44

University of Pittsburgh

Manas Saksena

Why Linux?
Linux as a Real-Time Operating System
Reliable, Full-featured Operating System Rich multi-tasking support Security, Protection

Networking Support TCP/IP, RSVP, SIP, MPLS, H.323 Multimedia Support JPEG, MPEG, GSM Unix Lineage Familiar environment for many users/developers POSIX Compliance

Standard, Known Environment and APIs

Manas Saksena

University of Pittsburgh

Manas Saksena

45

University of Pittsburgh

Manas Saksena

46

Why Linux?
The Cost Factor

Free run-time royalties A global team of programmers enhancing the environment literally all the time Availability of libraries, tools, and device drivers Source Code Access allowing peeking inside the hood (and customizing as necessary)

Why Linux?
Small Embedded Systems

Modular Kernel, possible to configure the kernel to suitable size Customizable Root File System BusyBox Utilities

The Open Source Factor

High-End Embedded Systems

High-Availability Clustering SMP Support

The Popularity Factor

Excellent textbooks and documentation

Manas Saksena 47 University of Pittsburgh

University of Pittsburgh

Manas Saksena

48

Linux API: Tasking

Process

Encapsulates a thread of control and an address space Address space may be shared giving threads in effect Schedulable Entity

Linux API: POSIX, SVR4, BSD

POSIX 1003.1.b (Real-Time Extensions) Priority Scheduling Memory Locking Clocks and Timers
Real-Time Signals

Threads

Are processes to the Linux kernel Scheduled by the Linux kernel Can be created such that they share the address space with the parent process, effectively giving threads
University of Pittsburgh Manas Saksena 49

POSIX 1003.1.c (Thread Extensions) Using pthreads library Thread creation, destruction, etc.
Mutexes, Condition Variables

SVR4 IPC Shared Memory

Semaphores

Networking:
BSD Sockets
University of Pittsburgh Manas Saksena 50

Linux Internals Architecture

Linux Internals: Scheduling

Schedulable Entities
Processes Real-Time Class: SCHED_FIFO or SCHED_RR TimeSharing Class: SCHED_OTHER Real-Time processes have Application defined priority Higher priority than time-sharing processes

Modules
ipc

mm Process Scheduler vfs

net

Non Schedulable Entities

Interrupt Handlers Have priorities, and can be nested Bottom Halves & Task Queues Run on schedule, ret from system call, ret from interrupt
Manas Saksena 52

Device Drivers

Core Mechanisms

University of Pittsburgh

Manas Saksena

51

University of Pittsburgh

Linux and Real-Time: Problems

Timer Granularity

Many real-time tasks are driven by timer interrupts In Standard Linux, the timer is set to expire at 10 ms intervals

Linux Internals: Priority Inversion

Non Preemptible Kernel

A process running in the kernel cannot be preempted by another (higher) priority process Introduces, unwanted priority inversion

Scheduler Predictability

Linux scheduler keeps tasks in an unsorted list Requires a scan of all tasks to make a scheduling decision Scales poorly as number of tasks increases, and is especially poor for real-time performance

Non Schedulable Entities

Interrupt Handlers Bottom Half Handlers No mechanism to bound priority inversion in the presence of resource sharing
Manas Saksena 54

Resource Sharing

University of Pittsburgh

Manas Saksena

53

University of Pittsburgh

The Real-Time Linux Challenge

Approaches to Real-Time Linux

Approaches limiting Real-time and Non Real-time Task Interactions
Compliant Kernel Approach

How to leverage the advantages of Linux, while making it suitable for real-time systems?

LynxOS/Blue Cat Linux Thin Kernel Approach RTLinux/RTAI

Approaches that integrate Real-time and Non Realtime tasks

Core Kernel Approach

TimeSys Linux/RT, Hard Hat Linux Resource Kernel Approach TimeSys Linux/RT

University of Pittsburgh

Manas Saksena

55

University of Pittsburgh

Manas Saksena

56

Approaches to Real-Time Linux

Compliant Kernel Approach

Compliant Kernel Approach

Linux Development Tools And Environment Linux Development Tools And Environment

Dual Kernel Approach

Linux System Call API

Linux System Call API

Core Kernel Approach Linux Kernel

(Embedded Applications)

Real-Time Kernel (Real-Time Applications)

Resource Kernel Approach

University of Pittsburgh

Manas Saksena

57

University of Pittsburgh

Manas Saksena

58

Compliant Kernel Approach

Basic Claim

Linux is defined by its API and not by its internal implementation The real-time kernel is a non Linux kernel

Compliance
100% Linux API

Support all of Linux kernel API Any Linux application can run on real -time kernel Development can be done on Linux Host, with rich set of host tools for development All Linux libraries are trivially available to run on real-time kernel Third party software Achieving 100% Linux API is non-trivial Consider the amount of effort put on Linux kernel development
59 University of Pittsburgh Manas Saksena 60

Implications

Implications

No benefits from the Linux kernel Not possible to benefit from the Linux kernel evolution Not possible to use Linux hardware support Not possible to use Linux device drivers
University of Pittsburgh Manas Saksena

10

Approaches to Real-Time Linux

Compliant Kernel Approach
User-Level

The Thin Kernel Approach

Linux Process Linux Process

Dual Kernel Approach

Kernel-Level

Core Kernel Approach

Real-Time Task

Real-Time Task

Real-Time Task

Linux Kernel

Real-Time Kernel (RT-Linux or RTAI)

Resource Kernel Approach

Hardware

University of Pittsburgh

Manas Saksena

61

University of Pittsburgh

Manas Saksena

62

Linux Preemptability
Linux kernel is run as the lowest priority task in the real-time kernel

Requires small changes to the Linux kernel

Programming Model
Real-Time Tasks Non Real-Time Tasks

Linux kernel can be preempted at any time by the real-time kernel Real-Time kernel is a small multi -tasking kernel providing multi-tasking, inter-task communication and synchronization, and a fixed priority scheduler
Very tight response times for rt-tasks
University of Pittsburgh Manas Saksena 63

Real-Time Tasks

Run in kernel Limited API, restricted to what is provided by the real-time kernel

Non Real-Time Tasks

Run in user-space under Linux

Manas Saksena 64 University of Pittsburgh

Programming Model
Memory Protection

Not available for real-time tasks A single errant real-time task can bring the system down

Soft and Dynamic Real-Time

Static Decomposition into
Real-time parts, and non real-time parts

Duplication of Functionality

Currently: Multi-tasking, task scheduling, inter-task communication and synchronization Future Extensibility E.g., IP protocol stack, file system,

Consider Real-time often involves many shades of gray (soft real -time)
The soft/hardness of real-time may be dynamically decided

Implications
Support soft real-time tasks in real-time kernel Limited API Support soft real-time tasks in Linux kernel No performance guarantees

Starvation of non real-time tasks

University of Pittsburgh Manas Saksena 65 University of Pittsburgh Manas Saksena 66

11

Approaches to Real-Time Linux

Compliant Kernel Approach

Core Kernel Approach

Basic Ideas
Make the kernel more suitable for real-time Ensure that the impact of changes is localized so that Kernel upgrades can be easily incorporated Kernel reliability and scalability is not compromised

Dual Kernel Approach

Core Kernel Approach

Mechanisms

Static Configuration Can be configured at compile time Dynamic Configuration Using loadable kernel modules
Manas Saksena 68

Resource Kernel Approach

University of Pittsburgh

Manas Saksena

67

University of Pittsburgh

Preemptive Kernel
Linux kernel is non-preemptive

Simplifies synchronization in the kernel since kernel data structures that are not touched by interrupts need not be locked. Introduces potentially large priority inversions

Minimize Non-Schedulable Work

Schedulable Entities

Processes or Threads Interrupt Handlers Bottom Halves

Non Schedulable Entities

But, Linux is already SMP safe

That is, the hooks needed to make it preemptive are already there. Solution Use SMP lock hooks to make the kernel preemptible
University of Pittsburgh Manas Saksena 69

Solution Use schedulable entities (interrupt threads) to do the bulk of the work These can be prioritized according to application requirements
University of Pittsburgh Manas Saksena 70

Priority Inheritance
A lower priority process holds a resource needed by a higher priority process
Can lead to unbounded priority inversion, unless some form of priority inheritance is used. Solution Provide a kernel mutex implementation that supports priority inheritance Use the mutex for locks in the kernel Export the mutex to user level processes using system calls

High Accuracy Timers

Linux uses a periodic heartbeat timer (by default 10 ms) to implement timers.

Provides poor accuracy Increasing the frequency improves accuracy, but increases overheads

Solution Use a one-shot hardware timer to support all software timers Use a 10 ms periodic timer to do regular Linux timer work Can provide high accuracy at a low overhead
71 University of Pittsburgh Manas Saksena 72

University of Pittsburgh

Manas Saksena

12

Predictable Scheduler
Current Linux Scheduler

Does not scale well as the number of processes increases

Core Kernel Approach

Allows the use of most if not all existing Linux primitives, applications, and tools.

Need to avoid primitives that can take extended time in the kernel

Real-Time Scheduler

Predictable, Low Overhead Separate real-time and non real-time Non real-time processes do not effect real-time scheduling behavior Implement a better scheduler
University of Pittsburgh Manas Saksena 73

Solution(s)

Allows the use of most existing device drivers written to support Linux.

Need to avoid poorly written drivers that unfairly hog system resources

Robustness and Reliability

Core kernel modifications can effect robustness, but source is available

Manas Saksena 74

University of Pittsburgh

URLs
www.embedded.com

Embedded Systems Conference papers available online

http://cs-www.bu.edu/pub/ieee-rts/Home.html

IEEE Real-Time Technical Committee Page many useful links

University of Pittsburgh

Manas Saksena

75

13