Synchronization Primitives

Several synchronization primitives have been introduced to aid in

multithreading the kernel. These primitives are implemented by atomic
operations and use appropriate memory barriers so that users of these
primitives do not have to worry about doing it themselves. The primitives are
very similar to those used in other operating systems including mutexes,
condition variables, shared/exclusive locks, and semaphores.

Mutexes
The mutex primitive provides mutual exclusion for one or more data objects.
Two versions of the mutex primitive are provided: spin mutexes and sleep
mutexes.

Spin mutexes are a simple spin lock. If the lock is held by another thread when
a thread tries to acquire it, the second thread will spin waiting for the lock to
be released. Due to this spinning nature, a context switch cannot be performed
while holding a spin mutex to avoid deadlocking in the case of a thread owning
a spin lock not being executed on a CPU and all other CPUs spinning on that
lock. An exception to this is the scheduler lock, which must be held during a
context switch. As a special case, the ownership of the scheduler lock is passed
from the thread being switched out to the thread being switched in to satisfy
this requirement while still protecting the scheduler data structures. Since the
bottom half code that schedules threaded interrupts and runs non-threaded
interrupt handlers also uses spin mutexes, spin mutexes must disable
interrupts while they are held to prevent bottom half code from deadlocking
against the top half code it is interrupting on the current CPU. Disabling
interrupts while holding a spin lock has the unfortunate side effect of
increasing interrupt latency.

To work around this, a second mutex primitive is provided that performs a

context switch when a thread blocks on a mutex. This second type of mutex is
dubbed a sleep mutex. Since a thread that contests on a sleep mutex blocks
instead of spinning, it is not susceptible to the first type of deadlock with spin
locks. Sleep mutexes cannot be used in bottom half code, so they do not need
to disable interrupts while they are held to avoid the second type of deadlock
with spin locks.
As with Solaris, when a thread blocks on a sleep mutex, it propagates its
priority to the lock owner. Therefore, if a thread blocks on a sleep mutex and
its priority is higher than the thread that currently owns the sleep mutex, the
current owner will inherit the priority of the first thread. If the owner of the
sleep mutex is blocked on another mutex, then the entire chain of threads will
be traversed bumping the priority of any threads if needed until a runnable
thread is found. This is to deal with the problem of priority inversion where a
lower priority thread blocks a higher priority thread. By bumping the priority of
the lower priority thread until it releases the lock the higher priority thread is
blocked on, the kernel guarantees that the higher priority thread will get to run
as soon as its priority allows.

These two types of mutexes are similar to the Solaris spin and adaptive
mutexes. One difference from the Solaris API is that acquiring and releasing a
spin mutex uses different functions than acquiring and releasing a sleep mutex.
A difference with the Solaris implementation is that sleep mutexes are not
adaptive. Details of the Solaris mutex API and implementation can be found in
section 3.5 of [Mauro01].

Condition Variables
Condition variables provide a logical abstraction for blocking a thread while
waiting for a condition. Condition variables do not contain the actual condition
to test, instead, one locks the appropriate mutex, tests the condition, and then
blocks on the condition variable if the condition is not true. To prevent lost
wakeups, the mutex is passed in as an interlock when waiting on a condition.

FreeBSD's condition variables use an API quite similar to those provided in

Solaris. The only differences being the lack of a cv_wait_sig_swap and the
addition of cv_init and cv_destroy constructors and destructors. The
implementation also differs from Solaris in that the sleep queue is embedded
in the condition variable itself instead of coming from the hashed pool of sleep
queue's used by sleep andwakeup.

Shared/Exclusive Locks
Shared/Exclusive locks, also known as sx locks, provide simple reader/writer
locks. As the name suggests, multiple threads may hold a shared lock
simultaneously, but only one thread may hold an exclusive lock. Also, if one
thread holds an exclusive lock, no threads may hold a shared lock.
FreeBSD's sx locks have some limitations not present in other reader/writer
lock implementations. First, a thread may not recursively acquire an exclusive
lock. Secondly, sx locks do not implement any sort of priority propagation.
Finally, although upgrades and downgrades of locks are implemented, they
may not block. Instead, if an upgrade cannot succeed, it returns failure, and
the programmer is required to explicitly drop its shared lock and acquire an
exclusive lock. This design was intentional to prevent programmers from
making false assumptions about a blocking upgrade function. Specifically, a
blocking upgrade must potentially release its shared lock. Also, another thread
may obtain an exclusive lock before a thread trying to perform an upgrade. For
example, if two threads are performing an upgrade on a lock at the same time.

Semaphores

Semaphore is used as a synchronization tool. A semaphore S is an integer

variable that, apart from initialization, is accessed only through two standard
atomic operations: wait and signal. P(for Wait) and V(for Signal).

The classical definition of Wait in pseudocode is

Wait(S) {

While(S<=0)

;//no-op

S--;

For Signal

Signal(S){

S++;

} When one process modifies the semaphore value, no other process can
simultaneously modify that same semaphore value.
Atomic Primitives
Atomic primitives are arguably the most important tool in programming that
requires coordination between multiple threads and/or processors/cores.
There are four basic types of atomic primitives: swap, fetch and phi, compare
and swap, and load linked/store conditional. Usually these operations are
performed on values the same size as or smaller than a word (the size of the
processor's registers), though sometimes consecutive (single address)
multiword versions are provided.

The most primitive (pun not intended) is the swap operation. This operation
exchanges a value in memory with a value in a register, atomically setting the
value in memory and returning the original value in memory. This is not
actually very useful, in terms of multiprogramming, with only a single practical
use: the construction of test and set (TAS; or test and test and set - TATAS)
spinlocks. In these spinlocks, each thread attempting to enter the spinlock
spins, exchanging 1 into the spinlock value in each iteration. If 0 is returned,
the spinlock was free, and the thread now owns the spinlock; if 1 is returned,
the spinlock was already owned by another thread, and the thread attempting
to enter the spinlock must keep spinning. Ultimately, the owning thread leaves
the spinlock by setting the spinlock value to 0.

C/C++ - Code

1. ATOMIC word swap(volatile word &value, word newValue)

2. {
3. word oldValue = value;
4. value = newValue;
5.
6. return oldValue;
7. }
8.
 9. void EnterSpinlock(volatile word &value)
10.{
11.// Test and test and set spinlock
12.while (value == 1 || swap(value, 1) == 1) {}
13.}
14.
15.void LeaveSpinlock(volatile word &value)
16.{
17.value = 0;
18.}
This makes for an extremely crude spinlock. The fact that there is only a single
value all threads share means that spinning can create a lot of cache coherency
traffic, as all spinning processors will be writing to the same address,
continually invalidating each other's caches. Furthermore, this mechanism
precludes any kind of order preservation, as it's impossible to distinguish when
a given thread began waiting; threads may enter the spinlock in any order,
regardless of how long any thread has been waiting to enter the spinlock.

Next up the power scale is the fetch and phi family. All members of this family
follow the same basic process: a value is atomically read from memory,
modified by the processor, and written back, with the original value returned
to the program. The modification performed can be almost anything; one of
the most useful modifications is the add operation (in this case it's called fetch
and add). The fetch and add operation is notably more useful than the swap
operation, but is still less than ideal; in addition to test and set spinlocks, fetch
and add can be used to create thread-safe counters, and spinlocks that both
preserve order and (potentially) greatly reduce cache coherency traffic.

1. ATOMIC word fetch_and_add(volatile word &value, word addend)

2. {
3. word oldValue = value;
4. value += addend;
5.
6. return oldValue;
7. }
8.
 9. void EnterSpinlock (volatile word &seq, volatile word &cur)
10.{
11.word mySeq = fetch_and_add(seq, 1);
12.while (cur != mySeq) {}
13.}
14.
15.void LeaveSpinlock(volatile word &cur)
16.{
17.cur++;
18.}
Next is the almost ubiquitous compare and swap (CAS) operation. In this
operation, a value is read from memory, and if it matches a comparison value
in the processor, a third value in the processor is atomically written in the
memory location, and ultimately the original value in memory is returned to
the program. This operation is very popular because it allows you to perform
almost any operation on a single memory location atomically, by reading the
value, modifying it, and then compare-and-swapping it back to memory. For
this reason, it is considered to be a universal atomic primitive.

1. ATOMIC word compare_and_swap(volatile word &value, word

newValue, word comparand)
2. {
3. word oldValue = value;
4. if (oldValue == comparand)
5. value = newValue;
6.
7. return oldValue;
8. }
Some architectures (such as the x86) support double-width compare and swap
(atomic compare and swap of two consecutive words). While this is
convenient, it should not be relied upon in portable software, as many
architectures do not support it. Note that double-width compare and swap is
NOT the same as double compare and swap (which we'll look at soon).

Another universal atomic primitive is the load linked/store conditional (LL/SC)

pair of instructions. In this model, when a value is loaded linked into a register,
a reservation is placed on that memory address. If that reservation still exists
when the store conditional operation is performed, the store will be
performed. If another thread writes to the memory location between a load
linked and store conditional, the reservation is cleared, and any other threads
will fail their store conditional (resulting in skipping the store altogether). If the
store conditional fails, the program must loop back and try the operation
again, beginning with the load linked.

In theory, the load linked/store conditional primitive is superior to the

compare and swap operation. As any access to the memory address will clear
the reservations of other processors, the total number of reads is 1, in contrast
to the 2 reads with compare and swap (1 read to get the value initially, and a
second read to verify the value is unchanged). Furthermore, the LL/SC
operation may distinguish whether a write has occurred, regardless of whether
the value was changed by the write (we'll come back to why this is a good
thing when we discuss hazard pointers). Unfortunately, my research indicates
that most implementations of LL/SC do not guarantee that a write will be
distinguished if the written value is the same as the value at the time of the
reserve.

Finally, the wild card: the double compare and swap (DCAS). In a double
compare and swap, the compare and swap operation is performed
simultaneously on the values in two memory addresses. Obviously this
provides dramatically more power than any previous operation, which only
operate on single addresses. Unfortunately, support for this primitive is
extremely rare in real-world processors, and it is typically only used by lock-
free algorithm designers that are unable to reduce their algorithm to single-
address compare and swap operations.

1. ATOMIC void double_compare_and_swap(volatile word &value1,

word newValue1, word comparand1, word &oldValue1, volatile
word &value2, word newValue2, word comparand2, word
&oldValue2)
2. {
3. oldValue1 = value1;
4. oldValue2 = value2;
5.
6. if (oldValue1 == comparand1 && oldValue2 == comparand2)
 7. {
8. value1 = newValue1;
9. value2 = newValue2;
10.}
11.}
Lastly, note that I'm using rather loose classification of these primitives.
Technically, you could place the swap (fetch and store) operations in the fetch
and phi family, as well, but it seemed more intuitive to me to separate them.

Ticket Lock
A ticket lock is a form of lockless inter-thread synchronization.

Overview

Conventionally, inter-thread synchronization is achieved by using

synchronization entities provided by the operating system, such as events,
semaphores and mutexes.For example, a thread will create a mutex and in the
act of creation, "claim" the mutex. A mutex can have only one owner at any
time. Other threads, when they come to the point where their behaviour must
be synchronized, will attempt to "claim" the mutex; if they cannot, because
another thread already owns the mutex, they automatically sleep until the
thread which currently owns the mutex gives it up. Then one of the currently
sleeping threads will automatically be awoken and given ownership of the
mutex.

A ticket lock works as follows; there are two integer values which begin at 0.
The first value is the queue ticket, the second is the dequeue ticket.

When a thread arrives, it atomically obtains and then increments the queue
ticket. It then atomically compares its ticket with the dequeue ticket. If they
are the same, the thread is permitted to enter the serialised code. If they are
not the same, then another thread must already be in the serialised code and
this thread must busy-wait or yield. When a thread has comes to leave the
serialised code, it atomically increments the dequeue ticket, thus permitting
the next waiting thread to enter the serialised code.

This approach is heavyweight (high-overhead, code intensive), in that such

entities have a significant impact upon the performance of the operating
system, since the operating system has to switch into a special mode when
dealing with operations upon synchronization entities to ensure they provide
synchronized behaviour.

A further drawback is that if the thread which owns the mutex fails, the entire
application halts. (This type of problem applies to all synchronization entities).

Lockless locking
Main article: Lock-free and wait-free algorithms

Lockless locking achieves inter-thread synchronization without the use of

operating system provided sychronization entities.Generally, two techniques
are involved.

The first technique is the use of a special set of instructions which are
guaranteed atomic by the CPU. This generally centers on an instrument known
as Compare-and-swap. This instruction compares two variables and if they are
the same, replaces one of the variable's values with a third value.

For example;compare_and_swap ( destination, exchange, comparand );

Here the destination and comparand are compared. If they are identical, the
value in exchange is placed in destination.

This first technique provides the vital inter-thread atomicity of operation. It

would otherwise be impossible to perform any lockless locking, since on
systems with multiple processors, even operations such as a simple assignment
(let a = b) could be half-way through occurring while other operations occur on
other processors; the software trying to sychronize across processors could
never be sure that any operation had reached a sane state permitting it to be
used in some way.

The second technique is to busy-wait or yield when it is clear that the current
thread cannot be permitted to continue processing. This provides the vital
ability to defer the processing done by a thread when such processing would
violate the operations which are being serialised between threads.
Test & Test & Set Lock
In computer science, the test-and-set CPU instruction is used to
implement mutual exclusion in multiprocessor environments. Although a
correct lock can be implemented with test-and-set, it can lead to memory
contention in busy lock (caused by bus locking and cache invalidation when
test-and-set operation needs to access memory atomically).

To lower the overhead a more elaborate locking protocol test and test-and-
set is used. The main idea is not to spin in test-and-set but increase the
likelihood of successful test-and-set by using the following entry protocol to
the lock:

boolean locked := false // shared lock variable procedure EnterCritical()

{ do { while (locked == true) skip // spin until
lock seems free } while TestAndSet(locked) // actual atomic locking }

Exit protocol is:

procedure ExitCritical() { locked := false }

The entry protocol uses normal memory reads to spin, waiting for the lock to
become free. Test-and-set is only used to try to get the lock when normal
memory read says it's free. Thus the expensive atomic memory operations
happens less often than in simple spin around test-and-set.

If the programming language used supports short-circuit evaluation, the entry

protocol could be implemented as:

procedure EnterCritical() { while ( locked == true or TestAndSet(locked) == true

) skip // spin until locked }
Array Based Locks
In array-based locks each lock is implemented as an array of size p where p is
the total number of processors. The lock acquire involves atomically copying
and incrementing a shared index variable and spinning on the array location
indexed by the copied value. Releasing a lock involves resetting the
corresponding array location and setting the next array location if a processor
is waiting. Each array location should be allocated on a different cache line to
avoid invalidations due to false-sharing. Assume a large enough cache line size
(say, 256 bytes) and put that amount of padding between consecutive lock
locations. Use LL/SC for implementing any atomic code section that may be a
part of lock acquire or release. The algorithm should be written in C with calls
to atomic primitives implemented with LL/SC in MIPS assembly.
It Solves the O (P2) traffic problem.

Barriers
Race conditions can cause a program to execute in a non-deterministic fashion,
producing inconsistent results. Synchronization routines are used to remove
race conditions from a code. In certain cases all threads must have executed a
portion of code before continuing. Barrier synchronization is a technique to
achieve this. A thread executing an episode of a barrier waits for all other
threads before proceeding to the next. Therefore, when a barrier is reached,
all threads are forced to wait for the last thread to arrive. Use of barriers is
common in shared memory parallel programming.

Some terminologies in the context of this report:

1. Threads – Units of execution initiated by the program using openMp which
might run on different processors but on the same parallel machine in which
they were initiated.
2. Parallel Machine – A collection of processors which are tightly coupled and
use shared memory. Jedi1 is considered a parallel machine. It has 4 processors.
3. Processor – A subunit of a parallel machine.

Two types of Barrier are

1.Centralized Barrier

2.Tree Barrier
Centralized Barrier

Each of the N threads entering the barrier atomically decrements a shared

integer, the initial value of which is N. If the value of the integer after
decrement is 0 then the thread resets the counter and changes a shared
central flag. Otherwise, the thread waits for notification. Arriving processor
decrements “count” and then wait until “sense” has a different value than it
had in the previous barrier. The last arriving processor resets “count” and
reverses “sense”. The algorithm relies on every thread reading and writing to a
single memory location: the counter. This memory location is known as a hot-
spot.

Tree Barrier

Does not need a lock, only uses flags

-Arrange the processors logically in a binary tree (higher degree also possible)
-Two siblings tell each other of arrival via simple flags (i.e. one waits on a flag
while the other sets it on arrival)
-One of them moves up the tree to participate in the next level of the barrier
- Introduces concurrency in the barrier algorithm since independent subtrees
can proceed in parallel
- Takes log(P) steps to complete the acquire
- A fixed processor starts a downward pass of release waking up other
processors that in turn set other flags
- Shows much better scalability compared to centralized barriers in DSM
multiprocessors; the advantage in small bus-based systems is not much, since
all transactions are any way serialized on the bus; in fact the additional log (P)
delay may hurt performance in bus-based SMPs

Binary Barrier Tree Signal

Binary barrier tree wakeup

Each processor will need at most log (P) + 1 flags

Avoid false sharing: allocate each processor’s flags on a separate chunk of
cache lines
With some memory wastage (possibly worth it) allocate each processor’s flags
on a separate page and map that page locally in that processor’s physical
memory.
Avoid remote misses in DSM multiprocessor
Does not matter in bus-based SMPs
What is Sense Reversal?
• Problem: reinitialization
—each time a barrier is used, it must be reset
• Difficulty: odd and even barriers can overlap in time
—some processes may still be exiting the kth barrier
—other processes may be entering the k+1st barrier
—how can one reinitialize?
• Solution: sense reversal
—terminal state of one phase is initial state of next phase
—e.g.
– odd barriers wait for flag to transition from true to false
– even barriers wait for flag to transition from false to true
• Benefits
—reduce number of references to shared variables
—eliminate one of the spinning episodes