Testing Concurrent Programs

Rice Computer Science Club Mathias Ricken Rice University October 4, 2007

Moores Law

Timeliness
CPU clock frequencies stagnate Multi-Core CPUs provide additional processing power
Multiple threads needed to use multiple cores

Writing concurrent programs is difficult!

Programming Examples

Unit Testing
Unit tests
Test a part, not the whole program Occur earlier Automate testing Serve as documentation Prevent bugs from reoccurring Help keep the shared repository clean

Effective with a single thread of control

Foundation of Unit Testing

Unit tests depend on deterministic behavior Known input, expected output Success correct behavior Failure flawed code Outcome of test is meaningful

Problems Due to Concurrency

Thread scheduling is nondeterministic and machine-dependent
Code may be executed under different schedules Different schedules may produce different results

Known input, expected output

Success correct behavior in this schedule, may be flawed in other schedule flawed code

Failure

Success of unit test is meaningless

Possible Solutions
Programming Language Features
Ensuring that bad things cannot happen May restrict programmers

Lock-Free Algorithms
Ensuring that if bad things happen, its ok May limit data structures available

Comprehensive Testing
Testing if bad things happen in any schedule Does not prevent problems, but does not limit solutions either

Contributions
Improvements to JUnit
Detect exceptions and failed assertions in threads other than the main thread

Annotations for Concurrency Invariants

Express complicated requirements about locks and threads

Tools for Schedule-Based Execution

Record, deadlock monitor Random delays, random yields

Improvements to JUnit
Uncaught exceptions and failed assertions
Not caught in child threads

Sample JUnit Tests

public class Test extends TestCase { public void testException() { throw new RuntimeException("booh!"); } } public void testAssertion() { Both tests assertEquals(0, 1); fail. } }
if (0!=1) throw new AssertionFailedError();

Problematic JUnit Tests

public class Test extends TestCase { public void testException() { new Thread(new Runnable() { public void run() { throw new RuntimeException("booh!"); } }).start(); Child thread } } end of
Main thread spawns uncaught! Main thread

test
success!

Child thread

Problematic JUnit Tests

public class Test extends TestCase { public void testException() { new Thread(new Runnable() { public void run() { throw new RuntimeException("booh!"); } }).start(); Child Uncaught exception, thread } test should fail but } does not!
Main thread

Improvements to JUnit
Uncaught exceptions and failed assertions
Not caught in child threads

Thread group with exception handler

JUnit test runs in a separate thread, not main thread Child threads are created in same thread group When test ends, check if handler was invoked

Thread Group for JUnit Tests

public class Test extends TestCase { public void testException() { new Thread(new Runnable() { public void run() { throw new RuntimeException("booh!"); } }).start(); Child thread } invokes }
TestGroups Uncaught Exception Handler Test thread

Thread Group for JUnit Tests

public class Test extends TestCase { public void testException() { new Thread(new Runnable() { public void run() { throw new RuntimeException("booh!"); } }).start(); Child thread } spawns and waits resumes } Main failure!
thread Test thread spawns end of test uncaught! invokes groups handler Test thread

check groups handler

Child thread

Improvements to JUnit
Uncaught exceptions and failed assertions
Not caught in child threads

Thread group with exception handler

JUnit test runs in a separate thread, not main thread Child threads are created in same thread group When test ends, check if handler was invoked

Detection of uncaught exceptions and failed assertions in child threads that occurred before tests end
Past tense: occurred!

Child Thread Outlives Parent

public class Test extends TestCase { public void testException() { new Thread(new Runnable() { public void run() { throw new RuntimeException("booh!"); } }).start(); Child thread } spawns and waits resumes } Main failure!
thread Test thread spawns end of test uncaught! invokes groups handler Test thread

check groups handler

Child thread

Child Thread Outlives Parent

public class Test extends TestCase { public void testException() { new Thread(new Runnable() { public void run() { throw new RuntimeException("booh!"); } }).start(); Child thread } check groups resumes handler } Main spawns and waits success!
thread Test thread spawns end of test Child thread uncaught! Too late! invokes groups handler Test thread

Improvements to JUnit
Child threads are not required to terminate
A test may pass before an error is reached

Detect if any child threads are still alive

Declare failure if test thread has not waited Ignore daemon threads, system threads (AWT, RMI, garbage collection, etc.)

Previous schedule is a test failure

Should be prevented by using Thread.join()

Enforced Join
public class Test extends TestCase { public void testException() { Thread t = new Thread(new Runnable() { new Thread(new Runnable() { public void run() { public void run() { throw new RuntimeException("booh!"); throw new RuntimeException("booh!"); throw new RuntimeException("booh!"); } } }); }); t.start(); t.join(); t.start(); t.join(); Child } thread }
Test thread

Improvements to JUnit
Child threads are not required to terminate
A test may pass before an error is reached

Detect if any child threads are still alive

Declare failure if test thread has not waited Ignore daemon threads, system threads (AWT, RMI, garbage collection, etc.)

Previous schedule is a test failure

Should be prevented by using Thread.join()

Testing ConcJUnit
Replacement for junit.jar or as plugin JAR for JUnit 4.2 Available as binary and source at http://www.concutest.org/ Results from DrJavas unit tests
Child thread for communication with slave VM still alive in test Several reader and writer threads still alive in low level test (calls to join() missing)

Conclusion
Improved JUnit now detects problems in other threads
Only in chosen schedule Needs schedule-based execution

Annotations ease documentation and checking of concurrency invariants

Open-source library of Java API invariants

Support programs for schedule-based execution

Future Work
Schedule-Based Execution
Replay given schedule Generate possible schedules Dynamic race detection Probabilities/durations for random yields/sleeps

Extend annotations to Floyd-Hoare logic

Preconditions, postconditions Representation invariants

Many Thanks To
My advisor
Corky Cartwright

My committee members
Walid Taha Bill Scherer

NFS and Texas ATP

For partially providing funding

Rice Computer Science Club

Extra Slides

Tractability of Comprehensive Testing

Test all possible schedules
Concurrent unit tests meaningful again

Number of schedules (N)

t: # of threads, s: # of slices per thread

detail

Extra: Number of Schedules

Product of s-combinations For thread 1: choose s out of ts time slices For thread 2: choose s out of ts-s time slices For thread t-1: choose s out of 2s time slices For thread t-1: choose s out of s time slices Writing s-combinations using factorial

Cancel out terms in denominator and next numerator

Left with (ts)! in numerator and t numerators with s!

back

Tractability of Comprehensive Testing

If program is race-free, we do not have to simulate all thread switches
Threads interfere only at critical points: lock operations, shared or volatile variables, etc. Code between critical points cannot affect outcome Simulate all possible arrangements of blocks delimited by critical points

Run dynamic race detection in parallel

Lockset algorithm (e.g. Eraser by Savage et al)

Critical Points Example

Local Var 1

lock
Thread 1

access unlock

All lock access unlock accesses protected by lock

Shared Var

Lock All accesses protected by lock lock access unlock All accesses protected by lock

Local variables dont need locking

Thread 2

Local Var 1

Fewer Schedules
Fewer critical points than thread switches
Reduces number of schedules Example: Two threads, but no communication N=1

Unit tests are small

Reduces number of schedules

Hopefully comprehensive simulation is tractable

If not, heuristics are still better than nothing

Limitations
Improvements only check chosen schedule
A different schedule may still fail Requires comprehensive testing to be meaningful

May still miss uncaught exceptions

Specify absolute parent thread group, not relative Cannot detect uncaught exceptions in a programs uncaught exception handler (JLS limitation)
details

Extra: Limitations
May still miss uncaught exceptions
Specify absolute parent thread group, not relative (rare)
Koders.com: 913 matches ThreadGroup vs. 49,329 matches for Thread

Cannot detect uncaught exceptions in a programs uncaught exception handler (JLS limitation)
Koders.com: 32 method definitions for uncaughtException method
back

Extra: DrJava Statistics

2004 2006

Unit tests passed failed not run Invariants met failed % failed KLOC event thread

736 610 36 90 5116 4161 965 18.83% 107 1

881 881 0 0 34412 30616 3796 11.03 129 99

back

Concurrency Invariants
Has to be called in event thread
TableModel, TreeModel

May not be called in event thread

invokeAndWait()

Have to acquire readers/writers lock

AbstractDocument DrJavas documents

Invariants Difficult to Determine

May be found in
Javadoc comments Only in internal comments Whitepapers

Often not documented at all Errors not immediately evident Impossible to check automatically

Java Annotations
Add invariants as annotations
@NotEventThread public static void invokeAndWait( Runnable r) { }

Process class files

Find uses of annotations Insert bytecode to check invariants at method beginning

Advantages of Annotations
Java Language constructs
Syntax checked by compiler

Easy to apply to part of the program

e.g. when compared to a type system change

Light-weight
Negligible runtime impact if not debugging (slightly bigger class files)

Automatic Checking

Predicate Annotations
In annotation definition, specify static boolean Java method
Method must be callable from every context completely static and public

Data in annotation, method arguments and value of this passed when method invoked

Predicate Annotation Example

@PredicateLink(value=Predicates.class, Refers to method="example", Predicates arguments=true) .example public @interface ExampleAnnotation { String foo; }
Definition

Predicate Annotation Example

public class TestCode { @ExampleAnnotation(foo="test") public void test(int param) { } } TestCode t = new TestCode(); t.test(5);
Usage

Call

Predicate Annotation Example

public class Predicates { public static boolean example( Object this0, int param, String foo) { return (foo.length()<param); }

Predicate Annotation Example

@PredicateLink(value=Predicates.class, method="example", arguments=true) public @interface ExampleAnnotation { String foo; } public class TestCode { @ExampleAnnotation(foo="test") public void test(int param){} } TestCode t = new TestCode(); t.test(5);

public class Predicates { public static boolean example( Object this0, int param, String foo) { return (foo.length()<param); // this0==t, param==5, foo=="test" }

Invariant Annotation Library

@OnlyEventThread, @NotEventThread @OnlyThreadWithName @NotNullArgument @DistinctArguments, @SameArguments

@OnlySynchronizedThis, @NotSynchronizedThis @OnlySynchronizedArgument, @NotSynchronizedArgument

etc. (ca. 80 annotations)

Problem: Multiple Annotations

Java does not allow the same annotation class multiple times
@OnlyThreadWithName("foo") @OnlyThreadWithName("bar") // error void testMethod() { }

Conjunctions, disjunctions and negations?

Annotation Subclasses?
Let annotation extend a supertype?
public @interface Invariant { } public @interface OnlyThreadWithName extends Invariant { String name(); } public @interface And extends Invariant { Invariant[] terms(); }

Subtyping not allowed for annotations

Generic Annotations?
Write @And as generic annotation?
public @interface And<T> { T[] terms(); } public @interface OnlyThreadWithName { String name(); }

Generics not allowed in annotations

Work-Around
Different meta-annotation, @Combine
@Combine(Combine.Mode.AND) public @interface SeveralNames { OnlyThreadWithName[] value(); } @SeveralNames({@OnlyThreadWithName("foo"), @OnlyThreadWithName("bar")}) void testMethod() { }

Combine Annotations
May only contain invariant annotations
Predicate annotations Combine annotations Arrays of the above

Predicate method automatically generated

Calls member predicate methods Accumulates using AND, OR or NOT
NOT first negates, then uses AND Default mode is OR De Morgans Law: NOT (a OR b) = (NOT a) AND (NOT b)

Invariant Inheritance
Invariants on a method
Apply to the method and all overriding methods in subclasses

Invariants on a class
Apply to all methods introduced in that class or subclasses

Methods can have invariants defined elsewhere

All annotations describe requirements for the client (and, due to subclassing, for subclasses)
Allows frameworks to describe requirements Description thread-safe is often wrong

Invariant Subtyping
To maintain substitutability, subclasses may not strengthen invariants Invariants can be modeled as special input parameter
Tuple of invariants (record in calculus [Pierce]) Subtyping rules for records declare the wider record as subtype In function types, parameter types are contravariant

I0 = {}, I1 = {inv1}, I2 = {inv1,inv2}, I2 <: I1 <: I0 F0 = I0 , F1 = I1 , F2 = I2 , F0 <: F1 <: F2

Invariant Subtyping
Analyze methods with invariants as parameter
class A { class B extends A { @Inv1 void f() { }; } class C extends B { @Inv2 void f() { }; }

void f() { };

Invariants subtyping:

A <@ B <@ C

IA = {}, IB = {inv1}, IC = {inv1,inv2}; IC <: IB <: IA FA = IA , FB = IB , FC = IC ; FA <: FB <: FC

Java subtyping:

C <: B <: A

Detection of Subtyping Problems

If Java subtyping and invariant subtyping disagree (A <: B but B <@ A)
Substitutability not maintained Statically emit warning

Detect if client subclasses do not use framework classes as prescribed

Safer multithreaded frameworks

Java API Annotations

Started to annotate methods in Java API
30 whole classes, 44 individual methods

Community project at http://community.concutest.org/

Anyone can suggest annotations Vote on suggested annotations Browse by class or annotation type

Annotations can be extracted into XML

Share annotations Add checks without needing source code

Testing Invariant Checker

Annotated two DrJava versions
3/26/2004 9/2/2006

Ran test suite, logged invariant violations

2004: 18.83% failed 2006: 11.03% failed

2006 version easier to annotate

Better documentation of invariants