Pex Unit Testing

Parameterized Unit Testing
with Microsoft Pex

(Long Tutorial)
Nikolai Tillmann Jonathan de Halleux

January 20, 2010
Abstract
This hands-on tutorial teaches the principles of Parameterized Unit Testing in
Visual Studio with Pex, an automatic test input generator
(http://research.microsoft.com/pex).
A parameterized unit test (PUT) is simply a method that takes parameters,
calls the code under test, and states assertions. Given a PUT written in a .NET
language, Pex automatically produces a small test suite with high code and as-
sertion coverage. Moreover, when a generated test fails, Pex can often suggest a
bug fix. To do so, Pex performs a systematic program analysis, similar to path
bounded model-checking. Pex learns the program behavior by monitoring execu-
tion traces, and uses a constraint solver to produce new test cases with different
behavior. At Microsoft, this technique proved highly effective in testing even an
extremely well-tested component.
From a specification, the developer (1) writes parameterized unit tests in C# to
reflect the specification, and (2) develops code that implements the specification.
The tutorial will outline key aspects to make this methodology successful in prac-
tice, including how to write mock objects, as well as the theoretical foundations on
which Pex is built.
This tutorial contains exercises which are specific to a particular version of
Pex. Please make sure that you use the tutorial that was included in the Pex distri-
bution when following the exercises.
Contents
1 Pex Cheat Page 4
2 Introduction 7
3 Unit testing today 7

3.1 What are unit tests? . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
3.2 Benefits of unit testing . . . . . . . . . . . . . . . . . . . . . . . . . 9
3.3 A critique of unit testing . . . . . . . . . . . . . . . . . . . . . . . . 9
1
3.4 Measurement of test quality: code coverage and assertions . . . . . . 10
3.5 Unit testing in .Net . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
3.5.1 Unit test frameworks . . . . . . . . . . . . . . . . . . . . . . 11
3.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
3.6.1 Exercise 1: Getting started with Visual Studio 2008 Profes-
sional (or better) . . . . . . . . . . . . . . . . . . . . . . . . 11
3.6.2 Exercise 2: Unit testing the Luhn Algorithm using Test Driven
Development . . . . . . . . . . . . . . . . . . . . . . . . . . 17
4 Parameterized unit testing 21

4.1 Separation of concerns . . . . . . . . . . . . . . . . . . . . . . . . . 23
4.2 Coverage through test input generation . . . . . . . . . . . . . . . . . 23
4.3 Theory of parameterized unit tests . . . . . . . . . . . . . . . . . . . 25
4.4 Patterns for parameterized unit testing . . . . . . . . . . . . . . . . . 25
4.5 Test driven development by parameterized unit testing . . . . . . . . . 25
5 Selecting test inputs for parameterized unit tests 26

5.1 Automated exploratory testing . . . . . . . . . . . . . . . . . . . . . 26
5.1.1 exploratory testing example . . . . . . . . . . . . . . . . . . 26
5.2 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
5.2.1 Exercise 3: Getting started with Pex in Visual Studio . . . . . 28
5.2.2 Exercise 4: Using Pex from the Command Line . . . . . . . . 31
5.2.3 Exercise 5: Parameterized Unit testing for the Luhn Algorithm 38
5.2.4 Exercise 6: Instrumentation Configuration . . . . . . . . . . . 40
5.2.5 Exercise 7: Regular Expressions (optional) . . . . . . . . . . 41
5.2.6 Exercise 8: Binary Encoding (optional) . . . . . . . . . . . . 41
5.2.7 Exercise 9: Compression and Decompression (optional) . . . 42
5.2.8 Exercise 10: String Interview Question (optional) . . . . . . . 43
6 Dealing with the environment 43

6.1 Unit testing is not integration testing . . . . . . . . . . . . . . . . . . 43
6.2 Mocks, stubs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
6.3 Stubs framework in Pex . . . . . . . . . . . . . . . . . . . . . . . . . 45
6.4 Parameterized mock objects . . . . . . . . . . . . . . . . . . . . . . 45
6.5 Parameterized mock objects with assumptions . . . . . . . . . . . . . 46
6.6 Dependency injection . . . . . . . . . . . . . . . . . . . . . . . . . . 47
6.6.1 Credit card validation client . . . . . . . . . . . . . . . . . . 47
6.6.2 A first attempt . . . . . . . . . . . . . . . . . . . . . . . . . 47
6.6.3 Extracting dependencies . . . . . . . . . . . . . . . . . . . . 48
6.6.4 Injecting dependencies . . . . . . . . . . . . . . . . . . . . . 49
7 Background on white box software testing (advanced) 50

7.1 Analysis techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
7.2 Specification techniques . . . . . . . . . . . . . . . . . . . . . . . . 51
7.3 The testing problem . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
7.4 White-box test input generation . . . . . . . . . . . . . . . . . . . . . 51
2
7.5 Symbolic execution . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
7.6 Dynamic symbolic execution . . . . . . . . . . . . . . . . . . . . . . 52
7.7 Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
8 Understanding Pex (advanced) 54

8.1 Why dynamic symbolic execution . . . . . . . . . . . . . . . . . . . 54
8.1.1 Exercise 11: Understanding the Path Condition . . . . . . . . 55
8.2 Monitoring by instrumentation . . . . . . . . . . . . . . . . . . . . . 56
8.3 Symbolic state representation . . . . . . . . . . . . . . . . . . . . . . 57
8.3.1 Exercise 12: Heap Constraints . . . . . . . . . . . . . . . . . 58
8.4 Search strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
8.4.1 Exercise 13: Search Frontiers . . . . . . . . . . . . . . . . . 59
8.5 Constraint solving . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
8.5.1 Exercise 14: Constraint Solving . . . . . . . . . . . . . . . . 60
8.6 Implicit branches . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
8.6.1 Exercise 15: Implicit Branches . . . . . . . . . . . . . . . . . 61
8.7 Assumptions and assertions . . . . . . . . . . . . . . . . . . . . . . . 61
8.8 When does a test case fail? . . . . . . . . . . . . . . . . . . . . . . . 62
8.9 When does Pex emit a test case? . . . . . . . . . . . . . . . . . . . . 63
8.9.1 Exercise 16: Test case emission filters . . . . . . . . . . . . . 64
8.10 When does Pex stop? . . . . . . . . . . . . . . . . . . . . . . . . . . 64
8.11 How does Pex suggest fixes for errors? . . . . . . . . . . . . . . . . . 67
8.12 Creating complex objects . . . . . . . . . . . . . . . . . . . . . . . . 67
8.13 Explorables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
8.14 Invariants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
8.15 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
9 Related work 71
10 Conclusion 72
3
1 Pex Cheat Page
Getting Started
Add Pex reference Microsoft.Pex.Framework.dll
Bind test project [assembly: PexAssemblyUnderTest("UnderTest")]
Custom Attributes
PexClassAttribute marks a class containing PUT
PexMethodAttribute marks a PUT
PexAssumeNotNullAttribute marks a non-null parameter
PexAssumeUnderTestAttribute marks a non-null and precise type parameter
using Microsoft.Pex.Framework;
[PexClass(typeof(Foo))]
public partial class FooTest {
[PexMethod]
public void Bar([PexAssumeNotNull]Foo target, int i) {
target.Bar(i);
}
}
Static Helpers
PexAssume evaluates assumptions (input filtering)
PexAssert evaluates assertions
PexObserve logs live values to the report and/or generated tests
PexChoose generates new choices (additional inputs)
[PexMethod]
void StaticHelpers(Foo target){
PexAssume.IsNotNull(target);
int i = PexChoose.Value<int>("i");
string result = target.Bar(i);
PexObserve.ValueForViewing("result", result);
PexAssert.IsNotNull(result);
}
Instrumentation
PexInstrumentAssemblyAttribute specifies to instrument an assembly
PexInstrumentTypeAttribute specifies to instrument a type
PexAssemblyUnderTestAttribute binds a test project to a project
[assembly: PexAssemblyUnderTest("MyAssembly")]
[assembly: PexInstrumentAssembly("Lib")]
[assembly: PexInstrumentType(typeof(Foo))]
4
PexAssume and PexAssert
PexAssume filters the input, PexAssert checks the behavior. Each method may have a number of overloads.
Basic
Fails unconditionally Fail()
c is true IsTrue(bool c)
c is false IsFalse(bool c)
Treats test as inconclusive Inconclusive()
Implication
p holds if c holds ImpliesIsTrue(bool c, Predicate p)
p holds if c holds (case-split) Case(bool c).Implies(Predicate p)
Nullarity
o is not null IsNotNull(object o)
a is not null or empty IsNotNullOrEmpty<T>(T[] a)
a elements are not null AreElementsNotNull<T>(T[] a)
Equality
expected is equal to actual AreEqual<T>(T expected, T actual)
l and r elements are equal AreElementsEqual<T>(T[] l, T[] r)
5
expected is not equal to actual AreNotEqual<T>(T expected, T actual)
Reference Equality
expected is same as actual AreSame(expected, actual)
expected is not the same as actual AreNotSame(expected, actual)
Type Equality
o is an instance of t IsInstanceOfType(object o, Type t)
o is not an instance of t IsNotInstanceOfType(object o, Type t)
Over collections
p holds for all elements in a TrueForAll<T>(T[] a, Predicate<T> p)
p holds for at least one element in a TrueForAny<T>(T[] a, Predicate<T> p)
Exceptional Behavior (PexAssert only)
action throws an exception of type TException Throws<TException>(Action action)
action succeeds or throws an exception of type TException ThrowAllowed<TException>(Action action)
Behavior Equality (PexAssert only)
l and r behave the same AreBehaviorsEqual<T>(Func<T> l, Func<T> r)
returns the result or exception resulting from the execution of f Catch<T>(Func<T> f)
PexChoose
Make choices, effectively adding new test parameters on the fly. Choices get serialized in the generated test code.
Value Choices
any value of type T PexChoose.Value<T>(string description)
any non-null value of type T PexChoose.ValueNotNull<T>(string description)
any valid enum value of type T PexChoose.EnumValue<T>(string description)
Range Choices
any value from a PexChoose.ValueFrom<T>(string description, T[] a)
any integer value within a [min, max) PexChoose.ValueFromRange(string description, int min, int max)
any index for a PexChoose.IndexValue<T>(string description, T[] a)
6
2 Introduction
Unit tests are becoming increasingly popular. A recent survey at Microsoft indicated
that 79% of developers use unit tests [50]. Unit tests are written to document customer
requirements, to reflect design decisions, to protect against changes, but also, as part of
the testing process, to produce a test suite with high code coverage that gives confidence
in the correctness of the tested code.
The growing adoption of unit testing is due to the popularity of methods like XP
(extreme programming) [6], test-driven development (TDD) [5], and test execution
frameworks like JUnit [27], NUnit [35] or MbUnit [26]. XP does not say how and
which unit tests to write. Moreover, test execution frameworks automate only test
execution; they do not automate the task of creating unit tests. Writing unit tests by
hand can be a laborious undertaking. In many projects at Microsoft there are more lines
of code for the unit tests than for the implementation being tested. Are there ways to
automate the generation of good unit tests? We think that Parameterized Unit Testing
is a possible answer, and this is the topic of the tutorial.
We describe how to design, implement and test software using the methodology
of Parameterized Unit Testing [48, 49], supported by the tool Pex [40, 47]. Pex, an
automated test input generator, leverages dynamic [21] symbolic execution [29] to test
whether the software under test agrees with the specification. As a result, software de-
velopment becomes more productive and the software quality increases. Pex produces
a small test suite with high code coverage from Parameterized Unit Tests.
We introduce the concepts and illustrate the techniques with some examples. We
assume deterministic, single-threaded applications.
This document is separated into two main parts. The first part provides detailed
walkthrough exercises on unit testing in Section 3, the methodology of Parameterized
Unit Testing in Section 4, the usage of the Pex tool in Section 5, and ways to deal with
the environment in Section 6.
The second part is for the advanced reader. It provides a background on white box
testing techniques in Section 7, and discusses in detail various aspects of the Pex tool in
Section 8. Section 9 gives an overview of related work. Finally, Section 10 concludes.
An earlier and shorter version of this tutorial on Parameterized Unit Testing with
Pex can be found in [15]. More documentation on Pex can be found on the Pex web-
site [40].
3 Unit testing today

3.1 What are unit tests?
A unit test is a self-contained program that checks an aspect of the implementation
under test. A unit is the smallest testable part of the program. One can partition each
unit test into three parts: (1) exemplary data, (2) a method sequence and (3) assertions.
exemplary data can be considered as the test input that is passed to the methods
as argument values,
7
method sequence based on the data, the developer builds a scenario that usually
involves several method calls to the code-under-test,
assertions encode the test oracle of a unit test. The test fails if any assertion fails
or an exception is thrown but not caught. Many unit test frameworks have spe-
cial support for expected exceptions, which can often be annotated with custom
attributes.
Program 3.1 is an example of a unit test that checks the interplay among some
operations of .NETs ArrayList class. The example is written in C#, omitting the
class context, as we will do often for brevity.
Program 3.1 A unit test for the ArrayList class
public void AddTest() {

// exemplary data
int capacity = 1;
object element = null;
// method sequence
ArrayList list = new ArrayList(capacity);
list.Add(element);
// assertions
Assert.IsTrue(list[0] == element);
}
The AddTest method first sets up its state, by picking the values 1 and null for
capacity and value, respectively.
Then, the test method performs a sequence of method calls, starting by creating
an instance of the ArrayList class with the selected capacity. An array list is a
container whose size may change dynamically. Internally, it uses a fixed-length array as
backing storage. The array lists capacity is the allocated length of its current backing
storage. Next, the test adds the element to the array list.
Finally, the test checks an assertion, making sure that the array list at position 0
contains element. This is the test oracle.
There is often more than one way to partition a unit test into the three different
parts (exemplary data, method sequence, assertions). For example, Program 3.2 is a
very similar test, in which the input data consists of more complex objects, including
an object for the element, and the initial instance of the array list itself.
8
Program 3.2 A typical unit test
public void AddTest2() {

// exemplary data
object element = new object();
ArrayList list = new ArrayList(1);
// method sequence
list.Add(element);
// assertions
Assert.IsTrue(list[0] == element);
}
In the unit testing frameworks NUnit [38, 35] and Visual Studio Unit Test in Visual
Studio 2008 Professional [36], a parameterless method such as AddTest is decorated
with a custom attribute like [TestMethod] to designate it as a unit test. The class in
which unit tests are defined is decorated with an attribute like [TestClass]. Usually,
each unit test explores a particular aspect of the behavior of the class-under-test.
3.2 Benefits of unit testing

Software developers (and sometimes testers) write unit tests for different purposes.
Design and specification Developers translate their understanding of the specification

into unit tests and/or code. Developers following the test-driven development
write the unit tests before code, and therefore use the unit tests to drive the design.
Nonetheless, unit tests may be written in all phases of the software development
process. Also, developers capture exemplary customer scenarios as unit tests.
Code coverage and regression testing Developers or testers may write unit tests to
increase their confidence in the correctness of code that they have already writ-
ten. It is well known that a test suite which achieves high code coverage and
checks many assertions is a good indicator of code quality. In this way, unit tests
represent a safety net that developers can use when refactoring the code. Unit
tests are usually small tests that run fast and give a quick feedback on effects of
code changes. Additionally, several tools exist to automatically execute a suite
of unit tests on each code change that is committed to the source code repository.
Short feedback loop As mentioned above, unit tests are usually written by the devel-
opers themselves before or after writing the product code. When a unit test fails
and exposes a bug, the feedback loop to get the bug fixed is very short.
Documentation The resulting unit tests are commonly considered as documentation

of correct program behavior.
3.3 A critique of unit testing

Unit testing faces several challenges:
9
Quality of unit tests The quality of the unit tests is mostly dependent on the time the
developer is willing to invest in them.
Amount of unit tests Writing more unit tests does not necessarily increase the code
coverage. Therefore, the size of the test suite is not an indicator of the code
quality.
New code with old tests While the unit tests are actively edited when the developer is
implementing the code, such tests are usually not updated later on (besides syn-
tactic corrections when APIs are refactored). Therefore, if the developer changes
the code implementation, for example by adding more special cases in the code,
but does not update the unit tests, he might introduce a number of new untested
behaviors.
Hidden integration test Ideally, a unit test should test the code in isolation. This
means that all environment dependencies, e.g database, file I/O, must be hidden
behind an abstraction layer. During testing, the abstraction layer provides a fake
implementation, also referred as mocks. In practice, it is very easy to leak such
dependencies.
3.4 Measurement of test quality: code coverage and assertions

What is a good test suite? When do you have enough unit tests to ensure a minimum
level of quality? Those are hard questions that developers face.
Our experience within Microsoft and from the industry indicates that a test suite
with high code coverage and high assertion density is a good indicator for code quality.
Code coverage alone is generally not enough to ensure a good quality of unit tests and
should be used with care. The lack of code coverage to the contrary clearly indicates a
risk, as many behaviors are untested.
A statement is covered when at least one unit test executes this statement. The code
coverage is then usually computed by executing the entire unit test suite and computing
the ratio of covered statements.
Different notions of code coverage exist [7]. Here we only discuss those that Pex
employs, namely those that rely on the control flow coverage:
Basic Block Coverage This coverage is based on basic block representation of the
programs control flow graph. A "basic block" is a sequence of instructions, here
the MSIL instructions of .NET, that has one entry point, one exit point, and no
branches within the block. It is commonly used in the industry.
Branch Coverage This coverage is computed by analyzing the coverage of explicit
arcs. An arc is a control transfer from one basic block to another in the program
control flow graph.
Implicit Branch Coverage This is an extension of the arc coverage where all explicit
and implicit arcs are considered. Implicit arcs occur for exceptional behavior
of instructions, for example when accessing a field, the arc that throws a null
dereference exception is implicit.
10
3.5 Unit testing in .Net
This section gives a quick introduction to unit testing in .NET. If you are familiar with
the techniques and tools, you might as well skip it.
3.5.1 Unit test frameworks
Several frameworks exist in .NET to help developers effectively author and run unit
tests. Although each framework has its own particularities, they all provide a core set
of services:
a custom attribute based system for tagging methods as unit tests,
automatic detection and execution of such unit tests,
a runner with reporting capabilities. The runner might be a simple console ap-
plication or an integrated GUI.
In this document, we will use the Visual Studio Unit Test test framework that comes
with Visual Studio.
3.6 Exercises
3.6.1 Exercise 1: Getting started with Visual Studio 2008 Professional (or better)
In this exercise, we go through the steps to create a new test project, author unit tests
and run them. This section is targeted to users who are new to Visual Studio Unit Test
projects in Visual Studio.
Part 1: Creating a New Test Project
1. Go to File|New|Project....
2. On the left pane, select Visual C#|Test, then select the Test Project item. Select
a location for the new project and click Ok.
11
The content of the window shown above might vary depending on your Visual
Studio installation.
3. Delete the sample files that were generated by the project wizard (Authoring-
Tests.txt, ManualTest1.mht, UnitTest1.cs). (You can right-click a file
name, and select Delete in the context menu.)
Part 2: Creating a Passing Unit Test
1. Right-click on the project node and select Add|New Test.
2. In the main pane, select the Unit Test item, update the test name to Hello-
WorldTest.cs and hit Ok.
12
3. Open HelloWorldTest.cs and clean the generated code to have an empty
class definition. This is a test class, tagged with a [TestClass] attribute. Such
a test class is often called a test fixture.
using System;
using System.Text;
using System.Collections.Generic;
using Microsoft.VisualStudio.TestTools.UnitTesting;
namespace TestProject1 {
[TestClass] // this class contains unit tests
public class HelloWorldTest {}
}
4. We start by adding a test that will pass. Add a new public instance method to the
HelloWorldTest class, tagged with the TestMethodAttribute, that writes
"Hello World" to the console:
[TestMethod] // this is a test
public void PassingTest() {
Console.WriteLine("hello world");
}
5. In the Test View window (Test|Windows|Test View), select the PassingTest

test, then click on Run Selected icon (upper left).
6. The test result window displays the status of the current run. Each selected test
is represented as a row in the report. In this case, PassingTest succeeded and
you can review the details of that particular test by double-clicking on the row.
13
7. The test details view gives various metrics about the test run (duration, etc.) as
well as the console output.
Part 3: Creating a Failing Unit Test
1. In the following, we add a test that fails. Add a new public instance method to
the HelloWorldTest class that throws an exception.
[TestMethod]
public void FailingTest() {
throw new InvalidOperationException("boom");
}
Open the Test View and execute both tests,
2. The test result window now contains two test results; one for PassingTest and
one for FailingTest.
3. Go to the FailingTest method and hit F9 to set a debugger breakpoint.
4. To start debugging the failing tests, go to the test result view and click on the
Debug original tests menu item
14
5. The debugger will automatically stop on the breakpoint set previously. If you
are not familiar with the Visual Studio debugger, this is a good time to get some
experience. The yellow line shows the statement that will be executed next.
Part 4: Creating a Negative Unit Test
1. Visual Studio Unit Test supports a special attribute, ExpectedExceptionAttribute,

that specifies that the test must throw an exception of a particular type. You can
use it to write unit tests that check that parameter validation code works prop-
erly.
[TestMethod]
[ExpectedException(typeof(InvalidOperationException))]
public void ExpectedExceptionTest() {
throw new InvalidOperationException("boom");
}
After running the three tests, you can see that the ExpectedExceptionTest
test was marked as a passing test since it threw the expected exception,
Part 5: Enabling Code Coverage (This exercise requires Visual Studio 2008 Test
Edition, or better. Visual Studio 2008 Professional is not sufficient.)
1. Visual Studio Unit Test in Visual Studio comes with a built-in code coverage
support. To enable this feature, go to Test|Edit Test Run Configurations|Local
Test Run.
15
2. On the left pane, select Code Coverage then select the TestProject1.dll
assembly to be instrumented and click Close. Execute PassingTest and Fai-
lingTest, leaving aside ExpectedExceptionTest.
3. In the Test Results window, click on the code coverage icon (last icon on the
right).
4. In the Code Coverage Results window, enable source code coloring.
16
5. Covered code is colored in light blue, while uncovered code is colored in red. In
this example, we did not run the ExpectedExceptionTest. That is why this
method is colored in red.
To summarize, in this exercise, we learned
how to create a new Visual Studio Unit Test test project in Visual Studio,
how to author, execute and debug unit tests,
how to enable code coverage and analyze the results.
3.6.2 Exercise 2: Unit testing the Luhn Algorithm using Test Driven Develop-
ment
In this exercise, we will implement the Luhn validation algorithm using a test driven
development (TDD) approach [5].
The TDD cycle consists of the following short steps:
1. Add a test,
2. Run it and watch it fail,
3. Change the code as little as possible such that the test should pass,
4. Run the test again and see it succeed,
5. Refactor the code if needed.
We apply the TDD steps in the following.
Part 1: Credit Card Number Validation Specification Most credit card companies
use a check digit encoding scheme [52]. A check digit is added to the original credit
card number, at the beginning or the end, and is used to validate the authenticity of the
number. The most popular encoding algorithm is the Luhn algorithm [53] which can
be computed by the following steps:
17
1. Double the value of alternate digits of the primary account number beginning
with the second digit from the right (the first righthand digit is the check digit.)
2. Add the individual digits comprising the products obtained in Step 1 to each of
the unaffected digits in the original number.
3. The total obtained in Step 2 must be a number ending in zero (30, 40, 50, etc.)
for the account number to be validated.
Now that we have a specification for the algorithm, we can start working on the
implementation.
Part 2: Add a Failing Test

1. Right-click on the solution node and select Add|New Project.
2. On the left pane, select Visual C#|Windows then select the Class Library item.
Change the project name to Creditar.
3. In the test project, right-click on the References node and select Add Refer-
ences.
18
4. Select the Projects tab and double-click the Creditar project row to add it as
a reference.
5. We start by writing a first unit test for the Validate method, before writing or
declaring the Validate method itself. Add a first unit test that verifies that the
Validate method throws ArgumentNullException when it receives a null
reference.
[TestMethod]
[ExpectedException(typeof(ArgumentNullException))]
public void NullNumberThrowsArgumentNullException() {
LuhnAlgorithm.Validate(null);
}
6. Right-click on the Creditar project node and select Add|Class.
7. Add a minimal implementation of the Validate method such that the projects
will compile.
public static class LuhnAlgorithm {
public static bool Validate(string number) {
19
return false;
}
}
8. Execute the unit test and make sure that it fails.
Part 3: Run the Unit Test and Watch It Pass
1. Make a minimal change to the Validate implementation such that the unit test
will pass.
if (number == null)
throw new ArgumentNullException("number");
return false;
}
}
2. Repeat the steps above to ensure that when a non-digit character is passed to the
Validate method, the implementation throws a ArgumentException,
[TestMethod]
[ExpectedException(typeof(ArgumentException))]
public void AThrowsArgumentException() {
LuhnAlgorithm.Validate("a");
}
Interestingly, the minimum change to get this test to pass is not really what one
would call a correct implementation, or even a useful implementation.
if (number == null)
if (number == "a")
throw new ArgumentException("number");
return false;
}
}
To write such an implementation really means to follow the incremental idea of

the test-driven development methodology. We could continue writing more unit
tests that will fail, in order to refine our implementation.
Part 4: Continue the iteration
1. Now that we have a passing test suite, we can refactor the code into a smarter
implementation that checks for any non-digit character
20
if (number == null)
foreach (var c in number)
if (!Char.IsDigit(c))
throw new ArgumentException("number");
return false;
}
}
The rest of the unit testing of the Validate method is left as an exercise. Do not
forget to use the code coverage view to ensure that the unit test reach a minimum level
of basic block code coverage. This usually means that executing the unit test suite
yields to a certain percentage of basic block coverage. In the case of this exercise, 80%
is a reasonable goal.
4 Parameterized unit testing

The unit test in Program 3.2 specifies the behavior of the array list by example. Strictly
speaking, this unit test only says that by adding a new object to an empty array list,
this object becomes the first element of the list. What about other array lists and other
objects?
Traditional unit tests do not take inputs. A straightforward extension is to allow
parameters. The result is a Parameterized Unit Test (PUT), which one can partition into
four parts: (1) parameters, (2) assumptions, (3) a method sequence and (4) assertions.
parameters represent the test input which is later passed on to other methods as
their argument values,
assumptions over the parameters can be used to shape legal test inputs,
method sequence specify a scenario (as before), and
assertions encode the test oracle of a unit test (as before).
Here is a parameterized version of the array list unit test that describes the normal
behavior of the Add method with respect to two observers, the property Count and the
indexing operator []. Under the condition that a given array list is not null, this PUT
asserts that after adding an element to the list, the element is indeed present at the end
of the list:
21
Tip: Where to get valid credit card numbers? To help you validate your implemen-
tation, you can use this number generator [53]:
public static int[] CreateNumber(int length) {
Random random = new Random();
int[] digits = new int[length];
// Set all but the last digit to a random number;
// the last digit remains zero
for (int i = 0; i < length - 1; i++) {
digits[i] = random.Next(10);
}
int sum = 0;
bool alt = true;
for (int i = length - 2; i >= 0; i--) {
if (alt) {
int temp = digits[i];
temp *= 2;
if (temp > 9) {
temp -= 9;
}
sum += temp;
}
else {
sum += digits[i];
}
alt = !alt;
}
int modulo = sum % 10;
if (modulo > 0) {
digits[length - 1] = 10 - modulo;
}
return digits;
}
22
Program 4.1 A parameterized unit test for the ArrayList class
public void AddSpec2(

// parameters
ArrayList list, object element) {
// assumptions
PexAssume.IsTrue(list != null);
// method sequence
int len = list.Count;
list.Add(element);
// assertions
PexAssert.IsTrue(list[len] == element);
}
This test is more general than the original test. PUTs like this one can be called with
various input values, perhaps drawn from an attached database. Unit testing frame-
works that support PUTs sometimes refer to them as data-driven tests (for example in
[2]).
Instead of stating the exemplary data values explicitly, a PUT may state assump-
tions about how valid input data must look like. Here, we assume that the list is not
null.
PUTs are more general specifications than traditional unit tests: PUTs state the
intended program behavior for entire classes of program inputs, and not just for one
exemplary input. And yet PUTs are still easy to write since they merely state what the
system is supposed to do, and not how to accomplish the goal.
Unlike many other forms of specification documents, PUTs are written on the level
of the actual software APIs, in the programming language of the software project. This
allows PUTs to evolve naturally with the code against which they are written.
4.1 Separation of concerns

Splitting the specification and test cases by parameterized unit testing is a separation
of concerns:
Firstly, we specify the intended external behavior of the software as PUTs; only
human beings can perform this specification task.
Secondly, a tool like Pex can automatically create a test suite with high code
coverage by determining test inputs which exercise different execution paths of
the implementation.
4.2 Coverage through test input generation

Adding parameters to a unit test improves its expressiveness as a specification of in-
tended behavior, but we lose concrete test cases. We can no longer execute a parame-
terized test by itself. We need actual parameters. But which values must be provided
to ensure sufficient and comprehensive testing? Which values can be chosen at all?
23
Consider the following code that implements Add and the indexing operator in the
.NET base class library.
Program 4.2 ArrayList implementation in .NET
public class ArrayList

...
{
private Object[] _items = null;
private int _size, _version;
...
public virtual int Add(Object value) {
if (_size == _items.Length) EnsureCapacity(_size + 1);
_items[_size] = value;
_version++;
return _size++;
}
}
There are two cases of interest. One occurs when adding an element to an array
list that already has enough room for the new element (when the array lists capacity is
greater than the current number of elements in the array list). The other occurs when
the internal capacity of the array list must be increased before adding the element.
We can assume that the library methods invoked by the ArrayList implemen-
tation are themselves correctly implemented (EnsureCapacity guarantees that the
_items array is resized so its length is greater or equal _size + 1), and we do not
consider possible integer overflows.
Then we only need to run two test cases to check that the assertion embedded in
AddSpec2 holds for all array lists and all objects given the existing .NET implemen-
tation. Two test cases are needed as there are only two execution paths through the
Add method shown above; accordingly, all inputs can be partitioned into two equiva-
lence classes: one where _size == _items.Length holds, and one where it does
not hold. Each of the two test cases below is a representative of one of the two equiva-
lence classes.
[TestMethod]
public void TestAddNoOverflow() {
AddSpec2(new ArrayList(1), new object());
}
[TestMethod]
public void TestAddWithOverflow() {
AddSpec2(new ArrayList(0), new object());
}
No other inputs are needed to test all behaviors of Add, since any other input will
execute exactly the same paths as the two inputs mentioned above.
24
4.3 Theory of parameterized unit tests
By adding parameters we turn a closed unit test into a universally quantified condi-
tional axiom that must hold for all inputs under specified assumptions. Intuitively, the
AddSpec2(. . .) method asserts that for all array lists a and all objects o, the following
holds:
ArrayList a, object o.
(a 6= null) let i = a.Count in a.Add(o) , a[i] == o
where , represents sequential composition from left to right: (f , g)(x) = g(f (x))1 .
See [48] for more background information on PUTs, and [8] for an overview of the
theory and practice of algebraic specifications.
4.4 Patterns for parameterized unit testing

As PUTs are really just a way to write algebraic specifications as code, many standard
patterns for algebraic specifications can be applied in the context of parameterized unit
testing.
We have collected a set of such patterns in the context of Pex, which can be found
in the documentation section of the Pex website [40].
4.5 Test driven development by parameterized unit testing

Test Driven Development [5] (TDD) is the activity of programming where all written
code is preceeded by writing tests which specify the intended functionality of the code.
In TDD, the main purpose of writing tests is to drive the design of the API of the
code. It is simply a side effect that the result is a test suite that can serve as docu-
mentation and specification of the intended behavior of the code. It is straightforward
to extend TDD to PUTs. In fact, it is usually more expressive to state the intended
properties of an API with PUTs instead of closed unit tests with exemplary data.
When writing PUTs, the TDD process is as follows:
1. The developer writes or changes a parameterized unit test (PUT), or code that
implements the behavior described by already existing PUTs.
2. The developer runs Pex on the PUT.
3. If Pex finds errors, the developer goes back to step 1, in order to change the PUT
or fix the code, possibly with Pex Fit-It feature.
4. The developer keeps the generated tests for future regression testing.
1 The axiom becomes more complicated when we specify side effects of sequential code precisely. We
explain later how to model references to objects on the heap, see Section 8.3.
25
5 Selecting test inputs for parameterized unit tests
Classic unit tests are methods without parameters, parameterized unit tests are methods
with parameters. In fact, parameterized unit tests have been around for a while now.
They were already supported by Visual Studio Unit Test in Visual Studio (referred to as
data-driven tests), and by MbUnit (referred to as row tests) starting from version one,
and they were recently added to JUnit [27] (referred to as theories).
Yet, it used to be the case that the user had to provide the input parameters for those
tests, as ranges, spreadsheet or database of some sort. Improper choice of inputs would
lead to missed corner cases or a hugely redundant test suite.
With Pex, things change for the better: the user does not have to provide any input
to the parameterized unit tests. By analyzing the program behavior at runtime, Pex can
generate inputs that matter, in the sense that those inputs will increase the coverage of
the test suite (advanced readers can refer to Section 8 for further details).
5.1 Automated exploratory testing

Selecting meaningful test inputs requires a certain understanding of the code under test,
which in turn requires an understanding of what the relevant parts of the (potentially
huge) code under test are.
Exploratory Testing [28, 3] (ET) is an incremental process during which the tester
learns more and more about the actual behavior of the code. Another characteriza-
tion of ET is that it is test design and test execution at the same time. Together with
experience and creativity the tester can craft more and better tests.
Pex uses dynamic symbolic execution [21], a technique that works in a way similar
to ET. The tool executes the code multiple times and learns about the program behavior
by monitoring the control and data flow. After each run, it picks a branch that was not
covered previously, builds a constraint system (a predicate over the test inputs) to reach
that branch, then uses a constraint solver to determine new test inputs, if any. The
test is executed again with the new inputs, and this process repeats. On each run,
Pex might discover new code and dig deeper into the implementation. In this way,
Pex explores the behavior of the code. We also refer to this process as Automated
Exploratory Testing.
For the advanced reader, Section 8 discusses the theoretical foundations and tech-
nical details of dynamic symbolic execution.
5.1.1 exploratory testing example
We will apply exploratory testing to test a simple method that takes two integers as an
input and prints different strings to the console based on those values. We will manually
go through all the steps involved in the analysis. (Pex would perform a similar analysis,
only fully automatically.)
26
Program 5.1 A method to explore
1 void Bar(int i, int j) {

2 if (i < 0) {
3 Console.WriteLine("line 3");
4 if (j == 123)
6 }
7 else
9 }
One way to explore the possible behaviors of this method is to throw different
values at Bar and analyze the output to see what is happening.
Iteration 1: pick arbitrary value Let us create a unit test that does exactly that and
step into the debugger. Since we do not really know anything about the Bar method
yet, we simply pick 0 for i and j.
[TestMethod]
void Zero() {
Foo.Bar(0, 0);
}
When we reach the statement Console.WriteLine("line 8"); on line 8, we
can figure out that we took this branch because the condition i < 0 on line 2 evaluated
to false. We remember this and let the execution continue (and the test finishes
successfully).
line 2, i 0, uncovered branch
Iteration 2: flip the last condition In the previous run, we have remembered that
some code was not covered on line 3. We also know that this code path was not covered
because the condition i < 0 evaluated to false. At this point, we usually intuitively
figure out a value of i in our head, to make this condition true. In this case, we need
to solve find i such that i < 0. Let us pick 1.
[TestMethod]
void MinusOne() {
Foo.Bar(-1, 0);
}
We run the test under the debugger. As expected on line 2, the condition evaluates
to true, and the program takes the other branch (compared to the one taken in the
previous test). The program continues and reaches line 4 where another if-statement is
evaluated. The condition j = 123 evaluates to false, so we remember the following:
line 4, j 6= 123, uncovered branch
The program continues to run and finishes.
27
Iteration 3: path condition + flipped condition There are still some uncovered
branches to cover in the method, guarded by the condition at line 4. To be able to cover
this code, we need two things:
1. reach line 4: i < 0
2. make the condition in line 4 evaluate to true: j = 123
So to cover the last statement in the method, we need to find parameter values such
that i < 0 j = 123.
Let us pick i = 1 and j = 123.
[TestMethod]
void MinusOneAndOneTwoThree() {
Foo.Bar(-1, 123);
}
The test executes and prints line 5 as we wanted. At this point, we have fully
covered the behavior of Bar.
5.2 Exercises
5.2.1 Exercise 3: Getting started with Pex in Visual Studio
Part 1: Adding Pex to a Project
1. Add a reference to the Microsoft.Pex.Framework.dll assembly to the

test project. In the Add Reference dialog, select the .NET pane, then scroll
down to Microsoft.Pex.Framework,
Part 2: Creating a Parameterized Unit Test
1. In the HelloWorldTest, add a new public instance method Parameterized-

Test that takes an int parameter. Mark this method with the PexMethod-
Attribute, written as [PexMethod] in C#.
[PexMethod]
public void ParameterizedTest(int i) {
if (i == 123)
throw new ArgumentException("i");
}
28
Tip: Adding Using Clauses Automatically Visual Studio can automatically add the
missing using clause: Move the editing caret on the [PexMethod] attribute. You will
see a little red rectangle, located at the bottom right of PexMethod. Click on the little
red rectangle. You get a context menu with two entries: The first will add a using
clause to your source code, and the second will embed the fully qualified namespace
to this occurrence of PexMethod. Just press Enter to insert the using clause. The
keyboard shortcut to open this context menu is Ctrl + ..
Tip: Using Snippets Pex also provides snippets to create a new parameterized unit test
skeleton. In the editor, write pexm and then press the tab-key. This will expand the
snippet.
Part 3: Run the Parameterized Unit Test

1. Move the mouse cursor inside the ParameterizedTest method, right-click
and select the Run Pex Exploration menu item.
2. Pex automatically displays the Pex Results window. Most of your interactions
with Pex will be through this window.
3. Each row in the table corresponds to a generated test for the current exploration.
Each row contains
29
an icon describing the status of the test (passing, failing),
a number indicating how often Pex had to execute the parameterized unit
test with different input values in order to arrive at this test,
a summary of the exception that occurred, if any
Pex also automatically logs the values of the input parameters of the test. Note
that often Pex runs the parameterized unit test several times until Pex outputs
the next test. The rationale behind this behavior is that Pex explores different
execution paths of the program, but it only outputs a new test when this test
increases the coverage (arc coverage, to be precise; see Section 8.9 for more
details). Many execution paths may result in the same coverage.
4. When exploring the ParameterizedTest that we wrote earlier, Pex generates

two unit tests. Each unit test can be accessed by selecting the corresponding row
and clicking on the Go to generated test link.
[TestMethod]
[PexGeneratedBy(typeof(TestClass))]
public void ParameterizedTest01() {
this.ParameterizedTest(0);
}
[TestMethod]
[PexGeneratedBy(typeof(TestClass))]
[PexRaisedException(typeof(ArgumentException))]
public void ParameterizedTest02() {
this.ParameterizedTest(123);
}
The TestMethod attribute indicates that the generated methods are unit tests, the
PexGeneratedBy attribute indicates that the test was generated by Pex by ex-
ploring parameterized unit tests in a particular test class, and the PexRaisedException
indicates that this test raised an (unexpected) exception.
5. The generated unit tests are written to a separate file that is located in the same
directory as HelloWorldTest.cs, the file that contains the parameterized
unit test. In the Visual Studio folder view of the project the generated file shows
up as an item of HelloWorldTest.cs:
Tip: Generated File Names The generated file name consists of the original file name
(HelloWorldTest, the name of the parameterized unit test from which the tests
were generated (here .ParameterizedTest), an identifier that indicates that this
is a generated file (.g), and the usual extension for C# code (.cs). This makes it easy
to differentiate and sort them outside of Visual Studio.
30
5.2.2 Exercise 4: Using Pex from the Command Line
In Section 5.2.1 we learned how to use Pex in the Visual Studio environment. In this
exercise, we will learn how to drive Pex from the command line and how to read the
HTML reports generated by Pex.
Before running Pex, you should have a .NET assembly (a .NET executable, end-
ing in .dll or .exe) that contains a class annotated with the PexClassAttribute,
containing a public instance method annotated with the PexMethodAttribute. We
will use the following example, which corresponds to the example that we created with
Visual Studio in Exercise 3.6.1.
Program 5.2 Self-contained parameterized unit test code
using System;
[PexClass]
public partial class TestClass {
[PexMethod]
public void ParameterizedTest(int i) {
if (i == 123)
throw new ArgumentException("i");
}
}
You can build it with the standalone C#-compiler csc.exe, Visual C# 2005/2008
Express Edition, or any other C# development environment. In any case, you need to
reference the Microsoft.Pex.Framework assembly.
Tip: Other test frameworks Pex works best with the unit test framework of Visual
Studio, but Pex can also generate tests for other unit test frameworks. Pex detects
the intended unit test framework by inspecting the referenced assemblies in your test
project.
We run Pex from the command line as follows.
1. Open a command prompt window.
2. Go to the build folder of the test project,
3. Apply the Pex command line program, pex.exe, on the test project assembly.
This will run all the parameterized unit tests in that assembly (you may need to
change name of the .dll file as appropriate):
> pex bin\Debug\TestProject1.dll
Pex runs and logs its activity to the console. At the end of the run, it automatically
opens a detailed HTML report.
31
Tip: Getting the path of a document in Visual Studio One can get the full path of
an open document in Visual Studio by right-clicking on the tab and select Copy Full
Path.
Part 1: Command Line Output The console output contains different aspects. The
precise output may vary, depending on your version of Pex and .NET.
1. Pex inspects the test project and launches a second process. (In this process, Pex
acts as a profiler in order to instrument code so that Pex can monitor a taken
execution path precisely.)
1 Microsoft Pex v0.18 -- http://research.microsoft.com/pex

2 Copyright (c) Microsoft Corporation 2007-2009.
3 All rights reserved.
4
5 instrumenting...
2. At this point, the instrumented process is running. Pex is loading the parameter-
ized tests. Any issue in the test metadata will be logged during this phase.
6 launched Pex x86 Edition on .NET v2.0.50727

7 [reports] report path:
8 reports\TestProject1.71115.170912.pex
9 00:00:00.0> starting execution
10 00:00:00.1> reflecting tests
3. In the next phase, Pex explores all parameterized unit tests.
11 00:00:02.8> TestProject1
12 00:00:02.8> TestProject1.TestClass
13 00:00:12.3> TestClass.ParameterizedTest(Int32)
4. At each new test, Pex writes a line that describes the test:
the test ParameterizedTest02 was generated on the second run and raised a
ArgumentException.
14 [test] (run 1) ParameterizedTest00 (new)

15 [coverage] coverage increased from 0 to 2 blocks
16 [test] (run 2) ParameterizedTest01 (new),
17 Exception: Exception of type System.Exception was thrown.
18 [coverage] coverage increased from 2 to 4 blocks
32
Tip: How does Pex name generated tests? The generated test names are created by
mangling the exploration method name, plus a number.
5. At the end of the exploration, Pex also gives a basic block coverage summary.
19 [dynamic coverage] 4/4 block (100.00 percent)
6. Once all parameterized unit tests have been processed, Pex displays some statis-
tics, then renders the HTML report. In this run, seven tests were generated, one
of which was considered as a failing test:
20 00:00:13.2> [finished]
21 -- 2 generated tests, 1 failing, 2 new
No errors or warnings were logged by the Pex infrastructure:
22 -- 0 critical errors, 0 errors, 0 warnings
Every time you run Pex, it generates a new folder in the reports directory
to hold all the generated files (and by default, the folders get recycled after a
while; in other words, the generated reports have only a limited lifetime, and you
should copy a report that you want to preserve). The generated folder name is
constructed from the test assembly name and a time stamp.
Tip: Where are the generated tests? Pex creates a subdirectory, here it would be
reports\TestProject1.71115.170912.pex\tests, which contains the gener-
ated tests as code.
23 [reports] generating reports...

24 [reports] html report:
25 reports\TestProject1.71115.170912.pex\pex.html
7. Finally, Pex displays the overall verdict, success or failure.
26 EXPLORATION SUCCESS
Part 2: Command Line Filters The command line supports a rich set of filters to
select which exploration to run; by namespace, by type name, by method name, by
category, etc.
by namespace,
pex /nf:TestProject ...
by type name,
pex /tf:HelloWorld ...
33
by method name,
pex /mf:Add ...
by test suite,
pex /sf:checkin ...
the explorations that had errors in the last runs,

pex /lef ...
of course, you can combine all those filters together,

pex /nf:TestProject /tf:HelloWorld /mf:Add ...
By default, a partial case-insensitive string match is used. To have a precise match,

add a bang (!) at the end of the filter:
pex /tf:HelloWorldTest!
Part 3: Using Pex HTML report The HTML reports gives a tabular view of each
assembly/fixture/exploration. Overall statistics such as the number of fixtures, gener-
ated tests, etc. are highlighted in the title,
1. Each exploration has a little menu to display associated data.
2. The parameter values displays a table where each row contains the values of
the parameters for a particular test. This table is very similar to the Pex Results
view in Visual Studio.
34
Tip: Do I have access to the HTML reports in Visual Studio? To enable HTML
report generation, go to Tools|Options then locate the Pex - General category, click
on the Reports option.
You need to run Pex again to actually generate a report. Afterwards, to open the HTML
report, click on the Open Report link in the yellow bar.
35
3. The log item opens a detailed activity log of the exploration process. The log is
a very important tool to understand in detail which events Pex encountered, and
you will get familiar with it soon.
4. The log of this test starts with a repro command that can be used to execute
that particular exploration from the command line. Then, the generated tests are
logged.
5. The coverage link opens the code coverage report document.
6. You can browse through the coverage data by assemblies, fixture and methods.
36
7. You can also view the coverage by source files.
8. Each source file is colored according to the legend found at the top of the page.
user code under test (covered) Covered statements of the code-under-test.
user code under test (uncovered) Statements of the code-under-test that were
not covered.
user code or test (covered) Statements that were covered, but that were not part
of the code-under-test.
user code or test (uncovered) Statements that were neither covered nor under
test.
tagged Statements which were covered, and which were tagged with some in-
teresting information during the exploration process. Hover with the mouse
over a tagged statement to see more information.
We will later discuss how to configure which parts of the code are under test.
9. In this particular case, the ParameterizedTest method was fully covered,
37
5.2.3 Exercise 5: Parameterized Unit testing for the Luhn Algorithm
We revisit the Luhn algorithm that we tested in Section 3.6.2 and write parameterized
unit tests for it.
Part 1: Setting Up the Code Instrumentation
1. Add a parameterized unit test that specifies that a valid credit card number, ac-
cording to the Luhn algorithm, does not contain any non-digit characters.
[PexMethod]
public void ContainsNumbers(string number) {
PexAssume.IsTrue(LuhnAlgorithm.Validate(number));
PexAssert.TrueForAll(number, delegate(char c) {
return char.IsDigit(c);
});
}
2. Run the test and analyze the results. As you may see, Pex only generated a single
(trivial) test with a null reference as the number string.
3. The problem is that Pex did not analyze the Validate method. (Pex can only
generate relevant test inputs for those portions of the code which get instru-
mented and can be monitored, and since the instrumentation is quite expensive,
Pex does not instrument all code by default.) You need to tell Pex which assem-
blies or types need to be instrumented. This is a process that is usually done
once.
In our case, the product assembly is not instrumented, thus Pex could not build
the constraints to explore the behavior of the Validate method, which eventu-
ally leads to poor code coverage of the generated test suite.
4. Pex shows important issues such as calls to methods that were not instrumented
in a red bar.
5. Click on the link Uninstrumented Methods to see the list of relevant methods
that were not instrumented. When you click on a method in the list, Pex offers
several actions that can be taken to get rid of the warning. Since we want to test
the Validate method, you want to click on the Instrument assembly link.
38
6. This action will add a custom attribute in the test project that tells Pex that the
product assembly should be instrumented:
[assembly: PexInstrumentAssembly("Creditar")]
Pex adds a special file PexAssemblyInfo.cs to hold the assembly level

attributes. (Pex persists all project-specific settings as attributes.)
7. Run Pex again and analyze the results. The coverage should be much better.
Part 2: Binding the Tests and the Code Under Test You should tell Pex what is
the code under test, so that Pex can focus its analysis on that code, and show a relevant
dynamic coverage graph. Pex provides several ways to specify this information.
1. The PexClassAttribute has a special constructor that takes a type. This con-
structor can be used to specify the type under test of a particular test fixture. This
information is used by Pex to prioritize the exploration. Update the LuhnAlgo-
rithmTest class to specify that it is testing the LuhnAlgorithm class.
[TestClass, PexClass(typeof(LuhnAlgorithm))]
public partial class LuhnAlgorithmTest {
2. Because we know that the project Creditar.Tests is the test project for Cre-
ditar, we can add a custom attribute that will provide this information to Pex.
[assembly: PexAssemblyUnderTest("Creditar")]
It is an assembly-level attribute that must be placed before the namespace dec-

laration in C#.
Part 3: Using the Wizard to Get Started Pex provides a code generation wizard
that can automatically produce parameterized test stubs for the entire public API of a
class. These stubs can be used as a starting point to write more elaborate scenarios.
1. Right-click inside the LuhnAlgorithm class and select Pex|Create Parameter-

ized Unit Test Stubs
39
2. The Pex wizard will compile and analyze the LuhnAlgorithm class to produce
the test stubs. The generated files will automatically be added to the test project
using the information provided by the PexAssemblyUnderTestAttribute at-
tribute.
5.2.4 Exercise 6: Instrumentation Configuration

Pex can only generate a test suite with high code coverage if Pex monitors the relevant
parts of the code. Therefore, it is most important to configure correctly which types
Pex should instrument.
Consider the following parameterized unit test.
[PexMethod(MaxBranches = 2000)]
public void Test(string s) {
DateTime dt = DateTime.Parse(s);
PexObserve.ValueForViewing("dt", dt);
}
The MaxBranches setting makes sure that Pex does not stop too early. We will explain
these bounds later, see Section 8.10.
When Pex generates tests, it will only generate a single test at first. However, we
do get a warning that some methods were not instrumented.
When you click on Uninstrumented Methods, you see in the log view the list of
uninstrumented methods.
You can select one of them, and click on the link Instrument type to tell Pex that it
should instrument this type in the future. (If you know that the code is irrelevant to the
code that we want to cover, you can also click on Ignore uninstrumented method.)
Pex will insert custom attributes such as the following for you.
40
using Microsoft.Pex.Framework.Instrumentation;
[assembly: PexInstrumentType("mscorlib", "System.DateTimeParse")]
[assembly: PexInstrumentType("mscorlib", "System.__DTString")]
After you instruct Pex to instrument a type, you have to re-run Pex to see the ef-
fects of more instrumentation. In turn, you might get more uninstrumented method
warnings.
Can you determine the set of types that Pex must instrument in order to generate
a valid DateTime string? It is usually easier when you want to test your own code
whose structure you know. Hint: Work in a loop, where you add one type to the list of
instrumented type which has the word DateTime in the name, and then re-run Pex to
see what changed.
Tip: Coverage filtering The type DateTime is defined in mscorlib. Pex normally
does not report coverage data achieved in this basic library. We can tell Pex to include
coverage data with the following assembly-level attribute:
using Microsoft.Pex.Framework.Coverage;
using Microsoft.ExtendedReflection.Coverage;
[assembly: PexCoverageFilterAssembly(
PexCoverageDomain.UserCodeUnderTest,
"mscorlib")]
5.2.5 Exercise 7: Regular Expressions (optional)

Try out your favorite regular expression and see if Pex can generate an input string that
matches that expression:
[PexMethod]
public void MatchRegex([PexAssumeNotNull]string value) {
if (Regex.IsMatch(value, @"^[0-9a-fA-F]+[\r\n]*$"))
Console.WriteLine("found it!");
}
The [PexAssumeNotNull] attribute indicates that Pex should pass null for the
value parameter. This is a simple case of an assumption on test inputs. See Section 8.7
for more details.
5.2.6 Exercise 8: Binary Encoding (optional)

An encoding is a reader-writer between byte arrays and char arrays.
A simple example of encoding simply copies the bits of a char to two byte, and
vice versa.
The .NET Encoding class is defined in the System.Text namespace and has four
abstract methods that need to be overloaded.
Using the following PUT, write an implementation for GetChars and GetChar-
Count
41
[PexMethod]
public void CharCountMatching(
[PexAssumeNotNull]Encoding encoding,
byte[] bytes,
int index,
int count) {
int charCount = encoding.GetCharCount(bytes, index, count);
char[] chars = encoding.GetChars(bytes, index, count);
Assert.AreEqual(charCount, chars.Length);
}
You can start from the Program 5.3 which contains a sloppy implementation of
those methods.
Program 5.3 A sloppy binary encoding class
public class SloppyBinaryEncoding : Encoding {

const int BytesPerChar = 2;
public override int GetCharCount(

byte[] bytes, int index, int count) {
return count / 2;
}
public override int GetChars(

byte[] bytes, int byteIndex, int byteCount,
char[] chars, int charIndex) {
int j = charIndex;
for (int i = 0; i < byteCount; i += 2) {
chars[j++] =
(char)(bytes[byteIndex + i] << 8 |
bytes[byteIndex + i + 1]);
}
return j - charIndex;
}
5.2.7 Exercise 9: Compression and Decompression (optional)
Write a run-length encoding (RLE) compression and decompression algorithm, that

take arrays of bytes as input and output. Program 5.4 describes the signature of the
RLE implementation.
42
Program 5.4 Signatures of a run-length-encoding compressor
public class RleCompressor {

public static byte[] Compress(byte[] input) {
throw new NotImplementedException();
}
public static byte[] Decompress(byte[] input) {

throw new NotImplementedException();
}
}
The parameterized unit test should simply state the roundtrip property: Compres-
sion followed by decompression should yield the original sequence of bytes.
For example, you can write the following PUT:
[PexMethod]
public void Roundtrip(byte[] data) {
PexAssume.IsNotNull(data, "data");
byte[] compressed = RleCompressor.Compress(data);

byte[] uncompressed = RleCompressor.Decompress(compressed);
// assertions
PexAssert.IsNotNull(uncompressed, "uncompressed");
Assert.AreEqual(data.Length, uncompressed.Length);
for (int i = 0; i < data.Length; i++)
PexAssert.AreEqual(data[i], uncompressed[i]);
}
5.2.8 Exercise 10: String Interview Question (optional)

A classic theme in programming job interviews is on manipulating strings. Here is one
problem that we could test with Pex:
Given an array of characters, with spaces, replace spaces with %20. You have
extra space on the end of the array. (No additional memory and do it as close to O(n)
as possible).
Write an implementation of EscapeSpaces.
Write a PUT and use Pex to validate your implementation.
6 Dealing with the environment

6.1 Unit testing is not integration testing
Each unit test, whether parameterized or not, should test a single feature, so that a
failing unit test identifies the broken feature as concisely as possible. Also, the fewer
43
system components a test interacts with, the faster it will run.
However, in practice it is often difficult to test features in isolation: The code may
take a file name as its input, and use the operating system to read in the contents of the
file. Or the test may need to connect to another machine to fulfill its purpose.
The first step towards making the code testable is to introduce abstraction layers.
For example, the following Parse method is not testable in isolation, since it insists
on interacting with the file system.
public void Parse(string fileName) {
StreamReader reader = File.OpenText(fileName);
string line;
while ((line = reader.ReadLine()) != null) {
...
}
}
The parser in the following code is better testable, since the actual parsing logic can
be driven from any implementation of the abstract StreamReader class. In particular,
it is no longer necessary to go through the file system to test the main code.
public void Parse(string fileName) {
this.Parse(File.OpenText(fileName));
}
public void Parse(StreamReader reader) {

string line;
while ((line = reader.ReadLine()) != null) {
...
}
}
Abstraction from the environment is necessary if you want to have a Pex-testable

design.
6.2 Mocks, stubs

When the code is written with abstraction layers, mock objects [32] can be used to
substitute parts of the program that are irrelevant for a tested feature. Mock objects
answer queries with fixed values similar to those that the substituted program would
have computed.
Today, developers usually define the behavior of mock objects by hand. (By behav-
ior, we mean the return values of mocked methods, what exceptions they should throw,
etc.) Several frameworks [51] exist which provide stubs. Stubs are trivial implemen-
tations of all methods of an interface or a class. Stubs do not perform any actions by
themselves and usually just return some default value. The behavior of the stubs must
still be programmed by the developer. (A capture-replay approach is used in [42] to
distill actual behavior of an existing program into mock objects.)
44
6.3 Stubs framework in Pex
In fact, Pex comes with its own Stubs framework to make writing stubs easier. This
framework also allows to detour legacy code that communicates with the environment
and does not provide an encapsulated abstraction layer. In the remainder of this tuto-
rial, we will not use any framework for mocks or stubs, not even the one that comes
with Pex, but we will define everything required by hand to illustrate the general mech-
anisms. Tutorials and more documentation on how to leverage the stubs framework
that comes with Pexcan be found on the Stubs website [41].
6.4 Parameterized mock objects

When manually writing mock objects, some of the main questions are: What values
should the mock object return? How many versions of a mock object do I need to write
to test my code thoroughly?
We have seen earlier how parameterized unit tests are a way to write general tests
that do not have to state particular test inputs. In a similar way, parameterized mock
objects are a way to write mock objects which do not have just one particular, fixed
behavior.
Consider the method AppendFormat of the StringBuilder class in the .NET
base class library. Given a string with formatting instructions, and a list of values to be
formatted, it computes a formatted string. For example, formatting the string "Hello
{0}!" with the single argument "World" yields the string "Hello World!".
public StringBuilder AppendFormat(
IFormatProvider provider,
string format, object[] args) {
...
}
The first parameter of type IFormatProvider provides a mechanism for retriev-
ing an object to control formatting according to the MSDN documentation:
public interface IFormatProvider {
object GetFormat(Type fmtType);
}
A non-trivial test calling AppendFormat needs an object that implements IFor-
matProvider. While the Stubs framework [41] that comes with Pex can generate
implementations for interfaces automatically, we will illustrate in the following how to
write such an implementation by hand, but leaving to Pex how it should behave.
public class MockFormatProvider : IFormatProvider {
public object GetFormat(Type fmtType) {
return PexChoose.Value<object>("format");
}
}
The mock method GetFormat obtains from the global PexChoose a handle called
call that represents the current method call. The PexChoose provides the values
which define the behavior of the mocked methods, for example their return values.
45
When the test case is executed, ChooseResult will initially return some simple
value, for example null for reference types. Pex symbolic analysis will track how the
value obtained from ChooseResult is used by the program, just as Pextracks all other
test inputs.
Depending on the conditions that are checked on the value obtained from Choose-
Result, Pex will execute the test case multiple times, trying other values that will be
different from null.
You may change the code of the mock type to allow more diverse behavior, for
example adding the choice to throw an exception, perform a callback, or change some
accessible state. For example, you can insert the following lines in the GetFormat
method.
if (PexChoose.Value<bool>("throw"))
throw new Exception();
return PexChoose.Value<object>("format");
The choice whether to throw an exception will cause the exploration to consider two
execution paths. Furthermore, if the caller of GetFormat would distinguish different
exception types, for example by having several catch statements, Pex may explore
even more execution paths.
As mentioned before, Pex will track the usage of the values obtained from Choose-
Result, and may execute the program with several different values to this end. The fol-
lowing call to GetFormat occurs in AppendFormat after checking provider!=null:
cf = (ICustomFormatter)provider
.GetFormat(typeof(ICustomFormatter));
Depending on the result of GetFormat, the cast to ICustomFormatter might fail.
Pex understands this type constraint, and Pex generates a test case with the following
mock object behavior:
var m = new MockFormatProvider();
PexChoose.Replay.Setup()
.DefaultSession
.At(0, "format", m);
Here, Pex creates a mock object and instructs the oracle that during the execution of
a unit test the first call to m.GetFormat should return the mock object itself! (The
test cases that Pex generate are always minimal, this is an example of how Pex tries to
use as few objects as possible to trigger a particular execution path.) This particular
mock object behavior will cause the cast to fail, since MockFormatProvider does not
implement ICustomFormatter.
6.5 Parameterized mock objects with assumptions

Unconstrained mock objects can cause the code to behave in unexpected ways. Just
as you can state assumptions on the arguments of parameterized unit tests, you can
state assumptions on the results of mock object calls. For example, the author of the
IFormatProvider interface probably had the following contract in mind:
46
public object GetFormat(Type fmtType) {
object result = PexChoose.Value<object>("format");
// constraining result
PexAssume.IsTrue(result != null);
PexAssume.IsTrue(fmtType.IsAssignableFrom(result.GetType()));
return result;
}
6.6 Dependency injection

Dependency Injection [18], also known as Inversion Of Control is a design pattern that
helps building components that are mockable.
Let us illustrate this concept with the implementation of a client application that
validates a credit card number.
6.6.1 Credit card validation client

The client is a simple graphical user interface that lets the user input a credit card
number of query for its validity. Then, the client should query the web service to
validate the credit card number and display the result to the user.
6.6.2 A first attempt

The dialog window usually has a Show window to pop up the dialog, a Number property
to access to the number string value and a Status property to update its display:
public class CreditarDialog {
public bool Show()

...
public string Number

...
public string Status

...
}
A first attempt at writing the validator would be the following:
47
public class CreditarClient {
public void Execute() {
CreditarDialog dialog = new CreditarDialog();
if (dialog.Show()) {
bool valid = LuhnAlgorithm.Validate(
dialog.Number);
if (valid)
dialog.Status = "validated";
else
dialog.Status = "invalidated";
}
}
}
The implementation of the Execute method causes several issues when it comes
to testing it. The control flow depends on the Show method of the CreditarDialog
dialog window:
CreditarDialog dialog = new CreditarDialog();
if (dialog.Show()) {
This means that the test will pop up a dialog window and we will need complex
automated tools to artificially click on the buttons of the dialog. In that sense,
we will spend a large amount of effort to test a functionality that is not directly
related to the CreditarClient method.
A similar problem arises with the validate of the credit card number where we
directly call the Validate method of the LuhnAlgorithm type:
bool valid = LuhnAlgorithm.Validate(
dialog.Number);
To work around these problems, we need to add an abstract layer between the
CreditarClient class and its dependencies, the user interface and the validator ser-
vice.
6.6.3 Extracting dependencies

We do this by extracting an interface for each dependency (we already have an interface
for the credit card validation service):
public interface ICreditarDialog {
bool Show();
string Number { get;}
string Status { get;set;}
}
Each dependency is then injected into the CreditarClient:
public partial class CreditarClient {
ICreditarDialog dialog;
48
ICreditCardValidator validator;

if (this.dialog.Show()) {
bool valid =
this.validator.Validate(dialog.Number);
if (valid)
this.dialog.Status = "validated";
else
this.dialog.Status = "invalidated";
}
}
}
6.6.4 Injecting dependencies

The last problem that remains is that we need a way to set the dialog and validator
fields in the CreditarClient class. There are many ways to implement this feature,
the following are common patterns:
Constructor Injection Each dependency is passed in the class constructor, thus the
inversion of control name of the pattern,
public class CreditarClient {
ICreditarDialog dialog;
ICreditCardValidator validator;
public CreditarClient(
ICreditarDialog dialog,
ICreditCardValidator validator) {
this.dialog = dialog;
this.validator = validator;
}
...
}
Service Locator The class instance is hosted in a container that can be queried through
a message for particular services. In .NET, this pattern is implemented in the
System.ComponentModel namespace. A Component element can query for a
service using the GetService method:
public class CreditarClient : Component {
ICreditarDialog dialog =
(ICreditarDialog)
this.GetService(typeof(ICreditarDialog));
ICreditCardValidator validator =
(ICreditCardValidator)
this.GetService(typeof(ICreditCardValidator));
49
...
}
}
7 Background on white box software testing (advanced)

7.1 Analysis techniques
In general, all program analysis techniques fall somewhere in between the following
two ends of the spectrum:
Static analysis techniques verify that a property holds on all execution paths.
Since the goal is program verification, these techniques are usually overly con-
servative and flag possible violations as errors (false positives).
Dynamic analysis techniques verify that a property holds on some execution
paths. Testing is a dynamic analysis technique that aims at detecting bugs, but
cannot usually proof the absence of errors. Thus, these techniques often fail to
detect all errors.
It is often not possible to detect bugs precisely when applying only static analysis,
or employing a testing technique that is not aware of the structure of the code. Consider
the following example.
Program 7.1 Code that is difficult to analyze
int Complicated(int x, int y) {

if (x == Obfuscate(y))
throw new RareException();
return 0;
}
int Obfuscate(int y) {
return (100 + y) * 567 % 2347;
}
Static analysis techniques tend to be overly conservative, and the non-linear in-
teger arithmetic present in Obfuscate causes most static analysis techniques to
give up and issue a warning about a potential error in Complicated.
Random testing techniques have very little chance of finding a pair of x and y
values that triggers the exception.
Pex implements an analysis technique which falls in between these two extremes:
dynamic symbolic execution, a white box test generation technique. Similar to static
analysis techniques, Pex proves that for most feasible paths (those within specified
exploration bounds) a property holds. Similar to dynamic analysis techniques, Pex
only reports real errors, and no false positives.
50
7.2 Specification techniques
All program analysis techniques try to proof and/or falsify certain specified properties
of a given program. There are different ways to specify program properties:
API Contracts (Spec# [4], Eiffel[34], etc.) specify the behavior of individual
API actions from the implementations point of view. Their goal is to guarantee
robustness, in the sense that operations do not crash and data invariants are pre-
served. A common problem of API contracts is their narrow view on individual
API actions, which make it hard to describe system-wide protocols. Complex
extensions like model-fields are necessary to raise the abstraction level.
Unit Tests (JUnit [27], NUnit [35], etc.) are exemplary usage scenarios from the
point of view of a client of the API. Their goal is to guarantee functional correct-
ness, in the sense that the interplay of several operations behaves as intended. A
common problem of unit tests is that they are detached from the details of the
implementation; measuring the code coverage achieved by the unit tests gives
only a rough idea of how adequate the unit tests are.
Pex enables Parameterized Unit Testing, which unites both. Supported by a test-
input generation tool like Pex, this methodology combines the client and the implemen-
tation point of view. The functional correctness properties (parameterized unit tests)
are checked on most cornercases of the implementation (test input generation).
7.3 The testing problem

Starting from parameterized unit tests as specification, we can state the testing problem
as follows.
Given a sequential program P with statements S, compute a set of pro-

gram inputs I such that for all reachable statements s in S there exists an
input i in I such that P (i) executes s.
Remarks:
By sequential we mean that the program is single-threaded.
We consider failing an assertion, or violating an implicit contract of the exe-
cution engine (for example raising NullReferenceException when null is
dereferenced) as special statements.
Since reachability is not decidable in general, we aim for a good approximation
in practice: high coverage of the statements of the program. Instead of statement
coverage, other coverage metrics like arc coverage can be used.
7.4 White-box test input generation

White box testing means leveraging information about how a software system is imple-
mented in order to validate or falsify certain properties. White box testing may involve
51
the analysis of data flow, control flow, exception handling, or other details of the im-
plementation. In order to obtain the necessary information, white box testing requires
access to the softwares source code, or the compiled binary. Pex is a white-box anal-
ysis tool that analyzes the compiled instructions sequence of the program (the .NET
MSIL instructions).
The opposite of white box testing is black box testing, which usually amounts to
using randomly generated test data.
7.5 Symbolic execution

Pex implements a white box test input generation technique that is based on the con-
cept of symbolic execution. Pex goal is to automatically and systematically produce
the minimal set of actual parameters needed to execute a finite number of finite paths.
In general, symbolic execution works as follows: For each formal parameter, a sym-
bolic variable is introduced. When a program variable is updated to a new value during
program execution, then this new value is often expressed as an expression over sym-
bolic variables. When a statement in the program has more than one possible successor,
execution is forked into two paths. For each code path explored by symbolic execution,
a path condition is built over symbolic variables. For example, the Add-method of the
ArrayList implementation shown in Program 4.2 contains an if-statement whose
condition is this._items.Length == this._size (where the field _items de-
notes the array holding the array lists elements and _size denotes the number of
elements currently contained in the array list). The symbolic execution conjoins this
condition to the path condition for the then-path and the negated condition to the path
condition of the else-path. In this manner all constraints are collected which are needed
to deduce what inputs cause a code path to be taken.
A constraint solver or automatic theorem prover is usually used to decide the feasi-
bility of individual execution paths, and to obtain concrete test inputs as representatives
of individual execution paths.
Analysis of all paths cannot always be achieved in practice. When loops and re-
cursion are present, an unbounded number of code paths may exist. In this case loops
and recursion are usually analyzed only up to a specified number of unfoldings. Even
if the number of paths is finite, solving the resulting constraint systems is sometimes
computationally infeasible, depending on the employed constraint solver or automatic
theorem prover.
Symbolic execution in its original form [29] is a static program analysis, since it
does not actually execute the program but merely analyzes possible execution paths.
7.6 Dynamic symbolic execution

Applying symbolic execution as described above to real-world program is problematic,
since such a programs interaction with a stateful environment cannot be forked.
Pex explores the behaviors of a parameterized unit test using a technique called
dynamic symbolic execution [21, 13]. This test generation technique consists in exe-
cuting the program, starting with very simple inputs, while simultaneously performing
a single-path symbolic execution to collect symbolic constraints on the inputs obtained
52
from predicates in branch statements along the execution, and then using a constraint
solver to infer variants of the previous inputs in order to steer future program executions
along alternative program paths. In this way, all program paths will be exercised even-
tually. Operations that are implemented by the external environment are not tracked
symbolically; instead, the actually observed input/output values become part of the
constraints.
As a result, dynamic symbolic execution extends static symbolic execution [29]
with additional information that is collected at runtime, which makes the analysis more
precise [21, 19]. By continuation, all analyzed execution paths are feasible, which
avoids the problem of spurious error reports common to overly conservative static anal-
ysis techniques.
While additional information is collected on the level of individual execution traces
which characterize individual execution paths, knowing the structure of the program
still enables the analysis of many execution paths at once, [21, 1, 10].
Algorithm 7.1 shows the general dynamic symbolic execution algorithm. The
choice of the new inputs in Step 1 decides in which order the different execution paths
of the program are visited. Pex uses several heuristics that take into account the struc-
ture of the program and the already covered branches when deciding on the next pro-
gram inputs. The goal of Pex strategy is to achieve high statement coverage fast. As
an effect, the user just has to set a time limit or another rough analysis bound. Most
other tools ([21, 19, 21, 13]) explore the execution paths in a fixed search order, and
they require the user to define detailed bounds on the size and structure of the input
data.
Algorithm 7.1 Dynamic symbolic execution

Step 0 Set J := (intuitively, J is the set of already
analyzed program inputs)
Step 1 Choose program input i /J (stop if no such i can be found)
Step 2 Output i
Step 3 Execute P (i); record path condition C (in particular, C(i) holds)
Step 4 Set J := J C (viewing C as the set {i | C(i)})
Step 5 Goto Step 1.
7.7 Example
In the following, we explain how Pex handles Program 7.1, starting from the method
Complicated.
In order to run the code for the first time, Pex needs to supply some argument values
to Complicated, for example x = 572 and y = 152 (arbitrary values). Pex monitors
the execution of the code, following the execution into and out of method calls. With
randomly chosen values, it is very unlikely that we will trigger the rare exception.
Pex remembers that we did not cover the throw statement. Pex also remembers all
conditions that were evaluated; here, it remembers that x 6= (100 + y) 567%2347.
53
Knowing that we have not yet covered the throw statement, Pex looks at the pro-
gram to find the branch that guards that statement. Last time, we had x 6= (100+y)567
mod 2347. So in order reach the other branch of the if-statement, Pex builds the
negated condition, x = (100 + y) 567 mod 2347, and hands it to a constraint solver.
In this case, it is quite simple to solve the constraint system, since one just has to supply
any value for y to compute x.
Pex runs the code again for with the new inputs, say x = (100 + 152) 567
mod 2347 and y = 152. This time the throw statement will be covered. Since all
statements have been covered now, Pex will stop.
8 Understanding Pex (advanced)

This section explains various aspects of the tool Pex, that is used to craft inputs to the
parameterized unit tests.
8.1 Why dynamic symbolic execution
Symbolic execution was originally proposed [29] as a static program analysis tech-
nique, which is an analysis that only considered the source code of the analyzed pro-
gram. This approach works well as long as all decisions can be made on basis of
the source code alone. It becomes problematic when the program contains constructs
that make reasoning hard (for example accessing memory through arbitrary pointers),
and when parts of the program are actually unknown (for example when the program
communicates with the environment, for which no source code is available, and whose
behavior has not been specified).
Many .NET programs use unsafe .NET features such as arbitrary memory accesses
through pointers for performance reasons, and most .NET programs interact with other
unmanaged (non-.NET) components for legacy reasons.
While static symbolic execution algorithms do not have or use any information
about the environment into which the program is embedded, dynamic symbolic execu-
tion does leverage dynamic information that it observes during concrete program exe-
cutions (information about actually taken execution paths, and the data that is passed
around between the analyzed program and the environment).
Knowing the actually taken execution paths allows Pex to prune the search space.
When the program communicates with the environment, Pex builds a model of the en-
vironment from the actual data that the environment receives and returns. This model
is an under-approximation of the environment, since Pex does not know the conditions
under which the environment produces its output. The resulting constraint systems that
Pex builds may no longer accurately characterize the programs behavior, and as a con-
sequence Pex prunes such paths. Thus, Pex always maintains an under-approximation
of the programs behavior, which is appropriate for testing.
54
8.1.1 Exercise 11: Understanding the Path Condition
Part 1: Solving constraints of Program 7.1 Call Program 7.1 from a parameterized
unit test, and inspect which values Pex uses as test inputs. Do the values match your
expectations? Would another set of values be more appropriate for testing? If so why,
if not, why not?
Part 2: Playing with Program 7.1 We revisit Program 7.1 and see how Pex handles
this example.
1. Add a new test in the project, copy the source of Program 7.1 and turn it into a
parameterized test.
2. One way to gain more insight on what is happening is the GetPathCondi-
tionString method of PexSymbolicValue class that returns the current path
condition. The following code will add a column to the parameter table and fill
it in with the path condition.
int Complicated(int x, int y) {
if (x == Obfuscate(y))
// logging the path condition

string pc = PexSymbolicValue.GetPathConditionString();
PexObserve.ValueForViewing("pc", pc);
return 0;
}
If you run the above code as part of a parameterized unit test, you will notice
that the path condition is only added to the table when no exception is thrown.
To make sure that the path condition is added in any case, you can embed the
logging code into a try-finally block. 2
3. Execute Pex and analyze the results. Do they match your understanding?
4. Another way to look into Pex symbolic analysis is the ToString method of
the PexSymbolicValue class that returns a string that represents Pex current
knowledge about how the given value was derived from the inputs:
int Complicated2(int x, int y) {
if (x == Obfuscate(y)) {
// logging the path condition
string obfuscateY =
PexSymbolicValue.ToString(Obfuscate(y));
PexObserve.ValueForViewing("obfuscate(y)", obfuscateY);
}
return 0;
}
55
Part 3: Additional Path Condition Samples Additionally, you can also try out to
use the GetPathConditionString and ToString methods of the PexSymbolicValue
class with other examples:
1. Dealing with loops

public void PathConditionLoop(int x, int y) {
for (int i = 0; i < x; ++i)
if (x == y + 1) {
string pc = PexSymbolicValue
.GetPathConditionString();
}
}
2. Dealing with strings

void PathConditionSubstring(string s, int x, string y) {
if (s.Substring(x) == y) {
}
}
void PathConditionStringMatch(string s) {
if (s == "Hello") {
}
}
8.2 Monitoring by instrumentation

Pex monitors the execution of a .NET program through code instrumentation. Pex
plugs into the .NET profiling API. Before any method is compiled from the Microsoft
Intermediate Language (MSIL) to native machine code by the Just-In-Time compiler
of .NET, the Pex profiler gets a callback to inspect and rewrite the instructions. All
.NET language compilers translate into this instruction set. Pex operates on the level
of these instructions. Pex does not care in which high level language the program was
written. (However, when Pex gives feedback to you in the form of code snippets, Pex
only supports C# syntax at this time.)
The instrumented code drives a shadow interpreter in parallel to the actual pro-
gram execution. The shadow interpreter
constructs symbolic representations of the executed operations over logical vari-

ables instead of the concrete program inputs;
maintains and evolves a symbolic representation of the entire programs state at

any point in time;
56
records the conditions over which the program branches.
Pex interpreter models the behavior of all verifiable .NET instructions precisely,
and models most unverifiable (involving unsafe memory accesses) instructions as well.
8.3 Symbolic state representation

A symbolic program state is a predicate over logical variables together with an assign-
ment of locations to expressions over logical variables, just as a concrete program state
is an assignment of locations to values. The locations of a state may be static fields,
instance fields, method arguments, locals, and positions on the operand stack.
Pex expression constructors include primitive constants for all basic .NET data
types (integers, floating point numbers, object references), and functions over those
basic types representing particular machine instructions, for example addition and mul-
tiplication. Pex uses tuples to represent .NET value types (structs). Pex uses maps
to represent instance fields and arrays, similar to the heap encoding of ESC/Java [17]:
an instance field of an object is represented by a single map which associates object
references with field values. Constraints over the .NET type system and virtual method
dispatch lookups are encoded in expressions as well. Predicates are represented by
Boolean-valued expressions.
Pex implements various techniques to reduce the enormous overhead of the sym-
bolic state representation. Before building a new expression, Pex always applies a set
of reduction rules which compute a normal form. A simple example of a reduction
rule is constant folding, for example 1 + 1 is reduced to 2. All logical connectives are
transformed into a BDD representation with if-then-else expressions [11]. All expres-
sions are hash-consed, which means that only one instance is ever allocated in memory
for all structurally equivalent expressions. Pex also employs independent constraint
optimization [13].
Based on the already accumulated path condition, expressions are further simpli-
fied. For example, if the path condition already established that x > 0, then x < 0
simplifies to false.
While strings can be represented in a way similar to arrays of characters, Pex rep-
resents certain string constraint as expressions using specialized string-functions as
well [9].
When running the following example with Pex, it will print to the console how the
state is represented symbolically, including the path condition, at a particular program
point.
static int Global;
[PexMethod]
public void SymbolicStateExample(int x, int y)
{
Global = 1 + x + x * x + x * x - 1;
if (y > Global)
{
Console.WriteLine("Global=" +
PexSymbolicValue.ToRawString(Global));
57
Console.WriteLine("x=" +
PexSymbolicValue.ToRawString(x));
Console.WriteLine("y=" +
PexSymbolicValue.ToRawString(y));
Console.WriteLine("pc=" +
PexSymbolicValue.GetRawPathConditionString());
}
}
ToRawString and GetRawPathConditionString return expressions representing

symbolic values and the path condition, formatted as S-expressions. Here, it will print
the following.
1 Global=(Add (Mul (Exp x 2) 2) x)

2 x=x
3 y=y
4 pc=(Clt (Add (Mul (Exp x 2) 2) x) y)
Add, Mul, Exp are binary functions representing addition, multiplication and exponen-
tiation, and Clt is a predicate that compares if the left operand is less than the second
operand. On the right-hand sides, x and y represent the symbolic test inputs. Note that
the expressions have been normalized by Pex.
8.3.1 Exercise 12: Heap Constraints
Consider the following type

public class C { public int F; }
that is used in the following method.

public void AliasChallenge(C x, C y) {
if (x != null)
if (y != null) {
x.F = 42;
y.F = 23;
// if (x.F == 42) throw new Exception("boom");
}
}
How many execution paths will Pex have to explore? (Pex only generates different test
inputs when they exercise different execution paths.)
How many are there if we include the commented code?
Again, we can inspect the constraints that Pex collects internally: 2 GetPathCon-
ditionString shows us a pretty-printed string that bends Pex internal representation
into a C#-like syntax. Try out GetRawPathConditionString instead to get a view
on the raw expressions that Pex uses internally. They are formatted as S-expressions.
In this example, you will see functions called select and update. These functions
operate on maps which describe the evolution of and accesses to the heap [33].
58
8.4 Search strategy
Pex builds the (reachable portion of the) execution tree of the program from all pre-
viously discovered execution paths. In each step of the test generation algorithm, Pex
picks an outgoing unexplored branch of the tree, representing an execution path that
has not been discovered yet. Pex implements a fair choice between all such unexplored
branches. Pex includes various fair strategies which partition all branches into equiv-
alence classes, and then picks a representative of the least often chosen class. The
equivalence classes cluster branches by mapping them
to the branch statement in the program of which the execution tree branch is
an instance (each branch statement may give rise to multiple execution tree
branches, for example when loops are unfolded),
the stack trace at the time the execution tree branch was created,
the depth of the branch in the execution tree.
Pex combines all such fair strategies into a meta-strategy that performs a fair choice
between the strategies.
See [54] for a more detailed description of the search strategies in Pex.
8.4.1 Exercise 13: Search Frontiers

Consider the following parameterized unit test. It has two loops, each of which can
make 32 separate decisions.
using Microsoft.Pex.Framework.Strategies;
[PexMethod, PexSearchFrontierDepthFirst]
public void TestWithLoops(int x, int y) {
int xbits = 0;
for (int i = 0; i < 32; i++)
if (((1 << i) & x) != 0) xbits++;
int ybits = 0;
for (int i = 0; i < 32; i++)
if (((1 << i) & y) != 0) ybits++;
PexObserve.ValueForViewing("xbits", xbits);
PexObserve.ValueForViewing("ybits", ybits);
}
1. Try the different search frontiers. Additional search frontier implementation can
be found under the Microsoft.Pex.Framework.Strategies namespace.
2. Can you explain what you see?
Note that, by default, Pex does not emit a test for each path it explores.
59
8.5 Constraint solving
For each chosen unexplored branch, Pex builds a formula that represents the condition
under which this branch may be reached.
Pex employs Z3 as its constraint solver. Z3 is a Satisfiability Modulo Theories
(SMT) solver, an automated satisfiability checker for first-order logic with several built-
in theories. This makes Z3 an efficient automatic theorem prover; historically, SMT
solvers have been used mainly to perform program correctness proofs, for example as
part of ESC/Java [17] and Spec# [4]. In addition to deciding satisfiability, Z3 can also
compute a model (satisfying assignments) for a satisfiable formula, which makes it an
ideal constraint solver for Pex.
Pex faithfully encodes all constraints arising in safe .NET programs such that Z3
can decide them with its built-in decision procedures for propositional logic, fixed sized
bit-vectors, tuples, arrays and quantifiers. Pex also has a specialized string solver [9]
that is integrated with Z3. Arithmetic constraints over floating point numbers are ap-
proximated by a translation to rational numbers, and heuristic search techniques are
used outside of Z3 to find approximate solutions for floating point constraints. Pex
encodes the constraints of the .NET type system and virtual method dispatch lookup
tables as universally quantified formulae.
8.5.1 Exercise 14: Constraint Solving

You can use Pex to solve constraint systems written in C#. For example, Pex can find
the two prime factors of 52605271 with a test like the following. Random testing would
most likely not be able to solve this problem.
[PexMethod]
public void DetermineFactors(int x, int y) {
if (x > 1 && x < 10000 &&
y > 1 && y < 10000 &&
x * y == 52605271)
throw new Exception("found it");
}
What are the two prime factors?

Can you write other interesting constraint systems and solve them with Pex?
8.6 Implicit branches

Pex treats all possible deterministic exceptional control flow changes in the code like all
other explicit branches: It tries to explore the successful case as well as the exceptional
outcome.
The following exceptions that the CLR execution engine may throw fall into this
category:
NullReferenceException
IndexOutOfRangeException
60
OverflowException
InvalidCastException
DivisionByZeroException
Pex does not systematically try to throw the following exceptions that the execu-
tion engine may raise non-deterministically: OutOfMemoryException, StackOver-
flowException, ThreadAbortException. (In fact, by default Pex tries hard to
avoid them so that only perfectly deterministic execution paths are explored.)
8.6.1 Exercise 15: Implicit Branches

The following method has two execution paths.
public void ImplicitNullCheck(int[] a) {
int x = a.Length;
}
How many paths will Pex explore in the following method? (Note that Pex checks for
each possible exception type separately, and considers checks for different exception
types as different branches.)
public void ImplicitIndexOutOfRangeCheck(int[] a) {
int x = a[0];
}
Pex understands checked code as well. Pex finds input that will cause the following
method to throw an OverflowException.
public void ImplicitOverflowCheck(int x, int y) {
int z = checked(x + y);
}
Can you write a parameterized unit test that could cause an InvalidCastException?
8.7 Assumptions and assertions

You can use assumptions and assertions to express preconditions (assumptions) and
postconditions (assertions) of tests.
While Pex tries different argument values during the exploration of the code, Pex
might inadvertently violate an assumption. When that happens, Pex remembers the
condition that caused the assumption violation, so that Pex will not generate another
test that violates the assumption. The test case itself that violated the assumption is
silently pruned.
The concept of assertions is well known in unit test frameworks. Pex understands
the built-in Assert classes provided by each supported test framework. However, most
frameworks do not provide a corresponding Assume class. For that case, Pex provides
the PexAssume class. If you do not want to use an existing test framework, Pex also
has the PexAssert class.
61
[PexClass]
public partial class MyTests {
[PexMethod]
public void Test1(object o) {
// precondition: o should not be null
PexAssume.IsNotNull(o);
...
}
}
The non-nullness assumption can also be encoded as a custom attribute:

public partial class MyTests {
[PexMethod]
public void Test2([PexAssumeNotNull] object o)
// precondition: o should not be null
{
...
}
}
When you write an assertion, Pex will not only passively detect assertion violations,
but Pex will in fact actively try to compute test inputs that will cause the assertion to
fail. The reason is simply that the assert statement is implemented as a conditional
branch, which throws an exception in the failure case, similar to the following code:
public class PexAssert {
public static void IsTrue(bool condition) {
if (!condition)
throw new PexAssertionViolationException();
}
}
Just like for any other branch in the code, Pex will build constraint systems which aim
at invalidating the asserted condition for each calling context. When Pex can solve the
constraint system for the negated condition, we get test inputs that exhibit an error.
Just as an assertion might throw an PexAssertFailedException, Pex uses a
PexAssertFailedException internally to stop a test case when an assumption fails.
8.8 When does a test case fail?

Pex considers the exceptional behavior of a test (whether the test throws an exception
that is not caught) to decide whether a test case fails or succeeds:
If the test does not throw an exception, it succeeds.
If the test throws an exception,
62
that is a PexAssumeFailedException, it succeeds. (But it is usually
filtered out, unless the TestEmissionFilter is set to All.)
that is a PexAssertFailedException, or any other assertion violation
exception of other unit test framework, it fails.
that is neither an assumption nor an assertion violation exception, it de-
pends on further annotations whether the test passes or fails.
You can annotate the test (or the test class, or the test assembly) with one of the fol-
lowing attributes to indicate which exception types may or must be thrown by the test
in order to be considered successful.
The PexAllowedExceptionAttribute indicates that a test method, or any

other method that it calls directly or indirectly, may throw a particular type of
exception for some test inputs.
The PexAllowedExceptionFromTypeAttribute indicates that any method

of a specified type may throw a particular type of exception for some test inputs.
The PexAllowedExceptionFromTypeUnderTestAttribute indicates that

any method of the designated type under test may throw a particular type of
exception for some test inputs.
The PexAllowedExceptionFromAssemblyAttribute indicates that any method

located in a specified assembly may throw a particular type of exception for some
test inputs.
8.9 When does Pex emit a test case?

Pex supports different filters that decide when generated test inputs will be emitted as a
test case. You can configure these filters with the TestEmissionFilter property that
you can set for example in the PexMethod attribute. Possible values are the following.
All - Emit every generated test input as a test case, including those which cause
assumption violations.
FailuresAndIncreasedBranchHits (default) - Emit tests for all unique fail-

ures, and whenever a test case increases coverage, as controlled by the Test-
EmissionBranchHits property (see below).
FailuresAndUniquePaths - Emit tests for all failures Pex finds, and also for
each test input that causes a unique execution path.
Failures - Emit tests for failures only.
Regarding increased branch coverage, the TestEmissionBranchHits property

controls whether Pex should just consider whether a branch was covered at all (Test-
EmissionBranchHits=1), or whether a test covered it either once or twice (Test-
EmissionBranchHits=2) and so on.
63
TestEmissionBranchHits=1 will give a very small test suite that will cover all
branches Pex could reach. (In particular, this test suite will also cover all reached basic
blocks and statements.) The default for this option is TestEmissionBranchHits=2,
which generates a more expressive test suite that is also better suited to detect future
regression errors.
8.9.1 Exercise 16: Test case emission filters

Consider the following max methods.
int max(int x, int y) {
if (x > y)
return x;
else
return y;
}
int max(int a, int b, int c, int d) {

return max(max(a, b), max(c, d));
}
Consider the following PexMethod.

[PexMethod]
public void MaxTest(int a, int b, int c, int d) {
int e = max(a, b, c, d);
PexObserve.ValueForViewing("max", e);
}
How many execution paths does it have? (And can you make Pex emit test cases
for all execution paths?) How many tests will Pex emit with TestEmissionBranch-
Hits=1? What is the relation between those two numbers?
8.10 When does Pex stop?

If the code under test does not contain loops, or unbounded recursion, Pex will typically
stop quickly because there is only a (small) finite number of execution paths to analyze.
However, most interesting programs contain loops and/or unbounded recursion. In
such cases the number of execution paths is (practically) infinite, and it is in general
undecidable whether a statement is reachable. In other words, Pex would take forever
to analyze all execution paths of the program.
In order to make sure that Pex terminates after a reasonable amount of time, there
are several exploration bounds. All bounds have predefined default values, which you
can override to let Pex analyze more and longer execution paths. The bounds are
parameters of the PexMethod, PexClass and PexAssemblySettings attributes.
There are different kinds of bounds:
Constraint Solving Bounds apply to each attempt of Pex to determine whether an
execution path is feasible. Pex may need several constraint solving attempts to compute
the next test inputs.
64
ConstraintSolverTimeOut Seconds the constraint solver has to figure out
inputs that will cause a different execution path to be taken
ConstraintSolverMemoryLimit Megabytes the constraint solver may use to

figure out inputs
Exploration Path Bounds apply to each execution path that Pex executes and moni-
tors. These bounds make sure that the program does not get stuck in an infinite loop,
or recursive method.
MaxBranches Maximum number of branches that may be taken along a sin-

gle execution path
MaxCalls Maximum number of calls that may be taken during a single exe-
cution path
MaxStack Maximum size of the stack at any time during a single execution
path, measured in number of active call frames
MaxConditions Maximum number of conditions over the inputs that may be

checked during a single execution path
Exploration Bounds apply to the exploration of each parameterized unit test.
MaxRuns Maximum number of runs that will be tried during an exploration

(each run uses different test inputs; not every run will result in the emission of a
new test case)
MaxRunsWithoutNewTests Maximum number of consecutive runs without

a new test being emitted
MaxRunsWithUniquePaths Maximum number of runs with unique execu-

tion paths that will be tried during an exploration
MaxExceptions Maximum number of exceptions that may be found over all

discovered execution paths combined
MaxExecutionTreeNodes Maximum number of conditions over the inputs

that may be checked during all discovered execution paths combined
MaxWorkingSet Maximum size of working set in megabytes
TimeOut Seconds after which exploration stops
The following example shows a parameterized test that involves a loop. The loop
bound depends on the test inputs, and the lurking exception can only be triggered if the
loop is unfolded a certain number of times. Here, we used an explicit bound of 10 runs
(MaxRuns = 10) to let Pex finish quickly. However, with this bound, Pex will most
likely not be able to trigger the exception, as Pex will not unroll the loop sufficiently
many times.
65
[PexMethod(MaxRuns = 10)]
public void TestWithLoop(int n) {
var sum = 0;
for (int i = 0; i < n; i++)
sum++;
if (sum > 20) throw new Exception();
}
In the Pex tool bar, you will see that the boundary button got enabled.
When you click on the button, Pex shows which bounds were exceeded, and it
offers several actions to increase the bounds.
Go ahead and select Set MaxRuns=20 to double the bound. This will update the
specified bounds in the source code. Run Pex again. If Pex is still not able to trigger
the exception, you might have to double the bound again.
The following example shows another parameterized test that involves a loop, but
this loop does not depend on the test inputs. Here, we used an explicit bound of 10
branches (MaxBranches = 10) to let Pex finish quickly. However, with this bound,
Pex cannot even once execute the code from beginning to end, as executing the embed-
ded loop will cause more than 10 branches to be executed.
[PexMethod(MaxBranches = 10)]
public void TestWithFixedLoop(int j) {
var sum = 0;
for (int i = 0; i < 15; i++)
sum++;
if (j == 10) throw new Exception();
}
In those cases, where a particular run exceeded some path-specific bounds, we get
a special row with the words path bounds exceeded in the table of all generated test
cases. When we click on the row, we see more detailed information about the exceeded
bound, and there is a button Set MaxBranches=... to increase the bounds. (The button
in the tool bar is enabled as well.)
66
Click on the button, increase the bounds, and run Pex again. If Pex can still not find
the exception, you might have to increase the bounds again.
8.11 How does Pex suggest fixes for errors?

Sometimes, when a test run fails, Pex can suggest a potential change to the source code
to prevent the same failure from happening again.
Algorithm 8.1 shows how Pex tries to locate the failure cause.
Algorithm 8.1 Fix It

1. Pex determines the point in the execution trace of the test case where the failure
manifested itself (where an exception was thrown that was not caught)
2. Pex looks back in the execution trace for the last public method call.
3. Pex computes a condition under which the failure happened, relative to the last
public method call.
4. If the condition only involves parameters, Pex will suggest the negation of the
condition as a missing precondition.
5. Otherwise, Pex will try to express the condition in terms of the fields of the class
of the last public method call. Pex will suggest the negation of the condition as
a missing invariant (a condition over the fields of the class which should always
hold). Furthermore, Pex will execute the test again, and try to find which pub-
lic method of this class first left the object behind in a state which violates the
suggested invariant. If Pex finds such a method, Pex will start over at 3.
This algorithm terminates because in each iteration, Pex tries to find the failure
cause at an earlier point in the execution trace. If Pex suggests a missing precondi-
tion or invariant, following Pex advice is guaranteed to prevent the same failure from
happening again, under the proviso that Pex could monitor all relevant parts of the pro-
gram. If this proviso is violated, Pex could suggest a precondition or an invariant which
is inconsistent, which basically means that Pex suggests to not run this code again.
8.12 Creating complex objects

Pex monitors the executed instructions when it runs a test and the program-under-test.
In particular, it monitors all field accesses. It then uses a constraint solver to determine
67
new test inputs including objects and their field values such that the test and the
program-under-test will behave in other interesting ways.
Thus, Pex needs to create objects of certain types and set their field values. If the
class is visible and has a visible default constructor, Pex can create an instance of the
class. If all the fields of the class are visible, Pex can set the fields automatically.
If the type is not visible, or the fields are not visible, Pex needs help to create objects
and bring them into interesting states to achieve maximal code coverage.
There are two ways to help Pex.
The first is for the user to provide parameterized factories for the complex objects
such that Pex can explore different object states, starting from factory methods.
The second is to define the invariants of the objects private fields, such that Pex
can manufacture different object states directly.
8.13 Explorables
Consider the following class.
public class MyCounter {
private int _count;
public MyCounter(int initialCount) {
if (initialCount < 0) throw new ArgumentException();
this._count = initialCount;
}
public MyCounter() {
this._count = 0;
}
public void Increment() { this._count++; }
public int Count { get { return this._count; } }
}
Consider the following test where something bad happens when the counter reaches
the number 99.
[PexMethod]
public void TestMyCounter([PexAssumeNotNull]MyCounter counter) {
if (counter.Count == 99) throw new Exception();
}
When you run Pex it may not find the exception, but instead Pex might show you
a warning such as the following:
Pex guessed how to create instances of MyCounter:
new MyCounter()
This means that, not knowing all ways to construct meaningful counters, Pex chose
the default constructor that takes no parameters. Of course, with that constructor we
would have to call the Increment() method 99 times in order to hit the exception,
which Pex would do only after a very long exploration time. We can improve the
situation by telling Pex to use the other constructor.
68
You can define a factory method that Pex can use to create instances of a given type.
In fact, when you use Pex from Visual Studio, you get the option to create a factory
method when necessary. The default code and attributes for such a factory are similar
to the following.
public static class Factory {
[PexFactoryMethod(typeof(MyCounter))]
public static MyCounter CreateMyCounter() {
return new MyCounter();
}
}
You can change this factory method as desired. You can add parameters, and then
Pex will determine which values are relevant. The following is a factory method that
creates MyCounter instances by calling the constructor that takes one parameter.
[PexFactoryMethod(typeof(MyCounter))]
public static MyCounter CreateMyCounter(int x)
{
return new MyCounter(x);
}
With these annotations in place, when you run Pex again, it will explore the con-
structor that takes an initial count, and Pex will find a test case such as the following:
[TestMethod]
[PexRaisedException(typeof(Exception))]
public void TestMyCounter_MyCounter_71115_194002_0_02() {
MyCounter mc0 = new MyCounter(99);
this.TestMyCounter(mc0);
}
As an alternative to factory methods, you can indicate in a declarative way which

constructor to use with the following attribute:
[assembly: PexExplorableFromConstructor(
typeof(MyCounter),
typeof(int))]
The first argument indicates the type to construct, the following arguments are the
parameter types for the desired constructor.
8.14 Invariants
Even when you tell Pex which constructor to use, or even when you write a factory
method by hand, Pex may have a hard time to explore all combinations of the construc-
tor and the mutator methods.
Tip: Advanced The section about invariants is for advanced developers. It is often
sufficient, and much easier, if you can get Pex to create objects through the explorables
that we described in Section 8.13.
69
The example that we showed in Section 8.13 had a nice property: All possible
and legal configurations of the MyCounter class could be constructed by calling the
constructor. The constructor was written to throw an exception when an attempt is
made to configure an invalid object.
When a class does not have such a constructor, we can always write such a special
constructor for testing purposes: a constructor that allows to configure the object in all
possible and legal ways. To describe what is possible and legal, you have to explicitly
define the condition under which the fields of an object are properly configured. This
condition is called the class invariant.
You can write an invariant as a private parameterless instance method that returns
bool. For example, the invariant of the array list class can be expressed as follows.
private bool Invariant() {
return
this._items != null &&
this._size >= 0 &&
this._items.Length >= this._size;
}
Now you can define the special public constructor for testing purposes. It simply re-
ceives all field values as parameters. When the supplied parameter values do not satisfy
the invariant, the constructor is aborted by throwing an exception. One way to make
sure that this constructor is only used for testing purposes is to define it conditionally.
#if DEBUG
public ArrayList(object[] items, int size) {
this._items = items;
this._size = size;
if (!this.Invariant()) throw new InvalidOperationException();
}
#endif
Another way to make sure this constructor is only used for testing purposes is to
define it as internal, and use the InternalsVisibleToAttribute attribute to
expose it to the assembly that contains the tests.
Now you can tell Pex to use this constructor, as we discussed in Section 8.13:
[assembly: PexExplorableFromConstructor(typeof(ArrayList),
typeof(object[]), typeof(int), typeof(int))]
Pex will explore the constructor and the conditions in the Invariant method,
filtering out all configuration where the invariant does not hold.
As an alternative to defining DEBUG-only constructors, Pex can also leverage desig-
nated invariant methods that can be specified in conjunction with the Code Contracts
framework[25]. How to write and leverage invariants with code contracts is docu-
mented in the paper on patterns for parameterized unit testing in the documentation
section of the Pex website [40].
8.15 Limitations
There are certain situations in which Pex cannot analyze the code properly:
70
Nondeterminism Pex assumes that the analyzed program is deterministic. If it is not,
Pex will prune non-deterministic execution paths, or it may go in cycles until it
hits exploration bounds.
Concurrency Pex does not handle multithreaded programs. (It might work in a sce-
nario where the main thread execution is deterministic, independent of the behav-
ior of other spawned threads.) CHESS [37] is a project that is similar to Pex, but
instead of exploring the non-determinism induced by input values, CHESS ex-
plores the non-determininism induced by different thread-interleavings in multi-
threaded applications.
Native Code or .NET code that is not instrumented Pex does not understand native
code (x86 instructions called through the P/Invoke mechanism of .NET). Pex
does not know how to translate such calls into constraints that can be solved
by a constraint solver. And even for .NET code, Pex can only analyze code it
instruments.
However, even if some methods are uninstrumented, Pex will still try to cover
the instrumented code as much as possible.
Language In principle, Pex can analyze arbitrary .NET programs, written in any .NET
language. However, the Visual Studio AddIn and Pex code generation only sup-
port C#.
Symbolic Reasoning Pex uses an automatic constraint solver to determine which val-
ues are relevant for the test and the program-under-test. However, the abilities
of the constraint solver are, and always will be, limited. In particular, Z3 cannot
reason precisely about floating point arithmetic.
9 Related work
Testing with algebraic specifications was started by Gaudel et al. [8]. They use axioms
in various ways: to describe the test purpose; to obtain concrete data, which is nec-
essary for the instantiations of the axioms; and to derive new theorems, which when
tested should uncover errors in state not captured by the model. For deriving those
theorems they introduced regularity assumptions.
Parameterized Unit Tests [48] are a way to write algebraic specification as code.
Another name for it is theories in the JUnit test framework, where Saff et al. [43]
found them to be effective. They appear as row tests in MbUnit [26], and under other
names in various other unit test frameworks.
It has also been investigated how such algebraic axioms can be synthesized auto-
matically [24, 46, 14] from a given implementation, possibly supported by a given test
suite [16].
In order to generate test inputs for parameterized unit tests, Pex [47] uses dynamic
symbolic execution. A recent overview on the combination of static and dynamic anal-
ysis can be found in Godefroid et al. [20]. The basic idea of symbolic execution [29]
was introduced more than three decades ago. By combining it later with other work
71
on dynamic test generation [30, 31], DART [21] was the first practical tool implement-
ing dynamic symbolic execution for C programs. Many other implementations have
been recently developed as well: CUTE [45], EXE [13], KLEE [12] for C programs.
jCUTE [44] for Java programs. SAGE [22] for x86 code.
Related to dynamic symbolic execution is model-checking of programs. XRT [23]
is a model checker for .NET programs, JPF [2] for Java programs. Both JPF and
XRT have extensions for symbolic execution. However, both can only perform static
symbolic execution, and they cannot deal with stateful environment interactions.
Many other approaches to automatic test generation exist. For example, Ran-
doop [39] is a tool that generates new test-cases by composing previously found test-
case fragments, supplying random input data.
10 Conclusion
We presented the methodology of parameterized unit testing [48], a generalization of
traditional unit testing. Parameterization allows the separation of two concerns: The
specification of the behavior of the system, and the test cases to cover a particular im-
plementation. Dynamic symbolic execution and constraint solving allow the automatic
generation of the test cases in many cases. To this end, we used Pex [47] to automati-
cally generate test inputs.
We studied how to write parameterized unit tests for various programs, how to
measure coverage of the code under test, how to test programs that interact with the
environment, and how path constraints are collected and solved in dynamic symbolic
execution.
References
[1] S. Anand, P. Godefroid, and N. Tillmann. Demand-driven compositional sym-
bolic execution. In Proc. of TACAS08, volume 4963 of LNCS, pages 367381.
Springer, 2008.
[2] S. Anand, C. S. Pasareanu, and W. Visser. JPF-SE: A symbolic execution exten-
sion to java pathfinder. In Proc. of TACAS 2007, pages 134138, Braga, Portugal,
March 2007.
[3] J. Bach. Exploratory testing explained, 2003. http://www.satisfice.
com/articles/et-article.pdf.
[4] M. Barnett, R. Leino, and W. Schulte. The Spec# programming system: An
overview. In M. Huisman, editor, Construction and Analysis of Safe, Secure,
and Interoperable Smart Devices: International Workshop, CASSIS 2004, volume
3362 of LNCS, pages 4969, 2005.
[5] K. Beck. Test Driven Development: By Example. Addison-Wesley, 2003.
[6] K. Beck and C. Andres. Extreme Programming Explained: Embrace Change
(2nd Edition). Addison-Wesley Professional, 2004.
72
[7] B. Beizer. Software Testing Techniques. Van Nostrand Reinhold Co., New York,
NY, USA, 2nd edition, 1990.
[8] M. Bidoit, H.-J. Kreowski, P. Lescanne, F. Orejas, and D. Sannella, editors. Al-
gebraic system specification and development. Springer-Verlag New York, Inc.,
New York, NY, USA, 1991.
[9] N. Bjrner, N. Tillmann, and A. Voronkov. Path feasibility analysis for string-
manipulating programs. In Tools and Algorithms for the Construction and Anal-
ysis of Systems (TACAS09), volume 5505 of LNCS, pages 307321. Springer,
2009.
[10] P. Boonstoppel, C. Cadar1, and D. Engler. Rwset: Attacking path explosion in

constraint-based test generation. In Proc. of TACAS08, volume 4963 of LNCS,
pages 351366. Springer, 2008.
[11] K. S. Brace, R. L. Rudell, and R. E. Bryant. Efficient implementation of a BDD

package. In DAC 90: Proceedings of the 27th ACM/IEEE conference on Design
automation, pages 4045, New York, NY, USA, 1990. ACM Press.
[12] C. Cadar, D. Dunbar, and D. R. Engler. Klee: Unassisted and automatic gen-
eration of high-coverage tests for complex systems programs. In OSDI, pages
209224, 2008.
[13] C. Cadar, V. Ganesh, P. M. Pawlowski, D. L. Dill, and D. R. Engler. Exe: automat-

ically generating inputs of death. In CCS 06: Proceedings of the 13th ACM con-
ference on Computer and communications security, pages 322335, New York,
NY, USA, 2006. ACM Press.
[14] C. Csallner, N. Tillmann, and Y. Smaragdakis. Dysy: dynamic symbolic execu-

tion for invariant inference. In ICSE, pages 281290, 2008.
[15] J. de Halleux and N. Tillmann. Parameterized unit testing with Pex (tutorial). In
Proc. of Tests and Proofs (TAP08), volume 4966 of LNCS, pages 171181, Prato,
Italy, April 2008. Springer.
[16] M. D. Ernst, J. H. Perkins, P. J. Guo, S. McCamant, C. Pacheco, M. S. Tschantz,

and C. Xiao. The Daikon system for dynamic detection of likely invariants. Sci-
ence of Computer Programming, 2007.
[17] C. Flanagan, K. R. M. Leino, M. Lillibridge, G. Nelson, J. B. Saxe, and R. Stata.

Extended static checking for Java. In Proc. the ACM SIGPLAN 2002 Conference
on Programming language design and implementation, pages 234245. ACM
Press, 2002.
[18] M. Fowler. Inversion of control containers and the dependency injection pattern.
web, Jan 2004. http://martinfowler.com/articles/injection.
html.
73
[19] P. Godefroid. Compositional dynamic test generation. In Proc. of POPL07,
pages 4754, New York, NY, USA, 2007. ACM Press.
[20] P. Godefroid, P. de Halleux, A. V. Nori, S. K. Rajamani, W. Schulte, N. Tillmann,
and M. Y. Levin. Automating software testing using program analysis. IEEE
Software, 25(5):3037, 2008.
[21] P. Godefroid, N. Klarlund, and K. Sen. DART: directed automated random test-
ing. SIGPLAN Notices, 40(6):213223, 2005.
[22] P. Godefroid, M. Y. Levin, and D. Molnar. Automated whitebox fuzz testing.
In Proceedings of NDSS08 (Network and Distributed Systems Security), pages
151166, 2008.
[23] W. Grieskamp, N. Tillmann, and W. Schulte. XRT - Exploring Runtime for .NET
- Architecture and Applications. In SoftMC 2005: Workshop on Software Model
Checking, Electronic Notes in Theoretical Computer Science, July 2005.
[24] J. Henkel and A. Diwan. Discovering algebraic specifications from Java classes.
In Proc. 17th European Conference on Object-Oriented Programming, pages
431456, 2003.
[25] http://research.microsoft.com/en us/projects/contracts/. Melitta andersen and
mike barnett and and manuel fhndrich and brian grunkemeyer and katy king and
francesco logozzo. http://research.microsoft.com/projects/
contracts, 2008.
[26] Jonathan de Halleux. Mbunit. http://mbunit.com, 2007.
[27] JUnit development team. JUnit. http://www.junit.org/.
[28] C. Kaner, J. Falk, and H. Q. Nguyen. Testing Computer Software. International
Thomson Computer Press, 2nd edition, 1993.
[29] J. C. King. Symbolic execution and program testing. Commun. ACM, 19(7):385
394, 1976.
[30] B. Korel. A dynamic approach of test data generation. In IEEE Conference On
Software Maintenance, pages 311317, November 1990.
[31] B. Korel and A. M. Al-Yami. Assertion-oriented automated test data generation.
In Proc. the 18th international conference on Software engineering, pages 7180.
IEEE Computer Society, 1996.
[32] T. Mackinnon, S. Freeman, and P. Craig. Endotesting: Unit testing with mock
objects, May 2000. XP2000.
[33] J. McCarthy. Towards a mathematical science of computation. In IFIP Congress,
pages 2128, 1962.
[34] B. Meyer. Eiffel: The Language. Prentice Hall, New York, N.Y., 1992.
74
[35] Michael C. Two, Charlie Poole, Jamie Cansdale, Gary Feldman, James W.
Newkirk, Alexei A. Vorontsov and Philip A. Craig. NUnit. http://www.
nunit.org/.
[36] Microsoft. Visual Studio 2008. http://msdn2.microsoft.com/

vs2008/.
[37] M. Musuvathi, S. Qadeer, T. Ball, G. Basler, P. A. Nainar, and I. Neamtiu. Finding

and reproducing heisenbugs in concurrent programs. In OSDI, pages 267280,
2008.
[38] J. W. Newkirk and A. A. Vorontsov. Test-Driven Development in Microsoft .NET.

Microsoft Press, Apr. 2004.
[39] C. Pacheco, S. K. Lahiri, M. D. Ernst, and T. Ball. Feedback-directed random

test generation. In ICSE07, Proceedings of the 29th International Conference on
Software Engineering, Minneapolis, MN, USA, May 2325, 2007.
[40] Pex development team. Pex. http://research.microsoft.com/Pex,

2008.
[41] Pex development team. Stubs, Lightweight Test Stubs and Detours for .NET.
http://research.microsoft.com/Stubs, 2009.
[42] D. Saff, S. Artzi, J. H. Perkins, and M. D. Ernst. Automatic test factoring for Java.
In Proc. 20th ASE, pages 114123, New York, NY, USA, 2005. ACM Press.
[43] D. Saff, M. Boshernitsan, and M. D. Ernst. Theories in practice: Easy-to-write

specifications that catch bugs. Technical Report MIT-CSAIL-TR-2008-002, MIT
Computer Science and Artificial Intelligence Laboratory, Cambridge, MA, Jan-
uary 14, 2008.
[44] K. Sen and G. Agha. CUTE and jCUTE: Concolic unit testing and explicit path
model-checking tools. In CAV, pages 419423, 2006.
[45] K. Sen, D. Marinov, and G. Agha. CUTE: a concolic unit testing engine for C. In
Proc. of ESEC/FSE05, pages 263272, New York, NY, USA, 2005. ACM Press.
[46] N. Tillmann, F. Chen, and W. Schulte. Discovering likely method specifications.

In ICFEM, pages 717736, 2006.
[47] N. Tillmann and J. de Halleux. Pex - white box test generation for .NET. In Proc.
of Tests and Proofs (TAP08), volume 4966 of LNCS, pages 134153, Prato, Italy,
April 2008. Springer.
[48] N. Tillmann and W. Schulte. Parameterized unit tests. In Proceedings of the 10th
European Software Engineering Conference held jointly with 13th ACM SIG-
SOFT International Symposium on Foundations of Software Engineering,, pages
253262. ACM, 2005.
75
[49] N. Tillmann and W. Schulte. Unit tests reloaded: Parameterized unit testing with
symbolic execution. IEEE Software, 23(4):3847, 2006.
[50] G. Venolia, R. DeLine, and T. LaToza. Software development at microsoft ob-
served. Technical Report MSR-TR-2005-140, Microsoft Research, Redmond,
WA, October 2005.
[51] Wiki. Mock objects. www.mockobjects.com.

[52] Wikipedia. Luhn algorithm. http://en.wikipedia.org/wiki/
Credit_card_number, 2007. [accessed 15-November-2007].
[53] Wikipedia. Luhn algorithm. http://en.wikipedia.org/w/index.
php?title=Luhn_algorithm&oldid=163863211, 2007. [accessed 12-
October-2007].
[54] T. Xie, N. Tillmann, P. de Halleux, and W. Schulte. Fitness-guided path ex-
ploration in dynamic symbolic execution. Technical Report MSR-TR-2008-123,
Microsoft Research, Redmond, WA, September 2008.
76

Pex Unit Testing

Загружено:

Сведения о документе

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

Pex Unit Testing

Загружено:

Авторское право:

Доступные форматы

Parameterized Unit Testing

with Microsoft Pex

Nikolai Tillmann Jonathan de Halleux

3 Unit testing today 7

4 Parameterized unit testing 21

5 Selecting test inputs for parameterized unit tests 26

6 Dealing with the environment 43

7 Background on white box software testing (advanced) 50

8 Understanding Pex (advanced) 54

3 Unit testing today

Program 3.1 A unit test for the ArrayList class

public void AddTest() {

public void AddTest2() {

3.2 Benefits of unit testing

Design and specification Developers translate their understanding of the specification

Documentation The resulting unit tests are commonly considered as documentation

3.3 A critique of unit testing

3.4 Measurement of test quality: code coverage and assertions

3.5.1 Unit test frameworks

a custom attribute based system for tagging methods as unit tests,

automatic detection and execution of such unit tests,

Part 1: Creating a New Test Project

Part 2: Creating a Passing Unit Test

1. Right-click on the project node and select Add|New Test.

5. In the Test View window (Test|Windows|Test View), select the PassingTest

Part 3: Creating a Failing Unit Test

Open the Test View and execute both tests,

3. Go to the FailingTest method and hit F9 to set a debugger breakpoint.

Part 4: Creating a Negative Unit Test

1. Visual Studio Unit Test supports a special attribute, ExpectedExceptionAttribute,

4. In the Code Coverage Results window, enable source code coloring.

To summarize, in this exercise, we learned

how to author, execute and debug unit tests,

how to enable code coverage and analyze the results.

2. Run it and watch it fail,

4. Run the test again and see it succeed,

5. Refactor the code if needed.

We apply the TDD steps in the following.

Part 2: Add a Failing Test

6. Right-click on the Creditar project node and select Add|Class.

8. Execute the unit test and make sure that it fails.

Part 3: Run the Unit Test and Watch It Pass

To write such an implementation really means to follow the incremental idea of

Part 4: Continue the iteration

4 Parameterized unit testing

method sequence specify a scenario (as before), and

assertions encode the test oracle of a unit test (as before).

public void AddSpec2(

4.1 Separation of concerns

4.2 Coverage through test input generation

Program 4.2 ArrayList implementation in .NET

public class ArrayList

4.4 Patterns for parameterized unit testing

4.5 Test driven development by parameterized unit testing

2. The developer runs Pex on the PUT.

5.1 Automated exploratory testing

5.1.1 exploratory testing example

1 void Bar(int i, int j) {

1. reach line 4: i < 0

2. make the condition in line 4 evaluate to true: j = 123

1. Add a reference to the Microsoft.Pex.Framework.dll assembly to the

Part 2: Creating a Parameterized Unit Test

1. In the HelloWorldTest, add a new public instance method Parameterized-

Part 3: Run the Parameterized Unit Test