Acceptance Test Plan

for

Chaerea
(regarding dynamic fault detection)

Version 1.2
1 May, 2012
Prepared by:
Joe Brightbill
John Greco
Ethan Mullins
and
John Troy

Revision History:
Name

Date

Reason

Version

Joe Brightbill

05/01/12

Finalized for submission

1.2

Joe Brightbill

04/15/12

Cleaning up for beta testing

1.1

Joe Brightbill
John Greco
John Tory

02/07/12

Construction proceeding
completion of SRS.

1.0

Table of Contents
1. Introduction......................................................................................................................................5
1.1 Purpose.................................................................................................................................5
1.2 Scope....................................................................................................................................5
1.3 Definitions............................................................................................................................5
2. Testing Approach..............................................................................................................................7
2.1 Testing Objectives................................................................................................................7
2.2 Testing Structure..................................................................................................................7
3. Assumptions and Exclusions............................................................................................................8
3.1 Assumptions.........................................................................................................................8
3.2 Exclusions............................................................................................................................8
4. Functional Test Plan.........................................................................................................................9
4.1 Hardware Sensor Level........................................................................................................9
4.1.1 CPU Temperature Sensor (temp.sh).............................................................................9
4.1.2 Hard Drive Temperature Sensor (smarttemp.sh)..........................................................9
4.1.3 Transfers per Second Sensor (iostat-tps).......................................................................9
4.1.4 Reads per Second Sensor (iostat-reads)........................................................................9
4.1.5 Writes per Second Sensor (iostat-writes)......................................................................9
4.1.6 Bytes Read Sensor (iostat-read)..................................................................................10
4.1.7 Bytes Written Sensor (iostat-write).............................................................................10
4.2 OS Sensor Level.................................................................................................................10
4.2.1 Process Sensor (vmstat-procs*).................................................................................10
4.2.2 Memory Sensor (vmstat-mem*).................................................................................10
4.2.3 Swap Memory Sensor (vmstat_swap*)......................................................................10
4.2.5 Block Sensor (vmstat-block*)....................................................................................10
4.2.6 CPU Sensor (vmstat-spu*)..........................................................................................10
4.2.6 Load Average Sensor (load_average.sh)....................................................................11
4.2.7 Eth0 In Sensor (ifstat-eth0_in)...................................................................................11
4.2.8 Eth0 Out Sensor (ifstat-eth0_out)...............................................................................11
4.3 VM Sensor Level...............................................................................................................11
4.3.1 Br0 In Sensor (ifstat-br0_in)......................................................................................11
4.3.2 Br0 Out Sensor (ifstat-br0_out)..................................................................................11
4.4 Application Sensor Level...................................................................................................11
4.4.1 Survivor Space 0 Sensor (jstat-S0*)...........................................................................11
4.4.2 Survivor Space 1 Sensor (jstat-S1*)...........................................................................12
4.4.3 Eden Space Sensor (jstat-E*)......................................................................................12
4.4.4 Old Space Sensor (jstat-O*)........................................................................................12
4.4.5 Permanent Space Sensor (jstat-P*).............................................................................12
4.4.6 Young Generation GC Sensor (jstat-YGC).................................................................12
4.4.7 Full GC Sensor (jstat-FGC*).......................................................................................12
4.4.8 Garbage Collection Time Sensor (jstat-GCT).............................................................12
4.5 Dispatcher Level................................................................................................................13
4.5.1 Dispatcher (dispatcher.c)............................................................................................13
4.5.2 Sensor (sensor.c)..........................................................................................................13
4.5.3 Sensor Configuration (sensors.json)..........................................................................13
4.6 Collector Level...................................................................................................................13
4.6.1 Moderated Individually with Graphite.......................................................................13

4.7 Detector Level....................................................................................................................13

4.7.1 Detector (math.tac).....................................................................................................13
4.8 Presenter Level...................................................................................................................14
4.8.1 Presenter (stack.py).....................................................................................................14

1. Introduction
1.1

Purpose
This document will provide testing specifications for the Chaerea dynamic fault

detection system. This system will diagnose faults in multi-level systems by reading in sensor
values at four different system levels (hardware, operating system, vm, application) and,
using these readings, create a hyperellipse that will encompass the detected values. This
enclosure will then be used in future instances to dynamically determine whether or not sensor
readings at a given time are indicative of some sort of system fault (based on their location
external or internal to the hyperellipse). For further details in regards the testing, requirements,
see the Software Requirements Specifications document for Chaerea.

1.2

Scope
This document outlines the acceptance test plan for the dynamic fault detection system

using computational geometry. There are not really any end users for this project so the intended
audience is just that which is comprised of the designers, testers and anyone else with whom
they may be working intimately.

1.3

Definitions
Block Device - A storage device that supports reading and (optionally) writing data in
fixed-size blocks, sectors, or clusters.
Computational Geometry - The study of algorithms to solve certain problems concerning
sets of points.
Eden Space - The pool from which memory is initially allocated for most objects.
Ellipse - An enclosed one-sided figure expressed by the equation:
(x^2/a^2) + (y^2/b^2) = 1
Graphite - an open source Scalable Realtime Graphing application in Python that can be
used to transfer data.
Hyperellipse - An n-dimensional generalization of an ellipse.
Hyperrectangle - An n-dimensional generalization on a rectangle.
Ifstat - A Unix utility that reports network interface activity.
Json - A lightweight data-interchange format.
Load Average - The average system load over a period of time.
Middleware - Software that connects software components.
Minimum Volume Enclosing Ellipse - The ellipse with the smallest possible volume that

encloses a given set of points.

Smartctl - A control and monitor utility for SMART disks.
Survivor Space - The pool containing objects that have survived the garbage collection
of the Eden space.
Uptime - A Unix utility that displays load average and several other bits of information
on one line.
VM - Isolated guest operating system inside a normal host OS.
VMStat - A Linux tool used for monitoring virtual memory.

2. Testing Approach
This section describes the overall approach and particular techniques that will be used
during the Acceptance Test Plan for Chaerea and any constraints that may apply. Testing will be
done in a continuous fashion, and therefore there will be no start and end dependencies or
expectations. Rather, there will just be expectant methods of testing for each individual file.

2.1

Testing Objectives
Validate the elements of the Chaerea fault detection system and verify that it performs in

accordance with the requirements set forth in the Software Requirements Specification
document.

2.2

Testing Structure
In the Software Requirements Specification, we separated requirements by files within

given modules. In this document, we will detail how each file listed in the Software
Requirement Specification document will be tested.

3. Assumptions and Exclusions

3.1

3.2

Assumptions

Software Requirement Specification document completed.

Scripts compile and run before they are tested.

Exclusions

The interface requirements included in Software Requirement Specification

document.

Structural integrity of the source code.

4. Functional Test Plan

The following will list the test cases for each file noted in the Software Requirements
Specification document. The modular separation has been retained for ease of flow between
documents. The description associated with each file will describe how it will be tested and
what results will indicate a successful test run. These requirements will be minimal, because of
the unknown variability in results for some of these things. Just because they meet the
requirements does not mean that they will not be improved upon later.

4.1

Hardware Sensor Level

4.1.1 CPU Temperature Sensor (temp.sh)

Sensor will be run, pushing output to the console, for one minute.

If the output is between a reasonable temperature (0 to 100), test will be

considered a success.

4.1.2 Hard Drive Temperature Sensor (smarttemp.sh)

Sensor will be run, pushing output to the console, for one minute.

If the output is between a reasonable temperature (0 to 100), test will be

considered a success.

4.1.3 Transfers per Second Sensor (iostat-tps)

Sensor will be run, pushing output to the console, for one minute.

If the output is a non-negative number (reasonable in accordance with

what is happening on the system) we will consider this a success.

4.1.4 Reads per Second Sensor (iostat-reads)

Sensor will be run, pushing output to the console, for one minute.

If the output is a non-negative number (reasonable in accordance with

what is happening on the system) we will consider this a success.

4.1.5 Writes per Second Sensor (iostat-writes)

Sensor will be run, pushing output to the console, for one minute.

If the output is a non-negative number (reasonable in accordance with

what is happening on the system) we will consider this a success.

4.1.6 Bytes Read Sensor (iostat-read)

Sensor will be run, pushing output to the console, for one minute.

If the output is a non-negative number (reasonable in accordance with

what is happening on the system) we will consider this a success.

4.1.7 Bytes Written Sensor (iostat-write)

4.2

Sensor will be run, pushing output to the console, for one minute.

If the output is a non-negative number (reasonable in accordance with

what is happening on the system) we will consider this a success.

OS Sensor Level

4.2.1 Process Sensor (vmstat-procs*)

Sensor will be run, pushing output to the console, for one minute.

If the sensor returns all non-negative integers we will consider this a

success (the values output can be arbitrarily high, but should be close tot
he number of processes we believe to be running (could use something
like top).)

4.2.2 Memory Sensor (vmstat-mem*)

Sensor will be run, pushing output to the console, for one minute.

If the sensor returns non-negative values that do not exceed the memory
threshold of the system, we will consider this a success.

4.2.3 Swap Memory Sensor (vmstat_swap*)

Sensor will be run, pushing output to the console, for one minute.

If the sensor returns non-negative values we will consider this a success

(the value output can be arbitrarily high.)

4.2.5 Block Sensor (vmstat-block*)

Sensor will be run, pushing output to the console, for one minute.

If the sensor returns all non-negative values we will consider this a

success (the value output can be arbitrarily high.)

4.2.6 CPU Sensor (vmstat-spu*)

Sensor will be run, pushing output to the console, for one minute.

If the sensor returns all non-negative values we will consider this a

success (the value output can be arbitrarily high.)

4.2.6 Load Average Sensor (load_average.sh)

Sensor will be run, pushing output to the console, for one minute.

If the sensor returns all non-negative values we will consider this a

success (the value output can be arbitrarily high.)

4.2.7 Eth0 In Sensor (ifstat-eth0_in)

Sensor will be run, pushing output to the console, for one minute.

If the sensor returns non-negative value below 1 Gb, we will consider this
a success

4.2.8 Eth0 Out Sensor (ifstat-eth0_out)

4.3

Sensor will be run, pushing output to the console, for one minute.

If the sensor returns non-negative value below 1 Gb, we will consider this
a success

VM Sensor Level

4.3.1 Br0 In Sensor (ifstat-br0_in)

Sensor will be run, pushing output to the console, for one minute.

If the sensor returns non-negative value below 1 Gb, we will consider this
a success

4.3.2 Br0 Out Sensor (ifstat-br0_out)

4.4

Sensor will be run, pushing output to the console, for one minute.

If the sensor returns non-negative value below 1 Gb, we will consider this
a success

Application Sensor Level

4.4.1 Survivor Space 0 Sensor (jstat-S0*)

Sensor will be run, pushing output to the console, for one minute.

If the sensor returns all non-negative values we will consider this a

success (the value output can be arbitrarily high.)

4.4.2 Survivor Space 1 Sensor (jstat-S1*)

Sensor will be run, pushing output to the console, for one minute.

If the sensor returns all non-negative values we will consider this a

success (the value output can be arbitrarily high.)

4.4.3 Eden Space Sensor (jstat-E*)

Sensor will be run, pushing output to the console, for one minute.

If the sensor returns all non-negative values we will consider this a

success (the value output can be arbitrarily high.)

4.4.4 Old Space Sensor (jstat-O*)

Sensor will be run, pushing output to the console, for one minute.

If the sensor returns all non-negative values we will consider this a

success (the value output can be arbitrarily high.)

4.4.5 Permanent Space Sensor (jstat-P*)

Sensor will be run, pushing output to the console, for one minute.

If the sensor returns all non-negative values we will consider this a

success (the value output can be arbitrarily high.)

4.4.6 Young Generation GC Sensor (jstat-YGC)

Sensor will be run, pushing output to the console, for one minute.

If the sensor returns a non-negative value for the events and a reasonable value
for the time (it does not exceed the uptime of the system) we will consider this a
success.

4.4.7 Full GC Sensor (jstat-FGC*)

Sensor will be run, pushing output to the console, for one minute.

If the sensor returns a non-negative value for the events and a reasonable value
for the time (it does not exceed the uptime of the system) we will consider this a
success.

4.4.8 Garbage Collection Time Sensor (jstat-GCT)

Sensor will be run, pushing output to the console, for one minute.

If the sensor returns a reasonable non-negative value (it does not exceed the
uptime of the system) we will consider this a success.

4.5

Dispatcher Level

4.5.1 Dispatcher (dispatcher.c)

If the sensors that this has been configured to initiate are initiated in
reasonable time and report data as shown in the individual sensor script
tests, we will consider this a success.

4.5.2 Sensor (sensor.c)

If the sensors are able to maintain and communicate their data accurately
and efficiently within this framework, we will consider this a success.

We should be able to add and remove sensors and access them using their
process ID.

4.5.3 Sensor Configuration (sensors.json)

4.6

If the sensors receive the arguments specified in sensors.json and report

their data in the recognized format, we will consider this a success.

Collector Level

4.6.1 Moderated Individually with Graphite

4.7

Initiated after the more individual tests have been tested.

If the metrics are collected and communicated as expected and in the

amount of time expected, we will expect this to be a success.

Detector Level

4.7.1 Detector (math.tac)

Enter random data into the code for the ellipse, hyperrectangle and
convex hull. Confirm manually that the shapes are created correctly.

Test the membership of points we know to be both internal and external

to the structure created. Make sure that the program returns the correct
membership (Y/N),

Ensure that the script allows for accurate selection of what shape to create
and creates the requested shape.

When the above is complete in a reasonable time and without throttling

the system, we will consider this a success.

4.8

Presenter Level

4.8.1 Presenter (stack.py)

Use industry standard JSON validator to validate output file.

View the web page to confirm that something is being displayed.

Confirm that the values displayed are being relayed accurately and in a
timely fashion (with graphs displaying the proper associated values).

If a fault is detected, ensure that the interface displays an acceptable

message indicating this fact.