Understandablity of Concrete versus Parametric Test Cases By Touseef Tahir, Ali Jafar, Mohan Maharajan

Lahore

Introduction to Unit Testing



Concrete Testing AND Parametric Testing

h h h h h h

The research Questions Hypothesis Planning and design of Experiment Experiment Analysis and discussion Conclusion

A test case is a description of a specific interaction that a tester will have in order to test a single behavior of the software. Test cases are very similar to use cases, in that they are step-by-step narratives which define a specific interaction between the user and the software. Test case is a triplet [I, S, O] where  I is input data  S is state of system at which data will be input  O is the expected output Test suite is set of all test cases

Testing of functional and non functional behavior of the software programs. Verifies correctness of the units of program in isolation. Detect and remove faults earlier from the specified units. Provides 100% code coverage, checks all the corner cases and detects error in components logic

executes for a single test input at an instance. execute a method of a class under test with fixed arguments and verify that it returns the expected result or not. The concrete input values are provided to the unit to determine its failure or success.

Numerous test inputs that has to be examined and revised. Tedious, labor intensive, error-prone, time consuming, unlikely to cover all possible behavior and often difficult task.

Does not ensure the high code coverage since only single examplary input value is provided as an input.  Increase in effort, cost, time, numerous test cases, and numerous iterations.

Parameters as an input to the unit of code. Exercised different code paths for the set of test inputs Detects more faults than concrete test cases. High code coverage with small test suite. Can generate more test instances and executes automatically. Less time consuming, and likely to cover all the possible behavior.

Assist in reusability, maintainability

evolution,

and

Hard to measure since depends on cognitive behavior of human

Documentation of source code Understanding of Structure and component Code Complexity Psychological Complexity

Reading Scalability for general unit test questions Closed ended questionnaires

Types Subjects

Definition MSc. Software Engineering MSc. Computer Sciences MSc Security engineering Class Room Document containing Questioners (Close ended question) Factorial Design Java, C# JUnit, NUnit, Pex JUnit Concrete Testing (6 Subjects) JUnit Parameterized Testing(6 Subjects) NUnit Concreter(6 Subjects) Pex Parameterized Testing (6 Subjects)

Environment Apparatus Experimental Design Method Programming Languages Testing Tools Groups

Variable type Independent

Variables Subjects Programming languages (Java, C#)

Treatment Dependent

JUnit, NUnit, Pex Concrete Testing Parameterized Testing Understandability Time

Extraneous

Accuracy (Correctness) Participants Gender/Sex, Age Experiment situation(noise, temperature) Equipment (fault equipment, calibration) The time of the experiment Conduct of experiment Carry over effect

Understandablity of Concrete versus Parametric Test Cases

Purposed Hypothesis H0: Concrete test cases are difficult to understand than Parameterized test cases and thus make overall software testing difficult. Alternative Hypothesis H1: Parameterized test cases are difficult to understand than concrete test cases and thus make overall software testing difficult.

General questions Questions regarding concrete test cases and parameterized test cases related to the code and Test Code.

4 3.5 3 2.5 2 1.5 1 0.5 0 SCALE

Note: CC=C# Concrete, JP=Java Concrete Time is measured in Minute, Correct Answer out of 37

Note: CP=C# Parameterized JP=Java Parameterized Time is measured in Minute, Correct Answer out of 37

Descriptive Statistics Hypothesis Testing Wilcoxon Signed Test

Summarize the available data

Accept OR Reject hypothesis

Note: As there is one tie in the data so the number of participants for this group will be 11 and the table value is 10

Time
 

No significant difference Not practical to conclude on the basis of time Significant difference as well as participants

Correctness ratio


We reject null hypothesis



Correctness ratio and general testing questions we concluded that

h

More general test cases are harder to understand than concrete ones and thus would lead to tests that are harder to understand.

 Understandability of concrete test cases versus general test cases using mixed approach.  Using practitioners besides using students.  Using more than two programming languages, more subjects and more examples.

After the early analysis of data it was seen that subjects have taken different amount of time to solve a specific example. In some cases the time difference was huge so it was not fitted for parametric hypothesis testing methods. The data was not in intervals as well so it was decided non parametric statistics was used on the collected data. The only factor involved in experiment was unit testing with two different treatments which were Parameterized test cases and Concrete test cases. The best suitable method for hypothesis testing was Wilcoxon signed test