Cantata Technical Brief!

Introduction
As software systems grow more complex, the need to test software thoroughly at every stage of the
development lifecycle, is becoming ever more apparent. Software project failures continue to show
the consequences of mediocre or non-existent testing.
Most software developers now appreciate the need to test software thoroughly and systematically,
but there is still a problem; testing is expensive. So despite a virtuous desire to improve quality,
developers are still faced with the dilemma of choosing between quality and cost.
Cantata now brings professional testing within the reach of all software developers by greatly
increasing the productivity of the testing process. Cantata has been specifically designed to offer
developers a high-productivity solution to the verification of C software in a unique, integrated toolset.
Cantata provides facilities for:
!! Dynamic Testing: executing the software under test in order to verify its compliance with
requirements -- does it do what it should?
!! Coverage Analysis: measuring the proportion of software exercised by dynamic testing -- are
the tests thorough?
!! Static Analysis: examining source code to assess its complexity and use of language constructs
-- is the software maintainable? Does it meet coding standards?
The Cantata toolset provides a wealth of facilities in these three areas.

1
Cantata - A Functional Overview
Cantata
!
Dynamic Testing!
Dynamic testing of software during its
development (unit and integration testing) has
repeatedly been shown to be highly effective in
finding defects. More importantly, this type of
testing finds defects early in the lifecycle,
minimizing the cost implications of faults. Until
now, testing at these levels has been ignored by
all but the most rigorous of software developers
because of its cost; unit and integration testing
could account for a very considerable proportion
of the effort expended in a software project.
!
Cantata’s dynamic testing facilities offer both
productivity and ease of use, allowing C developers the best of both worlds: high quality software at a
reasonable cost. Cantata allows testing to be performed in an intuitive way making the tool
exceptionally easy to use.
Cantata provides comprehensive facilities for all forms of unit and integration testing. The tool has
been specifically designed to operate in both host and target systems and so allow full portability of
tests between these environments.

Coverage Analysis
As all programmers know, it is very easy to
develop a software test suite that works. A
more important question is “Has the
software been fully exercised?”. Coverage
analysis answers this question by
measuring the proportion of the code that
has been executed.
Cantata coverage analysis facilities allow
the user to assess the coverage of
individual software units (functions),
programs/tasks and entire software
systems. Cantata provides all of the
commonly quoted coverage metrics.
Coverage information is reported in a form
that is easy to understand, allowing the
user to identify unexecuted code.
In addition to checking the usage of the program code, Cantata allows the user to check the
coverage of important data items. Using this facility it is possible for the user to automatically verify
that a particular variable has held a series of required values.
Coverage results can be "built-in" to the overall test pass/fail criteria, ensuring that code is fully
exercised as part of the dynamic testing process.

2
Cantata - A Functional Overview

Static Analysis
Even software that has been thoroughly
dynamically tested can have its problems.
Static analysis gives the developer useful
information on non-functional qualities of
the source code, such as its
maintainability and compliance with
coding standards.
Cantata’s static analysis facilities allow
the user to check that software complies
with coding standards and is within
acceptable limits of complexity.
Cantata provides a host of static analysis
measures against which the user can !
evaluate code. These include a large number of ‘common sense’ metrics, such as the number of
code statements, as well as measures of complexity, such as the McCabe’s cyclomatic complexity
metric.
Analysis data can be reported to the user in a variety of formats, but more importantly metrics can be
verified as part of the overall testing process. Users can specify their own language standards and
automatically ensure that all units tested are fully compliant.
Thus, users of Cantata can define their own quality model based upon dynamic testing, coverage
analysis and static analysis criteria. This model can be implemented in a single ‘test script’ that
verifies all facets of the software unit under test. The following sections describe Cantata’s dynamic
testing and analysis facilities in more detail.
It is important to note here that, while Cantata is principally a tool for use with C, it does
have the capability to test simple C++. However, most C++ developers will choose its sister
product Cantata++ for more demanding C++ testing applications.

3
Cantata - Dynamic Testing
The Testing Problem!
Dynamic testing, at the conceptual level, is a generic problem. This is illustrated in the diagram
below.
The first requirement of all software
testing, is that there must be some
form of specification for the software; Software
it is not possible to test without a
model of the software’s required
behavior. The developer must not only
implement the specification to produce Test Results
executable software, but also conceive
a set of test cases to properly test the
Specification
? Pass/Fail

software.
Given the prerequisites of the software
to be tested and test cases with which Test Cases
to exercise it, the next problem to
address is implementing the tests.
What is required is a tool which will
take the software to be tested and test case data, and produce test results.
Any such tool must be able to handle test cases generated using any of the popular functional
(black-box) or structural (white-box) test case selection techniques. It must also be a practical tool;
able to test a software unit in isolation, before the other units which it references are available to be
tested. Integration testing is another important requirement; testing that any integrated set of
software units performs as expected. Finally, the tool must apply best practice in testing; checking
not only for expected behavior (positive testing) but also that the software does not do anything
unexpected, such as corrupting memory (negative testing).

The Cantata Solution

Cantata solves the generic testing
problem for unit and integration testing Software
of C software.
Cantata tests are controlled by a ‘test
script’ which can be either coded by
the user or generated by Cantata’s Cantata Test Results
Specification Pass/Fail
Test Script generator (CTS). The test CTH
script is compiled and linked with
Cantata Test Harness (CTH) and the
software under test. This produces an Test Cases
executable program which, when run,
Manual
produces a test results file. Generation

The benefits of this approach are: Cantata Test Script

• Productivity: Developers no CTS
longer need to manually code all
the software that supports their
unit and integration tests, resulting in large productivity increases.
• Documentation: The tool provides documented proof of testing.
• Repeatability: Cantata tests can be easily repeated by QA or other staff.
• Maintainability: Tests Scripts are easy to understand and can be updated as the software is
enhanced.
• Quality: Systematic testing leads to a higher quality product.

4
Cantata - Dynamic Testing

Test Script Generation

Cantata Test Script generator (CTS) is a programming ‘aid’ which prepares dynamic test scripts
for execution with CTH. CTS takes information from a test case definition file (which is produced
by the user or a third party tool) and generates a test script. The test case definition file specifies
test cases for which initial conditions are established and expected results defined.
CTS greatly simplifies and expedites the task of developing a dynamic test suite. CTS allows the
user to concentrate upon the important aspects of testing: the data used to exercise the software,
rather than worrying about implementation details.
If the user prefers, CTS can also be used to produce comprehensive test script templates which
can be used as the basis of manually coded test scripts.
CTS works by scanning the test case definition file and the software under test to produce the test
script.
CTS is capable of generating test scripts
for software coded in either ANSI C or
C++. The format of the test case definition
file is one of several standard formats Software
generated by third party test case
generation tools, or an easy to comprehend
format which can be coded manually.
Cantata
All test scripts produced by CTS feature full Test Script
CTS
positive and negative data checking. CTS
automatically codes check routines for user
defined types and stubs for external Test Cases
functions/classes. Test scripts are easy to
understand and can be manually modified,
if required.
If the user prefers, CTS can also be used
to produce comprehensive test script
templates which can be used as the basis
of manually coded test scripts.

The Test Harness

Cantata Test Harness (CTH) provides facilities to run, verify the results of, document, and repeat
dynamic tests. CTH consists of a set of library directives, which are accessed from the test script.
The software under test is called from the test script and the effects of the call on the software
environment are checked by the CTH directives embedded in the script.
A Cantata test script is thus simply a C main function which exercises the software under test. The
use of C for test scripts, together with a simple script syntax means that Cantata test scripts are very
easy to understand and modify.

5
Cantata - Dynamic Testing
"#$%!&!'&%(&(&!($)(!*)!+,%-!&!+$),.()!/*.$!*)!0+12,3$2!2$(&*.*%4!$5$+6!)($0!1/!(#$!($)(!&%2!#*4#.*4#(*%4!
&%6! /&*.,+$)7! ! 8! +$),.()! (&9.$! *)! &.)1! 2*)0.&6$2! ),::&+*;*%4! (#$! +$),.()! /1+! $&3#! ($)(! 3&)$! &%2!
0+15*2*%4! (1(&.! /*4,+$)7! <1)(!
*:01+(&%(! 1/! &..-! &%! 15$+&..!
)(&($:$%(! 1/! "#$"! %&$$! '(! )&*+! *)!
0+15*2$2! &%2! &.)1! +$(,+%$2! (1! (#$!
31::&%2!)#$..7!
!
Test scripts can be either coded
manually or be produced by
Cantata’s Test Script generator
(CTS). CTS allows the user to
generate test scripts quickly, but
many users prefer to use CTH
manually, in order to retain full
control over the dynamic testing
process.

Checking Values
Probably the single most important aspect of dynamic testing is checking that the outputs from the
software under test are as expected. CTH verifies data using CHECK directives. These cause the
comparison of a data item with its expected value.
Check directives are provided within CTH for all C/C++ standard types. In addition, CTS
automatically generates check directives for user defined types, such as structures and enumerated
types, using CTH Check primitives.
Cantata has other checking facilities too: For example, for comparing blocks of memory, and for
checking an ‘observation’, i.e. the occurrence of a physical event.

Simulating External Software

Developers often need to test software units in isolation from one another. This means that calls to
external units and external data must be simulated.
Cantata provides a sophisticated yet easy-to-use set of simulation facilities for external code and
data. External calls can be simulated, ensuring that they are made in the expected order and that
input parameters have the correct values at each call. Return values can be individually set on
different calls to the same simulated function. Similarly, external data areas may be simulated and
checked.

Timing Analysis
In many systems it is not only the correct functioning of software that determines its acceptability.
Real-time applications need to perform certain activities both correctly and within defined time
constraints. Cantata permits execution times to be recorded and tests passed or failed depending on
the real-time performance of the software under test.

6
 !Cantata
Cantata - Dynamic Testing!
!
!
!
!
!

Cantata Results
8(! $=$3,(*1%! (*:$-! (#$! 0+14+$))! 1/! (#$! ($)(! +,%! *)! :&+>$2! 96! 1,(0,(! /+1:! (#$! '?@! 2*+$3(*5$)7! ! 8%6!
$5$%(! A#*3#! 3&%! 9$! 3.&))$2! &)! B,%$=0$3($2C! *)! #*4#.*4#($2! A*(#! >>-! &%2! &%! &00+10+*&($! 2*&4%1)(*3!
:$))&4$7!!D1+!$=&:0.$-!&!3#$3>!A#*3#!/&*.)!A*..!9$!:&+>$2!A*(#!>>FAILED-!&%2!&!2*&4%1)(*3!A*..!4*5$!
91(#!(#$!&3(,&.!&%2!$=0$3($2!5&.,$)!1/!(#$!*($:!9$*%4!3#$3>$27!
!
?#$!91=!9$.1A!31%(&*%)!&!(60*3&.!)$3(*1%!1/!&!'?@!1,(0,(!/*.$E!
========================================================================
CTH v3.0 (c) 1993 IPL Information Processing Ltd
------------------------------------------------------------------------------
Test Results For : example
Results File : example.ctr
Tests Run At : Thu Feb 17 09:38:32 1994
========================================================================
------------------------------ Start Test 001 -------------------------------
EXECUTE: my_function,
Expected calls = 1
START_STUB : my_stub
CALL_REF/ACTION: Action 1, Call 1
Check PASSED : my_stub_string
Item = "Hello, world"
END_STUB : my_stub
DONE : my_function
Check FAILED : my_external >>
Expected 0x0000712B 28971
Item 0x000080E8 33000
--------------------------------- End Test 001 ------------------------------

!
8(!(#$!$%2-!&!(&9.$!1/!+$),.()!4*5$)!(#$!31:0.$($!)(&(*)(*3)!91(#!/1+!$&3#!($)(!3&)$!&%2!/1+!(#$!15$+&..!
($)(7!
================================================================================
Tests Completed At : 09:38:32
--------------------------------------------------------------------------------
Test Script Checks Checks Checks Stubs Paths Assertions Status
Errors Passed Failed Warning Failed Failed Failed
--------------------------------------------------------------------------------
PTE 0 0 0 0 0 0 0 PASS
001 0 0 1 0 0 1 0 >> FAIL
ANS 0 0 0 0 0 0 0 PASS
--------------------------------------------------------------------------------
Total 0 0 1 0 0 1 0 >> FAIL
================================================================================

This table of results shows that test 1 has failed due to a failed data check and, therefore, the overall test has failed.

7
Cantata - Analysis
!
!
Cantata Analysis
'&%(&(&!8%&.6)*)!F'?8G!0+15*2$)!(#$!,)$+!A*(#!'15$+&4$!8%&.6)*)!&%2!H(&(*3!8%&.6)*)!/$&(,+$)7!?#$!
/1..1A*%4!)$3(*1%)!2$)3+*9$!(#$)$!(A1!($3#%*I,$)!&%2!#1A!(#$6!&+$!&22+$))$2!96!'&%(&(&7!

Coverage Analysis
'15$+&4$! &%&.6)*)! :$&),+$)! (#$! 0+101+(*1%! 1/! )1/(A&+$! $=$3,($2! 2,+*%4! 26%&:*3! ($)(*%4-! 4*5*%4!
2$5$.10$+)!&%2!,)$+)!&.*>$!31%/*2$%3$!(#&(!($)()!&+$!31:0+$#$%)*5$7!
?#$+$!&+$!:&%6!(60$)!1/!315$+&4$!:$(+*37!!?#$!1%$)!),001+($2!96!'&%(&(&!#&5$!9$$%!)#1A%!(1!9$!
(#1)$!(#&(!31%(+*9,($!:1)(!$//$3(*5$.6!(1!$++1+!2$($3(*1%7!!?#$6!&+$E!
• $"&"#,#-"!.'/#(&0#-!A#*3#!*)!&!:$&),+$!1/!(#$!0+101+(*1%!1/!312$!)(&($:$%()!$=$3,($2!2,+*%4!&!
($)(!+,%7!
• 1#.*$*'-! F1+! 2(&-.3G! .'/#(&0#-! A#*3#! *)! &! :$&),+$! 1/! (#$! 0+101+(*1%! 1/! 2$3*)*1%! 1,(31:$)!
$=$3,($27!!J$3*)*1%)!*%3.,2$!if-!switch-!while!&%2!(#$!?:!10$+&(1+7!
• 2''+#&-! #4%(#$$*'-! .'/#(&0#-! A#*3#! :$&),+$)! (#$! 0+101+(*1%! 1/! 31%2*(*1%&.! ($)()! *%! 31:0.$=!
2$3*)*1%)!A#*3#!#&5$!9$$%!$=$3,($27!!D1+!$=&:0.$-!*%!(#$!$=0+$))*1%
if (a>10 && a<20) {...
*(!*)!01))*9.$!(1!($)(!(#&(!(#$!K11.$&%!10$+&(1+ &&!*)!/,..6!$=$+3*)$2!
• .&++! %&*(! .'/#(&0#-! A#*3#! :$&),+$)! (#$! 0+101+(*1%! 1/! 3&..)! (1! 1(#$+! /,%3(*1%)! A#*3#! #&5$! 9$$%!
$=$+3*)$27! ?#*)! :$&),+$! *)! ,)$/,.! &(! *%($4+&(*1%! ($)(*%4-! *%! 5$+*/6*%4! (#&(! *%($+/&3$)! #&5$! 9$$%!
0+10$+.6!$=$+3*)$27!
• .&++!.'/#(&0#-!A#*3#!*)!&!3#$3>!(#&(!&..!$=0$3($2!/,%3(*1%)!#&5$!9$$%!3&..$27!!?#*)!*)!$)0$3*&..6!
+$.$5&%(!/1+!*%($4+&(*1%!($)(*%4!&%2!/1+!($)(*%4!'LL!3.&))$)7!
• 1&"&!/&+5#!.'/#(&0#-!A#*3#!3#$3>)!(#&(!0+14+&:!5&+*&9.$)!#&5$!#$.2!&!)$+*$)!1/!F,)$+!2$/*%$2G!
5&.,$)! 2,+*%4! (#$! ($)(*%4! 0+13$))7! ?#*)! /&3*.*(6! *)! ,)$/,.! *%! 3#$3>*%4! (#&(! 2&(&! 91,%2&+6! 5&.,$)!
#&5$!9$$%!$=$+3*)$2!&%2!(#&(!.110)!#&5$!9$$%!&2$I,&($.6!($)($27!

Cantata - High Integrity Package

An optional “Cantata High Integrity Package” is available which gives the safety-critical software
developer two additional benefits:
• MC/DC Coverage (Modified Condition/Decision Coverage) - as required by the RTCA DO-178B
standard for airborne software systems.
• The right to audit IPL’s Cantata product development and associated procedures, to ensure
that the requirements of DO-178B have been met.
This package is available as an additional cost option to the basic Cantata product and will be of
interest to all developers of critical systems - not just those developing to DO-178B.

8
Cantata - Analysis!
!
!
!
!
!

Static Analysis
H(&(*3!8%&.6)*)-!&)!(#$!%&:$!),44$)()-!*%51.5$)!(#$!)(&(*3!*%)0$3(*1%!1/!(#$!)1,+3$!312$!*%!1+2$+!(1!
0+15*2$!&%!&))$)):$%(!1/!5&+*1,)!%1%M/,%3(*1%&.!/$&(,+$)!+$.&(*%4!(1!(#$!)1/(A&+$7!'&%(&(&!3&%!&))*)(!
2$5$.10$+)!*%!(A1!*:01+(&%(!&+$&)E!
• &))*)(&%3$!*%!:&*%(&*%*%4!.'1*-0!$"&-1&(1$!
• 19N$3(*5$!:$&),+$:$%(!1/!.'1#!.',%+#4*"6!&%2!$"(5."5(#7!
!
Coding Standards
O+4&%*;&(*1%)!&+$!*%3+$&)*%4.6!&210(*%4!312*%4!)(&%2&+2)!&)!&!:$&%)!1/!*:0+15*%4!)1/(A&+$!I,&.*(6!
&%2! :&*%(&*%&9*.*(67! ! @1A$5$+-! ,%.$))! (#$)$! )(&%2&+2)! 3&%! 9$! 5$+*/*$2! *%! &%! &,(1:&($2! A&6! *(! *)!
2*//*3,.(!(1!:&*%(&*%!(#$:!$//$3(*5$.67!
!
'&%(&(&! 0+15*2$)! (#$! 2$5$.10$+! A*(#! &33$))! (1! :&%6! )(&(*3! &%&.6)*)! :$(+*3)! 1%! (#$! ,)$! 1/! 312*%4!
31%)(+,3()7! ! ?#$)$! :$(+*3)! 3&%! 9$! $=&:*%$2! &)! 0&+(! 1/! &! +$01+(-! 1+! 3#$3>)! 1%! (#$*+! 5&.,$! 3&%! 9$!
*%31+01+&($2!*%(1!15$+&..!($)(!0&))P/&*.!3+*($+*&-!&)!0&+(!1/!&!($)(!)3+*0(7!D1+!$=&:0.$-!,)$+)!:&6!3#$3>!
(#&(!%1!goto!)(&($:$%()!F1+!.&9$.)G!&+$!,)$2-!(#&(!1%.6!1%$!return!)(&($:$%(!*)!0+$)$%(!&%2!(#&(!
(#$+$!&+$!%1!switch!)(&($:$%()!A*(#1,(!&!default7!
!
'&%(&(&C)!2&(&!/.1A!:$(+*3)!&..1A!(#$!,)$+!(1!3#$3>!1%!(#$!,)$!1/!.13&.-!4.19&.!&%2!0&+&:$($+!2&(&!
*($:)7! '&%(&(&! 3&%! #*4#.*4#(! 2&(&! &+$&)! &%2! (60$)! A#*3#! &+$! 2$3.&+$2! 9,(! %1(! ,)$27! '&%(&(&! &.)1!
0+15*2$)!&!)0$3*&.!)$(!1/!:$(+*3)!+$.&(*%4!(1!(#$!O9N$3(!O+*$%($2!/$&(,+$)!1/!'LL!7!
!
Q)$+)!3&%!2$/*%$!(#$*+!1A%!)(&(*3!&%&.6)*)!:$(+*3)!(1!3#$3>!1%!(#$!,)$!1/!312$!31%)(+,3()!A#*3#!&+$!
%1(!1(#$+A*)$!:$&),+$2!96!'&%(&(&7!D1+!*%)(&%3$-!(#$)$!/&3*.*(*$)!3&%!9$!,)$2!(1!+$)(+*3(!&33$))!(1!
3$+(&*%!.*9+&+6!+1,(*%$!3&..)!F),3#!&)!!mallocG7!
!
Code Complexity
8)! &! :$&%)! 1/! *%3+$&)*%4! (#$! :&*%(&*%&9*.*(6! 1/! )1/(A&+$-! 1+4&%*;&(*1%)! &+$! (&>*%4! &%! *%($+$)(! *%!
19N$3(*5$! :$&),+$:$%(! 1/! 312$! 31:0.$=*(67! <&%6! 1/! (#$! +$314%*;$2! B&3&2$:*3C! :$(+*3)! &+$!
),001+($2!96!'&%(&(&7!!?#$)$!*%3.,2$E!!
!! <3'&9$C)!:$&),+$!&%2!<6$+)C!$=($%)*1%!
!! R))$%(*&.!<3'&9$C)!
!! @&%)$%C)!:$&),+$!1/!)1/(A&+$!31:0.$=*(6!96!(#$!0&*+!
!! @&.)($&2C)!)1/(A&+$!)3*$%3$!:$(+*3)!
!! @&++*)1%C)!)310$!+&(*1!
!
'&%(&(&! &.)1! ),001+()! &! %,:9$+! 1/! 31::1%! )$%)$! 31:0.$=*(6! :$(+*3)-! ),3#! &)! &5$+&4$! &%2!
:&=*:,:!%$)(*%4!.$5$.7!
!

9
Cantata - Analysis!
!
!
!
!
!
Using Cantata Analysis
!
'&%(&(&! 8%&.6)*)! 31:0+*)$)! &! )$+*$)! 1/!31:01%$%()!A#*3#!#&5$!9$$%!2$)*4%$2!(1!1//$+-!&915$!&..-!
/.$=*9*.*(6!(1!(#$!,)$+7!?#$6!:&6!9$!,)$2!*%!:&%6!2*//$+$%(!A&6)!&%2-!*/!+$I,*+$2-!9$!$=($%2$2!96!(#$!
,)$+!(1!:$$(!3.*$%(!)0$3*/*3!+$I,*+$:$%()7!
!
'&%(&(&!8%&.6)*)!31:0+*)$)E!
• &!)0$3*&.!'!0+$M0+13$))1+!F789G7!
• &%!S%)(+,:$%($+!0+14+&:-!(1!&%&.6;$!)1,+3$!312$!/*.$)!&%2!*%)$+(!315$+&4$!B0+19$)C!F78:GT!
• &%! &22*(*1%&.! .*9+&+6! 1/! ($)(! 2*+$3(*5$)-! A#*3#! :&6! 9$! *%31+01+&($2! *%(1! &! ($)(! )3+*0(! F"3#! 78;!
+*2(&(6G7!
!
'&%(&(&! 8%&.6)*)! 3&%! 9$! ,)$2! *%! (A1! :&*%! A&6)7! ?#$! /*+)(! 1/! (#$)$! *)! &)! &%! $=($%)*1%! (1! '?@-!
&..1A*%4!(#$!,)$+!(1!/,..6!*%($4+&($!26%&:*3!($)(*%4!A*(#!315$+&4$!&%&.6)*)!&%2!)(&(*3!5$+*/*3&(*1%7!?#*)!
2*&4+&:!*..,)(+&($)!(#$!,)$!1/!'&%(&(&!8%&.6)*)!*%!(#*)!A&6E!!!
!

!!H1,+3$! Cantata W+$M0+13$))$2! Cantata

'12$ W+$M0+13$))1+! H1,+3$! !S%)(+,:$%($+!

H(&(*3!8%&.6)*)! S%)(+,:$%($2!
X*)(!D*.$! H1,+3$!

Cantata
'?8!
'1:0*.$!U!
X*%>
Cantata
'?@!

J6%&:*3!U!
'15$+&4$!
Cantata
R=$3,(&9.$!
V$),.()

!
The Cantata pre-processor and instrumenter are used to produce instrumented source code: that is
source code containing ‘probes’ to facilitate the collection of coverage data. The Cantata
instrumenter also produces a ‘list’ file (an annotated code listing containing the source code and a
static analysis report).

10
Cantata - Analysis!
!

!
!
!
The instrumented source code is compiled and linked with a test script (based upon CTH) and the
CTA library. When run, the resultant executable produces test results which may include both static
analysis and coverage analysis information as well as the normal dynamic test results.

The Cantata approach is unique in that test coverage measurement can be completely
integrated into the dynamic testing process and that it is possible to enforce a required level
of test coverage. (No post processing of trace files is necessary).
!

Stand-Alone Cantata Analysis!

Cantata Analysis can also be used stand-alone. In this mode, developers can use their own test
software (or a third party tool) to exercise the application under test, while generating analysis reports
using Cantata Analysis:
!

Source Cantata Pre-processed Cantata

Code Pre-processor Source Instrumenter

Static Analysis Instrumented

List File Source

Application Compile &

Coverage
Executable Link
Results

Stimulate
Coverage
Trace File

!
In this case, complete coverage reports can by produced either directly by the application’s
executable or a coverage trace file can be output for later analysis.
!

11
Cantata - Analysis

Coverage Analysis Results

!
Cantata Analysis can produce coverage (and static) analysis results in a variety of forms. The
following sections give some idea of the flexibility of the product in this respect.

Coverage Analysis Within a Test Script

When used with CTH, Cantata Analysis is at its most powerful. Users have complete access to all
coverage analysis and static analysis results from within the test script. Checks on metrics can be
made and complete analysis reports generated as part of the dynamic test results. (Static analysis
results can be similarly reported and checked).
For instance, it is possible to verify that decision or statement coverage of any preset level has been
attained, or to give a complete report on any of the coverage metrics. All reports are cross-
referenced to the list file generated by the instrumenter.
This example shows a simple decision coverage report, indicating execution of each decision:
!
DECISION STATISTICS REPORT

DECISION NO LINE NO TYPE NO OF OUTCOMES BREAKDOWN

1 4 if 2 TRUE: 3
FALSE: 0 >> NOT EXECUTED

2 6 switch 4 case #1: 1

case #2: 1
case #3: 0 >> NOT EXECUTED
case #4: 0 >> NOT EXECUTED
imp default: 1 >> WARNING

3 9 ?: 2 TRUE: 1
FALSE: 0 >> NOT EXECUTED

4 12 for 2 TRUE: 8
FALSE: 1

5 18 while 2 TRUE: 0 >> NOT EXECUTED

FALSE: 0 >> NOT EXECUTED

6 28 while 2 TRUE: 0 >> NOT EXECUTED

FALSE: 0 >> NOT EXECUTED
>> WARNING: Switch executed with unknown case value

Total of decision outcomes = 14

Total outcomes exercised at least once = 6

Decision coverage = 42%

>> WARNING: DECISION COVERAGE INCOMPLETE
++++++++++++++++++++++++++++++++ END OF REPORT ++++++++++++++++++++++++++++++++

!
Stand-Alone Coverage Analysis!
When used in stand-alone mode, Cantata coverage analysis can be used to provide coverage
reporting from units, modules or complete application programs which are independently executed,
i.e. not under the control of CTH. The full range of coverage reports are still available in this mode of
operation.

12
 Cantata - Analysis

Static Analysis Results !

H(&(*3!&%&.6)*)!+$),.()!3&%!9$!&33$))$2!*%!&!%,:9$+!1/!A&6)E!
• &)!0&+(!1/!(#$!*%)(+,:$%($+!.*)(!/*.$!
• &)!&!31::&!)$0&+&($2!5&.,$!F<7=>G!/*.$-!A#*3#!3&%!9$!$=01+($2!(1!4+&0#*3&.!2*)0.&6-!)0+$&2)#$$(!
1+!2&(&9&)$!0&3>&4$)!
• /+1:!A*(#*%!(#$!'?@!($)(!)3+*0(-!A#$+$!(#$6!:&6!9$!3#$3>$2!
!
8%!$=&:0.$!1/!$&3#!1/!(#$)$!/1+:&()!*)!)#1A%!1%!(#$!/1..1A*%4!0&4$)7!?#*)!+$01+(!*)!&!)*:0.$!)(&(*3!
&%&.6)*)!*%)(+,:$%($+!.*)(*%47!
Static Analysis Measures for "control.c a03m00"
SOURCE_LINES 557
CODE_LINES 198
COMMENT_LINES 353
BLANK_LINES 45
EXPRESSION_STATEMENTS 53
FOR_LOOP_STATEMENTS 3
WHILE_LOOP_STATEMENTS 0
DO_LOOP_STATEMENTS 0
IF_STATEMENTS 13
SWITCH_STATEMENTS 1
RETURN_STATEMENTS 1
GOTO_STATEMENTS 0
STATEMENTS 74
DECLARATIONS 13
COMMENTS 215
.
.
.

MAXIMUM_NESTING_LEVEL 4
AVERAGE_NESTING_LEVEL 0.58
.

Spreadsheet Export
.
.

MCCABE 19
ESSENTIAL_MCCABE 1

?#$! '&%(&(&! *%)(+,:$%($+! 3&%! 10(*1%&..6!

MYERS_MCCABE_LOWER 19
MYERS_MCCABE_UPPER 21
HANSEN_CYCLOMATIC_NUM 18
HANSEN_OPERATOR_COUNT
HARRISON_SCOPE_RATIO
98
0.45
1,(0,(!)(&(*3!&%&.6)*)!),::&+*$)!*%!<7=>!
HALSTEAD_NUM_UNIQUE_OPERATORS
HALSTEAD_TOTAL_NUM_OPERATORS
21
194 F31::&! )$0&+&($2! 5&.,$G! /1+:&(! /1+!
2*+$3(! *:01+(! *%(1! 4+&0#*3&.! 2*)0.&6-!
HALSTEAD_NUM_UNIQUE_OPERANDS 75
HALSTEAD_TOTAL_NUM_OPERANDS 221
.

)0+$&2)#$$(! &%2! 2&(&9&)$! 0&3>&4$)7! S/!

.
.
CLASSES 0
NEW
DELETE
THROW
0
0
0 61,!A1,.2!.*>$!(1!:&>$!61,+!1A%!3,)(1:!
+$01+()!(#$!/1+:&(!/1+!(#$!7'HY!/*.$!&%2!'!
TRY_CATCH 0
ANONYMOUS_UNIONS 0

)1,+3$! 312$! /1+! 0+13$))*%4! 7'HY! /*.$)! *)!

PARAMETERS 2
UNUSED_PARAMETERS 0
AUTOMATICS 13
STATICS 0
UNUSED_DATA
LOCAL_TYPES
UNUSED_LOCAL_TYPES
0
0
0
*%3.,2$2! *%! (#$! )&:0.$)! 2*+$3(1+67
--------------------------------------------------------------------------------
DATA ANALYSIS
--------------------------------------------------------------------------------
Definitions and Declarations Outside of Any Function
----------------------------------------------------
Name Flags No. of References
---- ----- -----------------
a03m01 function 9
a03m02 function 5
a03m03 function 2
.
.
.
ct02_dirsep_ca extern 1
ct02_hdrftr_pca extern 6
ct02_notmdte_ca extern 0 >> UNUSED
.
.
.
Definitions and Declarations Within Function : a03m00
--------------------------------------------
Name Flags No. of References
---- ----- -----------------
cr_argv_pca parameter 1
cr_numargc_n parameter 0 >> UNUSED
vl_baselen_i 2
vl_error_b 10
vl_error_n 4
vl_exit_z 0 >> UNUSED
vl_fileloop_i 6
vl_format_pca 4

============== Overall Preprocessor Measures For The File =============

TOTAL_MACROS 1272
MACROS_TO_BE_EXPANDED 1270
MACROS_NOT_TO_BE_EXPANDED 2
SUBSTITUTED_MACROS 111
UNSUBSTITUTED_MACROS 0
DIRECT_INCLUDE_FILES 10
INDIRECT_INCLUDE_FILES 1
MAX_INCLUDE_NESTING 2
MAX_CONDITION_COMP_NESTING 3

==================== Overall Measures For The File ====================

FILE_SOURCE_LINES 767
FILE_CODE_LINES 238
FILE_COMMENT_LINES 465
FILE_BLANK_LINES 105
FILE_STATEMENTS 74
FILE_DECLARATIONS 50
FILE_COMMENTS 309
FILE_FUNCTIONS 1
IGNORED_SEQUENCES 0
TEST_CODE 0
FILE_CHECKSUM 846162015
FILE_CLASSES 0
FUNCTION_CLASSES 0
FILE_FRIENDS 0
FUNCTIONS_IN_CLASSES 0

13
Cantata - VisualMetricsTM!
!
!
Developed as an extension to Cantata, VisualMetrics is an easy-to-use tool that provides state-of-
the-art graphical presentations of the metrics data gathered on your software. VisualMetrics
transforms Cantata .CSV files into charts so that you can quickly document or present your
findings. VisualMetrics is a powerful tool that provides sorting, filtering, user definable reports,
over 20 different graphing options, in color or black and white.

?&(!@(&%3$ =.&""#(!A*&0(&,$!
Source/Comment/Blank Lines Lines of Code
40

getid
main
30
lstline Code Lines
nam
prtrefs 20
Functions
myalloc Comment
search Lines
nonres 10
initoutfile
initinfile Blank Lines
newref 0
0 1 2 3 4 5 6 7 8 9 10 11121314 15 1617181920 212223 242526 2728 2930 3132 33343536 3738 39
0 10 20 30 40 50 60

!
!
! BA!@(&%3$!
75$"',*C#D!
!
Q)$! Y*),&.<$(+*3)! (1! 3#11)$! N,)(! (#$! :$(+*3)! 61,! Cyclomatic Complexity > 3
A&%(! (1! )$$7! ! D*.($+! 1,(! 5&.,$)! A*(#*%! (#1)$! :$(+*3)!
(#&(!61,!21!%1(!A&%(!(1!)$$!)1!61,!3&%!;11:!*%!1%! Cyclomatic
0+19.$:!&+$&)7! 25 Complexity
! 20
Myers
15
Upper
Metric Selection Filter Selection 10

5
Hansen
Cyclomatic
0
nonres myalloc name main
search prtrefs lstline getid Harrison
Scope Ratio
Functions for Program AF2T
!
!
!
!
!
!
E&$6F"'F5$#!
VisualMetrics supports all the metrics produced in CantataTMso there are over 100 metrics available
to build custom reports from. VisualMetrics in a Windows application that can process .CSV files
on a PC or over a network. And since VisualMetrics is a Windows application, printing your graphs
is a snap, but even more useful is the cut-and-paste capability to put VisualMetrics graphs directly
into your documents or presentations.

14
Cantata - Analysis

Path Verification and Trace

!
8%1(#$+!/$&(,+$!1/!'&%(&(&!8%&.6)*)!A#*3#!*)!1/!*%($+$)(!(1!#*4#M*%($4+*(6!)1/(A&+$!2$5$.10$+)-!*)!"#$%!
&'()*)+#$),-7! ?#*)! /&3*.*(6! &..1A)! (#$! ,)$+!(1!5$+*/6!(#&(!&!0&+(*3,.&+!0&(#!(#+1,4#!(#$!312$!#&)!9$$%!
/1..1A$2!2,+*%4!26%&:*3!($)(*%47!!
!
'&%(&(&C)! 0&(#! 5$+*/*3&(*1%! /$&(,+$! *%2*3&($)! (#$! 01*%(! &(! A#*3#! (#$! )1/(A&+$! /*+)(! 2$5*&($)! /+1:! *()!
$=0$3($2!9$#&5*1+!&%2-!(#$+$/1+$-!0+15*2$)!&!01A$+/,.!2$9,44*%4!&*27!S(!3&%!&.)1!9$!,)$2!(1!0$+/1+:!
B/$&)*9.$! 0&(#C! 315$+&4$! &%&.6)*)! /1+! (#$! #*4#$)(! .$5$.)! 1/! I,&.*(6! &)),+&%3$-! ),3#! &)! )&/$(6! 3+*(*3&.!
)1/(A&+$!2$5$.10:$%(7!
!
S%! &22*(*1%-! '&%(&(&! 8%&.6)*)! 3&%! &.)1! 0+12,3$! &! 31:0.$($! (+&3$! .*)(*%4! )#1A*%4! $=$3,($2!
)(&($:$%()-!2$3*)*1%)!F&%2!2$3*)*1%!1,(31:$G!&%2!$%(+6!01*%()7

Summary!
!
'&%(&(&!8%&.6)*)!3&%!0+15*2$E!
• &!)(&(*3!&%&.6)*)!+$01+(!1%!!)1,+3$!312$7!
• &!)(&(*3!&%&.6)*)!+$01+(!&)!&!7'HY!/*.$!/1+!*:01+(!*%(1!&%6!)0+$&2)#$$(!0&3>&4$7!
• 315$+&4$!&%2!)(&(*3!&%&.6)*)!+$01+()!&)!0&+(!1/!(#$!26%&:*3!($)(*%4!0+13$))7!
• 5$+*/*3&(*1%!1/!315$+&4$!&%2!)(&(*3!&%&.6)*)!:$(+*3)!&)!0&+(!1/!(#$!26%&:*3!($)(*%4!0+13$))7!
• )(&%2!&.1%$!315$+&4$!&%&.6)*)!+$01+()7!
• &!+$31+2!1/!315$+&4$!&%&.6)*)!(+&3$!2&(&!/1+!01)(M&%&.6)*)!96!'&%(&(&7!
• &!(+&3$!1/!(#$!$=$3,(*1%!1/!(#$!)1/(A&+$!,%2$+!($)(7!
• 5$+*/*3&(*1%!(#&(!(#$!$=0$3($2!0&(#!(#+1,4#!(#$!)1/(A&+$!#&)!9$$%!/1..1A$2!

15
Cantata - Summary!
!
!
!
!
!
Cantata - Benefits!
• @*4#!0+12,3(*5*(6!*%!($)(*%4!
• H*:0.$+!&%2!:1+$!+$.*&9.$!($)(*%4!
• D,..!5*)*9*.*(6!1/!($)(*%4!&3(*5*(*$)!
• V$0$&(&9*.*(6!1/!($)()!
• <&*%(&*%&9*.*(6!1/!($)()!
• J13,:$%($2!0+11/!1/!($)(*%4!
• O%$!(11.!(1!),001+(!&..!($)(*%4!&3(*5*(*$)!
• S%3+$&)$2!31%/*2$%3$!*%!0+$2*3($2!(*:$)3&.$)!&%2!9,24$(!
• S:0+15$2!0+12,3(!I,&.*(6!

Cantata - Quality!
Z,&.*(6! *)! 1/! 0&+(*3,.&+!*:01+(&%3$!*%!)1/(A&+$!5$+*/*3&(*1%!(11.)7!?#$+$!*)!.*((.$!01*%(!*%!($)(*%4!A*(#!&!
(11.! (#&(! #&)! 9$$%! 0+12,3$2! (1! )(&%2&+2)! (#&(! &+$! .1A$+! (#&%! (#$! )1/(A&+$! 9$*%4! ($)($27! D1+! (#*)!
+$&)1%! '&%(&(&! #&)! 9$$%! 0+12,3$2! (1! (#$! #*4#$)(! 0+&3(*3&.! )(&%2&+2)! ,%2$+! (#$! 31%(+1.! 1/! SWXC)!
+*41+1,)! SHO[\\]P?*3>S?! Z,&.*(6! <&%&4$:$%(! H6)($:7! ?#$! 0+12,3(! *)! *()$./! /,..6! ($)($2! &(! (#$! ,%*(-!
(&)>!&%2!)6)($:!.$5$.-!&%2!*)!,)$2!96!SWX!/1+!&..!*%M#1,)$!'!&%2!'LL!)1/(A&+$!2$5$.10:$%(7!
!
!
!
Cantata - Support !
SWX! &%2! Z'H! 0+15*2$! /,..! A&++&%(6P:&*%($%&%3$! )$+5*3$! /1+! '&%(&(&7! ?#*)! 31:0+*)$)! ! ($.$0#1%$!
),001+(!&)!A$..!&)!&!0+12,3(!,02&($!)$+5*3$7!"$!3&%!&.)1!0+15*2$!/,..!(+&*%*%4-!&%2!1%M)*($!&))*)(&%3$!
2,+*%4!(#$!$&+.6!)(&4$)!1/!,)*%4!(#$!0+12,3(7!
!
S/! +$I,*+$2-! A$! 3&%! 0+15*2$! 31%),.(&%36! 1%! (#$! ,)$! 1/! (#$! 0+12,3(! &%2! *()! &210(*1%! A*(#*%! (#$!
/+&:$A1+>! 1/! &! Z,&.*(6! <&%&4$:$%(! H6)($:7! SWX! *)! &! )1/(A&+$! #1,)$! A*(#! :&%6! 31%),.(&%()! A#1!
)0$3*&.*;$!*%!)1/(A&+$!I,&.*(6!*)),$)7!?#$!31:0&%6!#&)!$=0$+*$%3$!1/!2$5$.10*%4!)1/(A&+$!(1!:&%6!
2*5$+)$!)(&%2&+2)-!*%3.,2*%4!#*4#!*%($4+*(6!)(&%2&+2)7!

Cantata - Features Summary

• Dynamic Testing
• Coverage Analysis
• Static Analysis
• Host and Target Testing
• High Productivity

16
Cantata - Your Questions Answered!
!
!
Q: On what platforms and in what environments can Cantata be used?
A: Cantata is currently available for: most Unix systems, Windows (98, NT, ME, 2000),
OS/2, VAX/VMS, and many cross compilation environments. However the product
can be ported to any C or C++ environment on request. Cantata supports the testing
of K&R C, ANSI C and many versions of C++. (However, CTS is not compatible with
K&R C or more complex/modern C++ syntax). Cantata can be used for testing
software with embedded SQL and assembly language inserts.

Q: If you do Cantata++ why does Cantata support C++ too?

A: Cantata was IPL’s original C and C++ testing tool. The rapid development of C++
necessitated the development of a new tool to support the language: Cantata++.
Cantata’s C++ support has not been removed from the product, as it is necessary to
support many existing users of the tool. However, no enhancements will be made to
Cantata’s C++ support after the release of Cantata++.
!
QG! :$! *"! %'$$*2+#! "'! "&H#! &! 7&-"&"&! "#$"! $.(*%"! I3*.3! I&$! '(*0*-&++6! (5-! '-! &! 3'$"!
,&.3*-#J!&-1!(#25*+1!*"!"'!(5-!'-!&!"&(0#"!'(!5-1#(!$*,5+&"*'-K!
AE! '&%(&(&! 3&%! 9$! 01+($2! (1! &%6! $%5*+1%:$%(! A*(#! &! '! 1+! 'LL! 31:0*.$+7! ! S%! &! 3+1))M
2$5$.10:$%(!$%5*+1%:$%(!(#*)!:$&%)!(#&(!($)(!)3+*0()!3&%!9$!+$31:0*.$2!&%2!($)()!+$+,%!
1%!(#$!(&+4$(!A*(#1,(!3#&%4$7!
!
Q:! What if Cantata does not provide the functionality I require?
A:! Cantata is an open and very flexible product. This means that if a facility you require is
not supported by the tool, you can almost certainly build it for yourself. Examples of this
are: special check procedures, hardware or operating system interface routines, output
of results to database packages etc.
!
Q:! We use a specific development environment / CASE design tool. Can Cantata be
integrated with the other tools that we use?
A: Cantata has already been integrated with a number of leading CASE tools and
development environments. Call IPL or QCS for further information on the availability of
such integrations.

Q: I am developing my application to a recognized standard. How can Cantata help

me?
A: Cantata can be used to fulfil the testing requirements of most software development
standards including: ISO9001, the SEI Capability Maturity Model, IEC 1508, Mil Std
498/DOD 2167A, RTCA DO-178B, EN50128, ESA PSS-05, IEC 880, MISRA,
IEC1508 and Def Stan 00-55. See our papers on this subject. We would be happy to
review your development standards and advise on how Cantata may be able to help
you.

17
Cantata - Want To Know More?
!
!
D1+!/,+(#$+!*%/1+:&(*1%!0.$&)$!31%(&3(E!
!
L=!A*$"(*25"'(G! M5&+*"6!73#.H#1!=')"I&(#J!N"1<!
! ! ! ! 9O!?'4!PPQP!
! ! ! ! ?#&/#("'-J!O(#0'-!!RSTTS!
!
! ! ! ! 93'-#G! QTBFPUQFQPVT!
! ! ! ! E,&*+G!! *-)'WX.$+"1<.',!
! ! ! ! Y#2G! ! III<X.$+"1<.',!
!
!
!
Z&-5)&."5(#(G! :9NJ!=')"I&(#!9('15."$!@('5%!
! ! ! ! E/#+#*03!['5$#!
! ! ! ! @('/#!="(##"!
! ! ! ! ?&"3!! ?;V!QN\!!!L]!
!
! ! ! ! 93'-#G! ^UU!V__Q!USQTTT!
! ! ! ! E,&*+G!! *%+W*%+2&"3<.',!
! ! ! ! Y#2G! ! III<*%+2&"3<.',!
!

!
!
!
L%1&"#1G!A#.#,2#(!VRR`!
a*+#G!7&-"&"&!8?M7=<1'.!
!
;++!"(&1#,&(H$!&.H-'I+#10#1<!!
!*$!"3#!(#0*$"#(#1!"(&1#,&(H!')!:9N!:-)'(,&"*'-!9('.#$$*-0!N*,*"#1<!
7&-"&"&
!VRRUFRS!:9N!:-)'(,&"*'-!9('.#$$*-0!N*,*"#1<!L-&5"3'(*C#1!.'%6*-0!&-1!1*$"(*25"*'-!%('3*2*"#1
83*$!1'.5,#-"! !
!

18