Conformance

Mark Skall
Lynne S. Rosenthal
National Institute of Standards and Technology
mark.skall@nist.gov
lynne.rosenthal@nist.gov
Introduction
 NIST works with industry to develop
standards and tests to improve the quality of
software and achieve interoperable solutions
 Many years experience with
– Formal standards organizations and Consortia
– Developing conformance test suites, tools,
reference implementations
– Developing validation and certification testing
programs
Today’s Objectives
 Present basic information about conformance
and conformance related topics
 Achieve a common understanding
 Lead a discussion about how conformance
applies to ebXML
 Present overview of TA Spec. Conformance
Clause
 Offer our assistance to WG in discussing and
developing conformance for their Spec.
 Conformance
Outline clause
Conformance Testing

Test suites Test tools

 Background Validation
Certification
Certificate Branding
 Terminology s

 Conformance Testing Procedures, reporting

– Testing Methodology
– Testing Ingredients
 ebXML Issues
 TA Conformance Guidance
 Background
 Standards not enough to ensure interoperability
– Standards only meaningful if implemented in a consistent way
 Need to ensure that implementations adhere to the
standard
– What is expected of implementations in order to claim
conformance – i.e., what are the requirements?
– How will we know if an implementation conforms?
• test suites, test tools
 Different ideas of what conformance is
– Past experience may have affected view of conformance
Terminology

Conformance

Specification Conformance Clause

Conformance Testing test suite, test tools

Validation process, reporting

Certification certificate, brand

Terminology - Conformance
 CONFORMANCE - the fulfillment of a product,
process or service of specified requirements
(ISO Guide 2)
– These requirements are specified in a standard or
specification as part of a conformance clause or in
the body of the specification
 CONFORMANCE CLAUSE - a section of a
specification that states all the requirements or
criteria that must be satisfied to claim
conformance
Terminology - Conformance Testing
 CONFORMANCE TESTING – a way to verify
implementations of a specification to determine
whether or not deviations from the specifications exist
(through the use of test suites).
– tests capture the technical requirements within the specification
– testing is bound in scope by the specification
– internal structure of product-under-test not accessible (black box testing)
 Necessary, but not sufficient, for interoperability

Note: referred to as conformity assessment in ISO Guide 2

Testing Methodology

 Falsification Testing
– find errors by means of experimentation
– show presence of errors not their absence
– prove non-conformance; can never prove conformance
 Tests built to check for:
– required functionality has been implemented (basic tests)
– logical errors by misunderstanding requirements
– errors from boundary conditions and divergence
– common programming errors
 Specification
Requirements

Implementation
Conformance Tests
Test programs

100% ????
non conforming conforming
Terminology - Testing Process
 VALIDATION - process necessary to perform
conformance testing in accordance with a prescribed
procedure and official test suite
– ensures that testing can be repeatable and reproducible
– ensures that conclusions are consistent with facts presented
in the evaluation

 CERTIFICATION - acknowledgement that a

validation was completed and the criteria established
by CIO for issuing certificates (brands) was met.
Testing Ingredients
Certification
qualified bodies to do the testing and certification
control board - advisory and arbiter

Validation
Process - policy and procedures for testing

Conformance Testing
Test assertions
Test suite
(test software, test scripts, test criteria)

Standard
Conformance clause
ebXML Conformance Issues
 Conformance Clause should address?
1. What Needs to Conform?
2. Profiles and Levels
3. Extensions
4. Implementation Defined Features
5. Alternate approaches
 Scope –what should be addressed?
a. Assertions
b. Test Suite
c. Testing Service
d. Certification
ebXML Issues -
Conformance Clause address:
1. What needs to conform
– identify the ‘class of products’ that will be
developed
– what are the conditions that need to be met in
order to claim conformance
– Examples:
registry, messaging service, business service
interface, ebXML message, testing partner agreements
 ebXML Issues -
Conformance Clause address:
2. Profiles and Levels
– Are profiles and/or levels needed?
– Should a profile have its own levels?
– minimal requirements (core)

Profile is a subset of the overall specifications that includes all of the functionality
necessary to satisfy the requirements of a particular community of users.
 Levels are nested subsets of the specifications. Level 1 = core that all must implement;
subsequent levels include lower level + added functionality
 ebXML Issues -
Conformance Clause address:
3. Extensions
– If allowed:
• shall not re-define semantics of existing functions
• shall not cause standard-conforming functions to execute incorrectly
– Standardized method for extensions?
• Define the mechanism for extending the specification, so extended in
a standard manner
– Register of extensions
• document parallel to but distinct from specification
• list of recognized, published extensions
• may eventually migrate into future versions
ebXML Issues -
Conformance Clause address:
4. Implementation Defined Features
– may want to address implementation dependent
ranges, min/max allowed values
5. Alternative Approaches
– specification may describe several different ways
to accomplish an operation (e.g., choice of
bindings)
– specify whether or not all approaches need to be
implemented
ebXML Issues - Scope
Should ebXML include or address:
a. Assertions (test requirements)
- Should assertions be part of the standard?
b. Test Suite
- Prescribe a test methodology?
- How test suite will be delivered/used (e.g., web based,
downloadable)?
c. Testing Service
- Guidelines for establishing a test service
d. Certification
- Guidelines for issuing certificates
Testing Program - Considerations
 How formal should testing be?
– Self testing, 3rd party testing
– Branding/certificates
 Testing program
– Who will ‘own’ the testing program
– Policy and procedures
– Testing laboratory
– Issue Certificates
– Control board
– Test suite maintenance
TA Conformance Guidance
 General conformance guidelines for ebXML
 All ebXML specs shall contain a conformance
clause
– Can have levels of ebXML conformance -
hierarchical
– Explains how conformance requirements can be
expressed (mandatory, conditional, optional)
 Encourage use of publicly available test
suites
Discussion