Вы находитесь на странице: 1из 65

Mission Success Starts With Safety

NASA Software Assurance


Training

April 8, 2005

Susan J. Sekira
GSFC Software Assurance Lead
Mission Success Starts With Safety

Course Agenda
• Course Objectives
• Overview of Software Assurance
• Acquirer Software Assurance
• Provider Software Assurance
• Software Measurement
• NASA Software Assurance Classification
Assessment
• Supplementary Information
• Summary

2 NASA Software Assurance V2.0


Mission Success Starts With Safety

Course Objectives

3 NASA Software Assurance V2.0


Mission Success Starts With Safety

Software Assurance Course Objectives

• To establish a common framework and knowledge base


among the NASA software community
– Establish roles and responsibilities
– Enable consistency in implementation and performance
– Eliminate misconceptions about software assurance

• To highlight the benefits of software assurance throughout


the development life cycle
• To encourage and strengthen implementation of software
assurance practices

General Audience: Project Managers, Software Managers,


Software Engineers, Safety Engineers, and Systems Engineers

4 NASA Software Assurance V2.0


Mission Success Starts With Safety

Overview of
Software Assurance

5 NASA Software Assurance V2.0


Mission Success Starts With Safety

NASA Software Assurance Challenges


• Performance Based Contracting
– Lack of software assurance
requirements in proposals
– Limited insight into interim products
• Increased code size and complexity MARS
• Resources
– Constrained budgets
– Compressed schedules
– Limited availability of trained personnel
• Increased visibility
– Loss of mission
– Loss of credibility
• Achieving “high quality” software
(and knowing what that means!)

6 NASA Software Assurance V2.0


Mission Success Starts With Safety

Agency Response and Actions…

• Renewed emphasis on process


improvement initiatives (e.g., CMM,
CMMI)
• Improved and accepted tools and
techniques
• Early Software Assurance involvement
during Software Acquisition activities
• Redefinition of NASA Software
Assurance and updates to standard
• Approved NASA Software Engineering
Requirements

NASA Software Assurance Standard was updated in July 2004!

7 NASA Software Assurance V2.0


Mission Success Starts With Safety

What is Software Assurance?


Software Assurance is an umbrella risk identification and
mitigation strategy for safety and mission assurance of all
NASA’s software

SQ IV&V V&V
Software Software
Safety Reliability

SQ = Software Quality
V&V = Verification and Validation
IV&V = Independent Verification and Validation

8 NASA Software Assurance V2.0


Mission Success Starts With Safety

What’s in it for Me?


Software Assurance:
• Strives to improve the quality of the product while employing risk mitigation
techniques
• Focuses on opportunities for early error detection, problem prevention, and
risk identification and mitigation
• Provides project management insight into the software development
processes and products throughout the life cycle
• Reviews and assesses interim products – can’t build quality in at the end!
• Improves the quality of future products/services

The level of Software Assurance needed is commensurate with


the software classification as well as software size, complexity,
criticality, and risk
9 NASA Software Assurance V2.0
Mission Success Starts With Safety

Value of Software Assurance


Relative Cost of Finding, Fixing
• Earlier detection is cheaper to fix Errors
Adapted from Software V&V, Steven Rakitin
with lesser impact to schedules
• Increases chances of delivering full 100
$100+
functionality within cost and …

schedule 60

• Safety issues are identified and $50


50
addressed early as part of overall

Cost to Fix
system safety 40

• Software assessments can be used 30

$20
to determine milestone completion 20

$5
• Metrics provide indicators of the 10 $1
quality and maturity of the 0

software/system Requirements Design Code Test

Phase Error Found & Fixed


Maintenance

10 NASA Software Assurance V2.0


Mission Success Starts With Safety

Who are the Practitioners?


Software Assurance practitioners INCLUDE a wide range of personnel, employed throughout the software development life cycle

Software Managers Safety Engineers

Software Engineers Systems Engineers

Software Quality Personnel IV&V Personnel

Software Assurance personnel aren’t just those Software Quality folks!!

11 NASA Software Assurance V2.0


Mission Success Starts With Safety

The Disciplines of
Software Assurance

12 NASA Software Assurance V2.0


Mission Success Starts With Safety

Software Quality

• Assures that quality is built into the software


through the functions of
– Software quality assurance
– Software quality engineering
– Software quality control

• Ensures conformance of software life cycle processes and


products to requirements, standards, and procedures
• Performs process and product activities throughout the life
cycle to provide objective insight into the maturity and quality
of the software processes and products
• Promotes continuous process improvement

13 NASA Software Assurance V2.0


Mission Success Starts With Safety

Sample Process and Product Activities


Process Product
 All plans (e.g., Configuration  All required plans are
Management Plan, Software developed in accordance with
Management Plan) and procedures specified requirements,
are implemented according to
specified standards and procedures. standards, or procedures
 Engineering peer reviews and  All software requirements are
management reviews are conducted documented and traceable
and action items are tracked to from system requirements to
closure design, code, and test
 Tests are planned, documented, and  Software development records
conducted using approved test are maintained and up-to-date
procedures and tools
 Configuration baselines are
 Project risks are documented and managed, maintained, and
addressed in accordance with the
risk management plan accurate
 Software quality metrics are in
place and used to manage the
software development effort

SQ identifies strengths, weaknesses, and areas for improvement!


14 NASA Software Assurance V2.0
Mission Success Starts With Safety

Software Safety
Software Safety is a systematic approach to
identifying, analyzing, tracking, mitigating and
controlling software hazards and hazardous
functions (data and commands) to ensure safer
software operation within a system.

Software Safety entails…


Ensuring that software Testing of software safety critical
safety requirements are components on actual hardware to
clearly identified, ensure that the safety requirements were
documented, traced and sufficiently implemented and that
controlled throughout the applicable controls are in place to verify
software lifecycle all safety conditions

Analysis of the consistency, Continuous analysis of


completeness, correctness proposed changes on system
and testability of software safety
safety requirements

Software Safety is a function of System Safety!


15 NASA Software Assurance V2.0
Mission Success Starts With Safety

What is Safety-Critical Software?


Does the software meet any of the following criteria:
• Reside in a safety-critical system (as determined by a hazard
analysis) AND at least one of the following:

– Cause or contribute to a hazard


– Provide control or mitigation for hazards
– Control safety-critical functions
– Process safety-critical commands or data
– Detect and report, or take corrective action, if the system reaches a
specific hazardous state
– Mitigate damage if a hazard occurs
– Reside on the same system (processor) as safety-critical software

16 NASA Software Assurance V2.0


Mission Success Starts With Safety

What is Safety-Critical Software? – cont.

Or does the software meet any of the following criteria:


• Process data or analyze trends that lead directly to safety
decisions (e.g., determining when to turn power off to a
wind tunnel to prevent system destruction)
• Provide full or partial verification and validation of safety-
critical systems, including hardware and software
subsystems

If the software is determined to be safety-critical, then the


project must adhere to the NASA-STD-8719.13, NASA
Software Safety Standard

17 NASA Software Assurance V2.0


Mission Success Starts With Safety

Safety-Critical Software Examples


At GSFC, the following software is considered safety-critical:

Example 1: Software on the Hubble Robotic Vehicle (HRV)


that controls the robotic arm for servicing the spacecraft

Example 2: Guidance and navigation software on HRV’s


De-Orbiting Module (DM) used to dock the DM and HST

NOTE: Engineering telemetry data used to provide safety information, but


not required for safety or hazard control, is not safety-critical!!

18 NASA Software Assurance V2.0


Mission Success Starts With Safety

Software Reliability
Software Reliability is concerned with incorporating and
measuring reliability in the products produced
throughout the life cycle

Specifically…
 Systems Engineering defines software requirements as they contribute to system robustness
 Software Engineering develops software containing required redundancy and fault tolerance
AND measures and analyzes software’s ability to withstand errors
 Software Quality assures quality metrics/measures are documented, monitored, analyzed and
tracked (e.g., error density) AND verifies that reliability requirements have been successfully
demonstrated

19 NASA Software Assurance V2.0


Mission Success Starts With Safety

Verification and Validation (V&V)


• Software Verification and Validation
– Ensures that software being developed or maintained
satisfies functional and performance requirements
– Ensures that each phase of the development process
yields the right products
• Every participant in the software life cycle process plays
a role in some aspect of V&V!

V&V activities include, but are not limited to:


– Analysis of system and software requirements
– Engineering peer reviews (e.g., code walkthroughs)
– Test planning and test execution
– Audits/assessments (e.g., baseline management)

20 NASA Software Assurance V2.0


Mission Success Starts With Safety

Independent V&V – IV&V


• IV&V is Verification and Validation performed by an
organization that is technically, managerially, and financially
independent of the development organization
• IV&V focuses on mission critical software, provides
additional reviews and analyses, and provides in-depth
evaluations of life cycle products that have the highest level
of risk

Examples:
– Validation of design to meet system needs/requirements
– Traceability of safety critical requirements
– Code analysis of mission-critical software components
– Design analysis of selected critical algorithms

21 NASA Software Assurance V2.0


Mission Success Starts With Safety

So what do I need to do?


As a Manager, ask the following:
• How much software is predicted to be on the project? How much
will be acquired vs. developed in-house?
• What is the software’s classification (i.e., Class A – G or
Exploratory)?
• What is its criticality? What functions will software control? What
hazard controls and mitigations?
• What metrics are being collected? Can they help me measure the
maturity and quality of the software product?
• Have I carefully planned for software assurance? Are SQ, IV&V,
and Software Safety personnel on-board?
• Are my contracts/MOU’s written to assure the safety and quality of
the software?

22 NASA Software Assurance V2.0


Mission Success Starts With Safety

Getting Started…
The NASA Software Assurance Standard [NASA-STD-8739.8]
specifies requirements for software assurance for use by NASA
projects, programs, and facilities

This standard…
– Provides a software life cycle perspective for the minimum required software
assurance procedures that contribute to quality software
– Provides specific requirements for each discipline of software assurance
– Provides the “Acquirer” and “Provider” requirements for software assurance to
obtain the most cost effective, best quality, and safest products

23 NASA Software Assurance V2.0


Mission Success Starts With Safety

Acquirer vs. Provider


Acquirer
(Usually NASA or an Organization within the Agency)
Specifies the requirements and
accepts the resulting software
products

Provider
(May be a contractor, a university, a separate organization
within NASA, or within the same organization as the Acquirer)
Designs, develops, implements, tests,
operates, and maintains software products

Software Assurance begins with the Acquirer!


24 NASA Software Assurance V2.0
Mission Success Starts With Safety

Acquirer
Software Assurance

25 NASA Software Assurance V2.0


Mission Success Starts With Safety

Acquirer Software Assurance


Getting started…

• Identify a person responsible for software assurance


(e.g., a software assurance manager)
• Secure budget/resources for software assurance EARLY
on (just like any other functional area)
• Participate in the Initiation phase of the program/project
to ensure appropriate oversight/insight into
requirements, including needed deliverables

Notify your Safety and Mission Assurance Organization


to arrange for Software Assurance Expertise

26 NASA Software Assurance V2.0


Mission Success Starts With Safety

Acquirer Software Assurance

Key Activities:
• Performs Software Assurance Classification Assessment to
identify and evaluate the characteristics of software in
determining the software’s classification
• Applies software classification to tailor software assurance
requirements
• Develops SA input to the Request for Proposal (RFP)
• Reviews proposals for compliance to SA requirements
• Establishes and maintains a software assurance plan for the
acquirer
• Ensures consistency between acquirer and provider SA plans
• Ensures SA training for both the acquirer and provider

27 NASA Software Assurance V2.0


Mission Success Starts With Safety

Acquirer Software Assurance

Key activities (cont.)


• Provides surveillance to assure both the acquirer and provider SA
organizations perform to their plans and procedures
• Assures that problem reports, action items, and test anomalies
are documented, addressed, analyzed, and tracked to closure
• Assures that software products are reviewed and any metrics
collected, analyzed, and trended
• Reports at all formal software reviews (e.g., CDR)
• Fosters effective communication with project management, other
project assurance personnel, and the provider

28 NASA Software Assurance V2.0


Mission Success Starts With Safety
Acquirer’s Software Assurance
Deliverables
• SA Input to RFP
• Software Assurance Plan
– Addresses all SA disciplines
– Outlines activities and deliverables

• Software Assurance Status Reports


– Results from software assurance
activities
– Issues/risks/recommendations

• Software Assurance Records


– Reports
– Analyses
– Metrics

• SA Input at Formal Reviews

29 NASA Software Assurance V2.0


Mission Success Starts With Safety

Acquirer’s Role by Phase…


INITIALIZATION,
RETIREMENT PRE-AWARD
• Assure preparation, approval, •Secure SA expert/resources
and execution of retirement plan •Conduct software classification assessment
•Ensure plans for archival or •Specify SA reqts for inclusion in RFP -
disposal of SA records including provider deliverables for SA
•Plan acquirer’s SA activities – prepare
Preliminary SA Plan
MAINTENANCE
•Ensure SA processes in place
•Assure Metrics are transferred to POST-RFP; PRE-AWARD
Maintenance Organization and maintained •Evaluate proposal for SA requirements
•Conduct Pre-award Surveys
•Participate in contract negotiations
ACQUIRER’S
ROLE POST-AWARD; PRE-IMPLEMENTATION
OPERATION
•Review SA plan •Ensure provider SA plan meets requirements
•Ensure SA processes in place •Verify consistency between acquirer and
•Conduct periodic audits of OPS provider SA plans
•Ensure trained SA personnel

ACCEPTANCE CONTRACT
•Conduct Audits prior to delivery IMPLEMENTATION
•Assure Facility readiness •Conduct surveillance
•Assure acceptance •Assure CM
documentation •Assure provider SA
•Capture Lessons Learned •Ensure acquirer performs tasks

30 NASA Software Assurance V2.0


Mission Success Starts With Safety

Provider
Software Assurance

31 NASA Software Assurance V2.0


Mission Success Starts With Safety

Provider Software Assurance

Key Activities:
• Identifies one or more persons to direct and manage a
software assurance program (e.g., a software assurance
manager) – independent from the project
• Develops a software assurance program that includes SQ,
Software Safety, Software Reliability, V&V, and IV&V (when
required)
• Establishes and maintains a working relationship with
project management and the acquirer
• Establishes and maintains a software assurance plan that
conforms to IEEE STD 730-2002
• Conducts periodic reviews, audits, and assessments of the
development processes and products

32 NASA Software Assurance V2.0


Mission Success Starts With Safety

Provider Software Assurance

Key activities (cont.)


• Prepares and maintains software assurance records from software
assurance activities
• Participates in formal and informal reviews
• Prepares software assurance status reports, highlighting
- Assurance accomplishments from reviews, tests, etc.
- Significant problems, their status, solutions, and remedial and preventive
actions
- Trends in software quality metric data
- Plans for upcoming software assurance activities
- Recommendations and lessons learned
• Flows down software assurance requirements to any subcontractors
and assures compliance

33 NASA Software Assurance V2.0


Mission Success Starts With Safety

Provider Deliverables
• Sample Software Deliverables
– Software Management Plan
– Configuration Management Plan
– Software Requirements Specification
– Requirements Traceability Matrix
– Test plans and procedures
– User’s Guides
– ** Etc.
• Software Assurance Deliverables
– Software Assurance Plan
– Software Assurance Status Reports
– Software Assurance Records

** See NPR 7150.2 for more on software products and required content
34 NASA Software Assurance V2.0
Mission Success Starts With Safety

Provider’s Role by Phase…

RETIREMENT INITIALIZATION,
• Send final SA records to project PRE-AWARD
•Draft SA approach
•Respond to SA requirements in RFP

MAINTENANCE
•Ensure SA processes in place POST-RFP; PRE-AWARD
•Address SA questions from acquirer
•Conduct periodic audits or assessments
•Participate in contract negotiations for SA

PROVIDER’S
ROLE
OPERATION POST-AWARD; PRE-IMPLEMENTATION
•Develop New/updated SA plan •Develop SA plan
•Ensure SA processes in place CONTRACT •Review Acquirer SA plan
•Conduct periodic audits of OPS •Ensure trained SA personnel
IMPLEMENTATION
•Establish communication with

acquirer, project team
ACCEPTANCE •Maintain SA plan
•Conduct audits prior to delivery •Conduct /participate in reviews,
•Assure acceptance documentation inspections, audits and prepare SA
•Prepare Lessons learned reports
•Collect, utilize metrics to identify
trends, assess quality
•Deliver SA reports, maintain records

35 NASA Software Assurance V2.0


Mission Success Starts With Safety

Communication is Essential!
Acquirer SA Provider SA
• Verify Provider SA Plan is
• Develop Provider SA Plan per
complete and compatible with Software Assurance Plan
Acquirer requirements
Acquirer SA Plan
• Review Acquirer plans and
• Review Provider plans and
procedures Plans and Procedures procedures
• Review Provider SA reports and • Flow down requirements to any
records from surveillance subcontractors
Software Products and
activities Deliverables • Prepare and maintain SA
• Evaluate software products and records (accessible by
deliverables throughout life Acquirer)
cycle Software Assurance Records
• Prepare SA Status reports for
• Prepare SA Status reports for Project and Acquirer
Project Software Assurance Status Reports • Establish and maintain working
• Establish and maintain working relationship with Project,
relationship with Project, Meetings, Telecons, and software assurance personnel,
software assurance personnel, and Project Reviews and the Acquirer
and the Provider
• Share issues/concerns/risks

36 NASA Software Assurance V2.0


Mission Success Starts With Safety

Software Measurement

“…when you can measure what you are speaking about, and express
it in numbers, you know something about it; when you cannot
express it in numbers, your knowledge is of a meager and
unsatisfactory kind…”

Lord Kelvin, 1889

37 NASA Software Assurance V2.0


Mission Success Starts With Safety

Why Measure?
Measurements
– provide meaningful information to enable informed control decisions
– provide data from which to model and predict future software trends

 Are we there yet?


How much time and effort required to detect
remaining errors?
To what extent is product error free?
 What will it take to get there?
 Why did/didn’t we get there?
Post mortem analysis – lessons learned

38 NASA Software Assurance V2.0


Mission Success Starts With Safety

Sample Metrics

Start simple with only a few metrics…


• Requirements
– Ambiguity = weak phrases
– Completeness = TBD + TBA + TBR
– Volatility = excessive requirement changes

• Design & Implementation


– Structure/Architecture = complexity & size
• Verification (e.g., formal reviews, peer reviews, testing)
– Defect density (defects per 1000 lines of code)
– Problem report tracking = open, closed, severity

39 NASA Software Assurance V2.0


Mission Success Starts With Safety

Requirements Metrics
Completeness & Volatility Analysis

Total Number of Requirements Modifications to Requirements

1000 450

900 400 New


800 350 Modified
700 Deleted
300
600
Quantity

Quantity
250
500
200
400
150
300
100
200

100
50

0 0
1Q94 2Q94 3Q94 4Q94 1Q95 2Q95 3Q95 1Q94 2Q94 3Q94 4Q94 1Q95 2Q95 3Q95

Calendar Quarter Calendar Quarter


CRR CRR
Looks Good! Excessive
(Stable) Changes!
NOT Stable
Combination of BOTH views indicates risk area - requirements are NOT YET stable

40 NASA Software Assurance V2.0


Mission Success Starts With Safety

Defect Trending
Cumulative Problem Reports Problem --> Opening Errors
Submitted & Closed Continues sharp climb
3500

Cum. Submitted = 3140


3000 Cum. Closed = 2043
Expected
Remaining Open = 1097

2500

2000
No. of PR's

1500

1000

500

0
4/26/96

5/26/96

7/25/96

2/20/97

4/21/97

8/19/97

1/16/98
8/24/96

9/23/96

1/21/97

3/22/97

5/21/97

6/20/97

7/20/97

9/18/97

3/17/98

4/16/98
6/25/96

2/15/98

5/16/98
10/23/96

10/18/97

11/17/97
11/22/96

12/22/96

12/17/97

41 NASA Software Assurance V2.0


Mission Success Starts With Safety

Hints for Successful Measurement


• Define your measures of success early in your project and
track your progress toward them
• Collect only what you need, and use what you collect
• Use more than one metric to interpret the entire picture
• Use defect data trends to help you decide when to release a
product
• Measure complexity to help you optimize design decisions
and create a more maintainable product
• Ensure reasonable data collection, interpretation of data
• Categorize defects to identify product and process
weaknesses

42 NASA Software Assurance V2.0


Mission Success Starts With Safety

NASA Software Assurance


Classification Assessment

43 NASA Software Assurance V2.0


Mission Success Starts With Safety

SA Classification Assessment Process

• The Software Assurance Classification Assessment was


developed by NASA to identify and evaluate the
characteristics of software to determine the software’s
classification and level of software assurance to be applied
• This assessment is conducted for all NASA software
• The process entails 4 basic steps:
1. Perform the Software Safety Litmus Test to determine if the software
has safety implications for the system, property, humans, or the
environment
2. Determine the software engineering class of software (Class A-H)
3. Determine the software criticality score based on
project/mission/software characteristics
4. Classify the software and software assurance effort based on the
results from Steps 1-3

44 NASA Software Assurance V2.0


Mission Success Starts With Safety

SA Classification Assessment (cont.)


• After each step of the process, record the results in a Software
Assurance Classification Report and maintain as a quality
record
• Provide report to the SMA Director and Systems Management
Office (SMO)
Report contents include:
– Project name and date
– Names of software assurance manager and
project manager
– Result of software safety litmus test
– Software Class (A-H)
– Software Criticality score
– Software Assurance Effort/Priority

45 NASA Software Assurance V2.0


Mission Success Starts With Safety

Software Assurance Classes

Class A Human Rated Software Systems


Class B Non-Human Space Rated Software Systems
Class C Mission Support Software
Class D Analysis and Distribution Software
Class E Development Support Software
Class F General Purpose Computing Software (Multi-
Center or Multi-Program/Project)
Class G General Purpose Computing Software (Singe
Center or Project)
Class H General Purpose Desktop Software

46 NASA Software Assurance V2.0


Mission Success Starts With Safety

Tailoring Guidelines for SA Requirements

• The level of Software Assurance needed is commensurate


with the software classification, criticality, unique software
characteristics (e.g., software size, complexity), and
perceived risk
• Based on the NASA SA Standard and NPR 7150.2, tailoring
guidelines are being developed to assist assurance
practitioners in defining the software assurance
requirements required to meet specific
project/mission/software objectives
• The tailoring guidelines will be provided as part of the NASA
Software Assurance Guidebook (currently under revision)

47 NASA Software Assurance V2.0


Mission Success Starts With Safety

Supplementary
Information

48 NASA Software Assurance V2.0


Mission Success Starts With Safety

GSFC Contacts

Software Working Group (SWG) Representatives


– Sally Godfrey 301 286-5706
– Susan Sekira 301 286-6160

Center Software Assurance Lead (Code 303)


– Susan Sekira 301 286-6160
Software Assurance Technology Center (Code 304)
– Al Gallo, Manager 301 286-3756
Systems Safety and Reliability Office (Code 302)
– Karen Fisher, Chief 301 286-7123
Flight Software Branch (Code 582)
– Elaine Shell, Head 301 286-2628

49 NASA Software Assurance V2.0


Mission Success Starts With Safety

Recommended Web Sites

• NASA Software Assurance


http://software.nasa.gov/
• NASA Software Working Group
http://nasa-software-pbma-kms.intranets.com/
• NASA Process Based Mission Assurance (PBMA)
http://pbma.hq.nasa.gov/pbmamaster.html
• NASA Technical Standards
http://standards.nasa.gov/
• Carnegie Mellon- Software Engineering Institute
http://www.sei.cmu.edu/
• Software Engineering Laboratory
http://sel.gsfc.nasa.gov/

50 NASA Software Assurance V2.0


Mission Success Starts With Safety

More Web Sites

• Software Assurance Technology Center


http://satc.gsfc.nasa.gov/
• NASA Independent Verification and Validation
http://www.ivv.nasa.gov
• GSFC Engineering Process Group
http://software.gsfc.nasa.gov
• GSFC Software Assurance
http://sw-assurance.gsfc.nasa.gov

51 NASA Software Assurance V2.0


Mission Success Starts With Safety

GSFC SA Groups and Initiatives


• Engineering Process Group (EPG)
– Center-wide team that establishes and continuously improves
system and software processes and products
• Software Assurance Technology Center (SATC)
– Center of Excellence that provides project support, outreach,
and research in software metrics and software assurance tools
and techniques
• Quick-Look Project Team
– Team of software experts organized to take a quick look at
potential (or impending) software problem areas for a specific
project

52 NASA Software Assurance V2.0


Mission Success Starts With Safety

GSFC Procedures and Products


• GPG 8700.5, In-House Development and Maintenance of Software Products
• 580-PG-8730.3.1, ISC Product Development Handbook
• GPG 8700.6, Engineering Peer Reviews
• 300-PG-7120.2.2, Mission Assurance Guidelines Tailoring
• 303-PG-7120.2.1, Developing and Implementing Software Quality Programs

Also available work instructions, checklists and templates for:


Software Quality (SQ) process and product assessments
SQ Checklists for evaluating processes and products
FSW Product Plans and Test Plan Templates
FSW Coding Standards for Ada and C
FSW CCB Process and Policy
FSW Development Life Cycle
ARM tool, code analysis and metrics, reliability tools (via SATC)

53 NASA Software Assurance V2.0


Mission Success Starts With Safety

Additional Training Opportunities

• SOLAR Training https://solar.msfc.nasa


. gov/
– Software Assurance
– Software System Safety

• NASA Engineering Training Program


(NET) http://net.larc.nasa.gov/
– Introduction to CMMI
– Intermediate CMMI
– Defining World-Class Processes

54 NASA Software Assurance V2.0


Mission Success Starts With Safety

Recommended Reference Documents

Standard Description
NASA STD 8739.8 Software Assurance Standard
NASA GB A201 Software Assurance Guidebook
NASA STD 8719.13 Software Safety Standard
NASA GB 8719.13 Software Safety Guidebook
NPD 2820.1 NASA Software Policies
NPR 7150.2 NASA Software Engineering Requirements
ISO/IEC 12207:1995 Software Life Cycle Processes
IEEE-STD-730-2002 IEEE Standard for SQA Plans
IEEE-STD- 982.1-1998 IEEE Standard Dictionary of Measures to Product Reliability
Software
SEI-CMM Software Engineering Institute Capability Maturity Model
SEI-CMMI Software Engineering Institute Capability Maturity Model
Integration

55 NASA Software Assurance V2.0


Mission Success Starts With Safety

Additional References
• Conte, Dunsmore, Shen,
Software Metrics Engineering Metrics and Models,
Benjamin/Cummings, 1986
• Leveson, Nancy G.,
SAFEWARE System Safety and Computers,
Addison-Wesley Publishing Company, Inc., 1995
• Lyu, Michael R.,
Handbook of Reliability Engineering,
IEEE Computer Society Press, McGraw Hill, 1995
• Shulmeyer, Gordon G., and James I. McManus,
The Handbook of Software Quality Assurance,
Prentice Hall, 1999
• Kan, Stephen H.,
Metrics and models in Software Quality Engineering,
Addison Wesley, 2002

56 NASA Software Assurance V2.0


Mission Success Starts With Safety

Summary

57 NASA Software Assurance V2.0


Mission Success Starts With Safety

Benefits of Software Assurance

• Provides insight into the software


development processes and products
throughout the life cycle
• Focuses on opportunities for
– Early error detection
– Problem prevention
– Risk identification and mitigation

• Increases chances of delivering full


functionality within cost and schedule
• Enables improvement of future software
products and services!!

58 NASA Software Assurance V2.0


Mission Success Starts With Safety

Example SA Activities

• Software Assurance Classification Process


• Proposal and contract evaluations
• Requirements traceability
• Design and code analyses
• Engineering peer reviews
• Test planning and test execution
• Software measurement and trends
• Audits and assessments

Team work and communication!!

59 NASA Software Assurance V2.0


Mission Success Starts With Safety

Keys to a Successful SA Program


• Identify a person responsible for directing and managing the
Software Assurance Program
• Define software assurance requirements early in the life cycle
• Develop a Software Assurance Plan
• Monitor processes throughout the system and software
development life cycle
• Evaluate software and deliverables to assure that quality and
safety are being built into the products
• Ensure compliance with established standards and procedures
• Establish metrics to help measure quality

60 NASA Software Assurance V2.0


Mission Success Starts With Safety

More Keys…
• Assure that problems and risks are documented,
reported, addressed, and tracked to closure
• Prepare and maintain software assurance records
and status reports
• Capture Lessons Learned to improve the quality of
future products/services!

61 NASA Software Assurance V2.0


Mission Success Starts With Safety

Backup Slides

62 NASA Software Assurance V2.0


Mission Success Starts With Safety

V&V Activity Matrix


Peer Formal Analyses Audits/ Similarity SWTests Demonstration Acceptance
Review/ Reviews (V&V) Assessments (Verify) (Verify) (Verify) (Validate)
Inspection (V&V) (V&V)
(V&V)
SQA/SQE Concur, Assure, Assure, Conduct, Assure, Assure, Assure, Concur, Conduct,
Assure, Participate Participate, Sponsor, Concur Participate, Participate Participate,
Participate Conduct Assure, Concur Approve
Participate

IV&V* Participate Participate Assure, Assure, N/A Assure Assure, Assure


Participate, Conduct, Participate
Conduct, Provide,
Concur Participate

SWEng/Mgt. Approve, Conduct, Conduct, Participate, Conduct Conduct Conduct Conduct,


Conduct, Participate Concur, Concur Participate
Participate Approve,
Participate
SystemEng. Participate Participate Participate, Participate, Participate, Participate Participate Participate,
Concur Concur Concur Conduct
Project Sponsor Approve, Sponsor Approve Approve Approve, Approve, Approve
Mgmt Sponsor, Sponsor Sponsor
Participate
Safety Participate Participate Participate, Participate, Concur, Participate, Participate, Approve,
Concur, Conduct, Assure Concur, Concur Concur
Conduct, Sponsor, Approve
Sponsor, Provide
Provide

Customer Participate Participate N/A N/A Approve Participate Participate Approve


SCM** Support Support Support Support Support Support Support Support

Concur : Assure: Participate: Conduct : To Provide: Support: Sponsor : To Approve :


Agree make certain be a lead and Supply Maintain the assume Formally
the contributing manage the product baseline responsibility for consent to a
prescribed member of prescribed and/or configuration prescribed prescribed
activity the activity to service to for these activity activity or
occurs prescribed completion the prescribed sign-off on
according to activities, in prescribed activities successful
established accordance activities completion
plan, policies, with a
procedures, defined role,
Stds responsibility
.
*IV&V will only apply to some projects depending on risk assessment classification, mission criticality, or project category.
** At some centers or on some projects, SCMmay play a more of a supportive role rather than that of a direct participant in V&V
activities.

63 NASA Software Assurance V2.0


Mission Success Starts With Safety

Fundamental Differences
SQ IV&V
• Provides Center-level services • Provides Agency-level services
• Focuses on ALL Project • Focuses on MISSION CRITICAL
software Project software
• Emphasizes compliance to • Emphasizes completeness and
standards and procedures correctness of the product
• Reviews, monitors and audits all • Reviews, analyzes, and provides
Project processes and products in-depth evaluations of life cycle
products which have the highest
for completeness and accuracy
risk
• Matrixed to the Project as part • Independent from the Project and
of the Project Team and provides analyses and
provides daily insight/oversight evaluations per IV&V priorities
• Reports to Project and Center • Reports to Project, GPMC’s, and
Director through S&MA NASA Headquarters

64 NASA Software Assurance V2.0


Mission Success Starts With Safety

Frequently Asked Questions

• How do we levy software assurance requirements on an


existing software development activity?
• How do these requirements apply to a small development
effort?
• Are these requirements applicable to in-house work?
• Who determines the software classification for my
software?
• Do I need to document the classification?

65 NASA Software Assurance V2.0