Briefing to ICOTE

Value of Testing and Evaluation

Todd E. Stevenson Chief, Plans, Analysis and Evaluation Office HQ, Developmental Test Command 20 Nov 2001

Developmental Test Command

Purpose

Provide status of the efforts to address the Defense Science Board Report in regard to the Value of Testing and Evaluation (T&E) and Ask for your assistance

Developmental Test Command

Background

DSB Report, Test and Evaluation Capabilities (Oct 00) No measure of T&E available to determine the return on investment of the T&E process. Recommendations: DOT&E should collaborate with USD(AT&L) to develop methodology to determine value of testing. DOT&E, with Services, should seek a set of common metrics and objectives to measure Service and DoD performance of T&E.

Developmental Test Command

Actions To Date
Surveyed published reports Collected anecdotal examples Initiated / hosted three meetings 14 Jun 01 attendees: DOT&E, JPO (T&E), ATEC HQ, AEC, AMSAA, and DTC 27 Jun 01, above organizations plus Dr. Seglie 10 Oct 01 working group meeting Developing / Refining Metrics

Developmental Test Command

Metrics Under Consideration

Reliability Growth Program and Cost Savings or Avoidance Return on Investment Percent of RDT&E Budget Spent on T&E Number of SCRs/ECPs/TIRs over time
SCR = Software Change Request ECP = Engineering Change Proposal TIR = Test Incident Report

Good T&E Programs versus Bad Programs Cost Savings or Avoidance on Environmental Testing

Developmental Test Command

Metric #1

Reliability Growth Program and Cost Savings or Avoidance

SCOPE: Identify reliability problems surfaced in testing Assess effectiveness of implemented re-designs Calculate savings / cost avoidance ISSUES: Labor and resource intensive AEC-N identified initial candidate systems Reliability improvement quantified for each system Failure rate for each configuration & failure mode identified Cost data identified for each system Initial cost avoidance analysis at system level Crosswalking failed LRUs/components from Reliability Analysis to log data difficult to do (still working) Using Operating & Support Management Information System Using Central Demand Data Base for LRU/component crosswalk
6

Developmental Test Command

Example: XYZ Truck Subsystem

Test History (1992-1993) PQT of XYZ Truck - 120,000 miles on 6 Trucks @ APG & YPG - 58 Failures Final Assessment - Based on fix effectiveness factors verified in 2-month IOTE & 24,000 mile Truck & Trailer Final Assessment Validation Test (FAVT). Reliability Results Requirement 2250 MMBHMF PQT Test 2068 Final Assessment 3733

Reliability Improvement Ratio = 3733/2068 = 1.8

MMBHMF - Mean Miles Between Hardware Mission Failure
7

Developmental Test Command

Example: XYZ Truck Subsystem

Improvement factor = 3733/2068 = 1.8 Two year O&S cost (1999-2000) for XYZ Truck from OSMIS (consumables and net repairables) = $5,457,118 XYZ truck average density per year (1999-2000) = 1,555 Average cost per vehicle per Year = $1,756 Potential cost if no testing had occurred = $5,457,118 x 1.8 = $9.8M Potential cost avoidance due to testing = $4.4M
Caveats: Replacements due to Ordering Wrong Part, Maintenance Induced Failures (MIFs) and NEOFs included in Net Cost. Test & Evaluation costs not factored into analysis BUT, Potential for more MIFs & NEOFs with Lower Reliability Savings only applied for two years - true savings likely longer
8

Developmental Test Command

Metric #2

Return on Investment
SCOPE: Consider testing as an investment Net cost benefit / the cost of the test ISSUES: Labor and resource intensive Candidate Systems: Joint Services Lightweight Integrated Suit Technologies and Multiple Integrated Laser Engagement System 2000 Working with PMs to provide cost savings data. Data Required: Problem identification Fix verification and cost of fix Validation of Life Cycle Cost Reduction
9

Developmental Test Command

Example: Return on Investment

MILES 2000 test findings which may imply significant life-cycle cost savings: - Universal Laser Transmitter alignment problem - EMI problem caused false kills - CVS M2 Bradley ammo run-out problem. - CVS M2 TOW tubes

Return on Investment =

Net Life Cycle Cost Reduction ----------------------------------------Cost of Test

10

Developmental Test Command

Metric #3

Percent of RDT&E Budget / TimePhasing of Funding Spent on T&E

SCOPE: This is a template approach. For a successful program, consider the percent of program dollars spent on T&E, and the timing of T&E expenditures. Candidate Systems: Comanche M1 Family of Medium Tactical Vehicles Objective Individual Combat Weapon Engineering Development (OICW) Brilliant Anti-Armor (BAT) Submunition Data Required: Research, Development, and Test and Evaluation Budget Tests funded at DTC and Contractor
11

Developmental Test Command

Percent of RDT&E Budget / TimePhasing of Funding Spent on T&E

Percent Spent on T&E

Program T&E Dollars = ---------------------------------------------Program RDT&E Budget

Data Source: Army RDT&E Budget Item Justification (R-2 Exhibit) - Only using data for one Program Element (65) - Procurement dollars not included Questions: Does a higher percent spent earlier lead to a better product? Is success more highly correlated to earlier or larger percent?

12

Developmental Test Command

Metric #4

Number of SCRs/ECPs/TIRs over Time

SCOPE: For each program, track the number over time and then over test hours Assumption: TIRs, SCRs and ECPs lead to system re-designs, which lead to improvements. Candidate Systems: JSLIST Protective Services Concealable Body Armor Modular Lightweight Load Carrying Equipment (MOLLE) Data Required: Test Incident Reports (TIRs) Software Change Requests Engineering Change Proposals Questions: Is there a correlation between system change orders and TIRs? Is there a correlation between TIRs and test hours? Is there an effect resulting from open vs. closed TIRs?
13

Developmental Test Command

EXAMPLE: MODULAR LIGHTWEIGHT LOAD CARRYING EQUIPMENT (MOLLE)

180 160 140 120

TIRS

100 80 60 40 20 0 73.8 216.4 293.1 589.9 607 751 Hours

TIRS

14

Developmental Test Command

EXAMPLE: MODULAR LIGHTWEIGHT LOAD CARRYING EQUIPMENT (MOLLE)

50 45 40 35 30 25 20 15 10 5 0 Jul- Aug- Sep- Oct- Nov- Dec- Jan- Feb- Mar- Apr- May00 00 00 00 00 00 01 01 01 01 01 Minor Major
15

Developmental Test Command

Metric #5

Good T&E Programs versus Bad T&E Programs

SCOPE: Successful versus unsuccessful programs Adequate versus inadequate T&E Questions: Can we define success? Can we define adequate T&E? Are adequate T&E and success correlated?

16

Developmental Test Command

Metric #6

Cost Savings / Avoidance on Environmental Testing

SCOPE: Look at new equipment fieldings and their relationship to the amount of testing. Attempt to go beyond anecdotal evidence, and expand eventually to the value of simulation in T&E. Data: Quality Deficiency Reports, HIVE Database or PM databases Analysis: Identify problems Relate problems to environments Relate problem/environment pairs to life-cycle cost savings Cost Avoidance = Cost if not fixed, minus cost of the fix Assumption: We can determine valid life-cycle cost savings.

17

Developmental Test Command

Whats Next

Look at Expanding Metrics to ACAT I and Major Oversight Systems Next Working Group meeting in Feb 02 to review additional research and data collection More interaction needed with private industry developers. Value of T&E Workshop July 16 19, 2002, Annapolis

18

Developmental Test Command

Last Slide Looking for Help!

Cisco Systems Built a $30M testing lab to prototype customer networks. Question: How did they build a business case to support such a large outlay? Lucent Design, test and give away network solutions for customers Question: What methodology did they use to predict the outcome of this business practice? How do they assess an economically advantageous level of involvement? Microsoft T&E of Windows 2000 integrated into product development process Questions: Do they have activity based costing? Can they tell us the amount of development budget consumed in the T&E activity? How much T&E do they build into the business plan for a product development effort? Do they ever go back at the end of development and assess whether it was too much, too little, or about right?

19

Developmental Test Command