Assertion-Based Verification

Introduction to ABV
Harry Foster
Chief Scientist Verification

info@verificationacademy.com | www.verificationacademy.com
Session Overview

After completing this session you will. . .

Understand todays productivity

challenges

Understand how to improve

verification productivity with ABV

2014 Mentor Graphics Corporation, all rights reserved.

Outline

Effort Spent in Verification Today

Observability and Controllability Challenge

Assertion-Based Verification

Industry Case Studies

Conclusions

2014 Mentor Graphics Corporation, all rights reserved.

 More and More Verification Engineers
Mean peak number of design vs. verification engineers ~ 1-to-1 ratio
of peak design
and verification
58% 11%
engineers

8.4 Verification
7.6
4.8 Engineers
4% 5%

7.8 8.1 8.5 Design

Engineers

2007 2010 2012

Source: Wilson Research Group and Mentor Graphics, 2012 Functional Verification Study

2014 Mentor Graphics Corporation, all rights reserved.

 Designers Doing More and More Verification

Design Engineer Project Time

2007 - 2012
54%
53%

47%
46%

2007 2012 2007 2012

Doing Design Doing Verification
Source: Wilson Research Group and Mentor Graphics, 2012 Functional Verification Study

2014 Mentor Graphics Corporation, all rights reserved.

 Where Verification Engineers Spend Their Time

More time spent in debug

than any other task!
36%
4%
Test Planning
Testbench Development
16% Creating and Running Test
23% Debug
Other
22%

Source: Wilson Research Group and Mentor Graphics, 2012 Functional Verification Study

2014 Mentor Graphics Corporation, all rights reserved.

Outline

Effort Spent in Verification Today

Observability and Controllability Challenge

Assertion-Based Verification

Industry Case Studies

Conclusions

2014 Mentor Graphics Corporation, all rights reserved.

 Fundamental Challenge of Verification

DUT
1. Activate

Stimulus

2014 Mentor Graphics Corporation, all rights reserved.

 Fundamental Challenge of Verification

DUT
1. Activate

Stimulus 2. Propagate

2014 Mentor Graphics Corporation, all rights reserved.

 Fundamental Challenge of Verification

A
DUT
0111010100111010101010000000001110101101101111011
1. Activate
A
A 3. Detect
Stimulus A 2. Propagate

Checkers

A = Assertions

2014 Mentor Graphics Corporation, all rights reserved.

Observability vs. Controllability

bug

0
1 0

1
0

Test didnt set up the condition to propagate the bug

2014 Mentor Graphics Corporation, all rights reserved.
Observability vs. Controllability

bug

A 0
1 0

1
0

Test didnt set up the condition to propagate the bug

2014 Mentor Graphics Corporation, all rights reserved.
Code Coverage Measures Controllability

100% code coverage does not mean all bugs

are detected [S. Devadas, A. Ghosh, and K. Keutzer. DAC 1996]

DAC paper study found cases where:

Code Coverage Achieved % of covered lines observable

90% Covered Only 54% Observable

100% Covered Only 70% Observable

2014 Mentor Graphics Corporation, all rights reserved.

Assertions Improve Observability
Testbench

2014 Mentor Graphics Corporation, all rights reserved.

Assertions Improve Observability
Testbench

= Bugs missed due to

poor observability
=
Reduce debugging up to 50% [CAV 2000, IBM FoCs paper]
Bugs detected closer to their source due to improved observability
2014 Mentor Graphics Corporation, all rights reserved.
Outline

Effort Spent in Verification Today

Observability and Controllability Challenge

Assertion-Based Verification

Industry Case Studies

Conclusions

2014 Mentor Graphics Corporation, all rights reserved.

Assertion-Based Verification

How can one check a large routine in the

sense of making sure that its right? In
order that the man who checks may not
have too difficult a task, the programmer
should make a number of definite assertions
which can be checked individually, and from
which the correctness of the whole program
easily flows.
Alan Turing, 1949

2014 Mentor Graphics Corporation, all rights reserved.

Property
Property
a statement of design intent Testbench
test
used to specify behavior
env

DUT

2014 Mentor Graphics Corporation, all rights reserved.

Assertion
Property
a statement of design intent Testbench
test
used to specify behavior
env
Assertion
A verification directive

A
Trace from
simulation DUT

2014 Mentor Graphics Corporation, all rights reserved.

High-Level Assertion
Property
a statement of design intent Testbench
test
used to specify behavior
env
Assertion
A verification directive
High-level
Architectural focused
Can be part of testbench
A
Trace from
simulation DUT

2014 Mentor Graphics Corporation, all rights reserved.

Low-Level Assertion
Property
a statement of design intent
used to specify behavior
Assertion RTL
A verification directive A

High-level
Architectural focused A
Can be part of testbench
// Assert that the FIFO controller
Low-level // cannot overflow nor underflow
Implementation focused
Embedded in or bind to the RTL
2014 Mentor Graphics Corporation, all rights reserved.
Who should create the assertions?

Verification Engineer Design Engineer

High-Level Assertions Low-Level Assertions

Requirement focused Implementation focused
Black-box assertions White-box assertions
Accounted for in testplan Not accounted for in testplan
Compliance traceability Improve observability
Create reusable ABV IP Reduce debugging time
2014 Mentor Graphics Corporation, all rights reserved.
Who should create high-level assertions?

Verification Engineer Design Engineer

High-Level Assertions Low-Level Assertions

Requirement focused Implementation focused
Black-box assertions White-box assertions
Accounted for in testplan Not accounted for in testplan
Compliance traceability Improve observability
Create reusable ABV IP Reduce debugging time
2014 Mentor Graphics Corporation, all rights reserved.
Who should create low-level assertions?

Verification Engineer Design Engineer

High-Level Assertions Low-Level Assertions

Requirement focused Implementation focused
Black-box assertions White-box assertions
Accounted for in testplan Not accounted for in testplan
Compliance traceability Improve observability
Create reusable ABV IP Reduce debugging time
2014 Mentor Graphics Corporation, all rights reserved.
 Use of Advanced Verification Techniques
2007
2012
48%
Code coverage 70%

37%
Assertions 68%

40%
Functional coverage 71%

41%
Constrained-Random Simulation 62%

0% 20% 40% 60% 80%

Non-FPGA Study Participants
Source: Wilson Research Group and Mentor Graphics, 2012 Functional Verification Study

2014 Mentor Graphics Corporation, all rights reserved.

Specifying Design Intent

Assertions allow us to specify design intent

in a way that lends itself to automation

clk
grant0
reset_n
Arbiter grant1
req0
req1

// Assert that the grants for our simple arbiter are mutually exclusive

2014 Mentor Graphics Corporation, all rights reserved.

Identifying the Error Condition

For our arbiter example, we can write a Boolean

expression for the error condition, as follows:

clk
grant0
reset_n
Arbiter grant1
req0
req1

(grant0 & grant1) // error condition

2014 Mentor Graphics Corporation, all rights reserved.
Checking the Error Condition before Assertions

Doesnt lend itself to automation.

module arbiter (clk, rst_n, req0, req1, grant0, grant1);
...
always @(posedge clk or negedge rst_n) begin
Error
if (rst_n != 1b0) Condition
Boolean
if (grant0 & grant1) Expression
$display (ERROR: Grants not mutex);
...
endmodule

2014 Mentor Graphics Corporation, all rights reserved.

 Assertion Language & Library Trends
80%
2007
2012
60%
Non-FPGA Study Participants

Next Year

40%

20%

0%
OVL SVA PSL Synopsys OVA OTHER
Assertion Languages and Libraries
Wilson Research Group and Mentor Graphics, 2012 Functional Verification Study, Used with permission * Multiple answers possible

2014 Mentor Graphics Corporation, all rights reserved.

IEEE 1800 SystemVerilog Mutex Example
grant0 and grant1 must be mutually exclusive

clk

grant0

grant1
error

assert property ( @(posedge clk) disable iff (rst_n) !(grant0 & grant1));

2014 Mentor Graphics Corporation, all rights reserved.

IEEE 1850 PSL Fair Arbiter Example
grant0 and grant1 must be mutually exclusive

clk

grant0

grant1
error

assert always (!(grant1 & grant2) abort !rst_n) @(posedge clk);

2014 Mentor Graphics Corporation, all rights reserved.

Accellera OVL Memory Address Example
grant0 and grant1 must be mutually exclusive

clk

grant0

grant1
error

ovl_always a_mutex (clk, rst_n, !(grant1 & grant2));

2014 Mentor Graphics Corporation, all rights reserved.

Outline

Effort Spent in Verification Today

Observability and Controllability Challenge

Assertion-Based Verification

Industry Case Studies

Conclusions

2014 Mentor Graphics Corporation, all rights reserved.

 Published Data on Assertions Use
Percentage bugs found by various techniques

Assertion Monitors 34%

Cache Coherency Checkers 9% 17% of bugs found by assertions on Cyrix M3(p1) project
Register File Trace Compare 8%
Memory State Compare 7% [Krolnik '98]
End-of-Run State Compare 6%
PC Trace Compare 4% 50% of bugs found by assertions on Cyrix M3(p2) project
Self-Checking Test 11%
Simulation Output Inspection 7% [Krolnik 98]
Simulation Hang 6%
Other 8% 85% of bugs found using over 4000 assertions on an HP server

Kantrowitz and Noack [DAC 1996] chipset project

[Foster and Coelho HDLCon 2001]
Assertion Monitors 25% Thousands of assertions in Intel Pentium project
Register Miscompare 22%
Simulation "No Progress 15% [Bentley 2001]
PC Miscompare 14%
Memory State Miscompare 8% 10,000 OVL assertion in Cisco project
Manual Inspection 6%
Self-Checking Test 5% [Sean Smith 2002]
Cache Coherency Check 3%
SAVES Check 2%

Taylor et al. [DAC 1998]

2014 Mentor Graphics Corporation, all rights reserved.

DAC 2008 Sun paper with lots of metrics
Assertion-Based Verification of a 32 thread SPARC CMT Processor
[Turumella, Sharma, DAC 2008]

Category Unique Instantiated

Low-Level 3912 132773
Interface 5004 44756
High-Level 1930 18618

Bugs Found Using Assertions Bugs Found by Type of Asse rtion

Low-level
Form al Interface
Sim ulation
High-level

2014 Mentor Graphics Corporation, all rights reserved.

Significant reduction in debugging time
Assertion-Based Verification of a 32 thread SPARC CMT Processor
[Turumella, Sharma, DAC 2008]

Category Unique Instantiated

Low-Level 3912 132773
Interface 5004 44756
High-Level 1930 18618

Average Debug Time

16
14
12 >50%
10 85% Formal
Hours

8 Sim + Assert
6 Sim + None
4
2
0
Formal Sim + Assert Sim + None
2014 Mentor Graphics Corporation, all rights reserved.
Outline

Effort Spent in Verification Today

Observability and Controllability Challenge

Assertion-Based Verification

Industry Case Studies

Conclusions

2014 Mentor Graphics Corporation, all rights reserved.

Assertion-Based Verification

The process of creating assertions forces the

engineer to think. . . and in this incredible
world of automation, there is no substitute for
thinking.

2014 Mentor Graphics Corporation, all rights reserved.

Session Recap

This session discussed. . .

Todays productivity challenges

How to improve verification

productivity with ABV

2014 Mentor Graphics Corporation, all rights reserved.

Training and Consulting Resources
Mentor Graphics Training
Scalable Verification Courses
- A wide range of instructor led classes
- Located in public training centers in major cities or onsite at your workplace
- Web-based events with live instructors are also available.

Mentor Graphics Consulting

Questa Verification Methodology JumpStart
Knowledge-Sourcing Model
- Infuse knowledge into your organization while addressing your immediate
product development challenges
2014 Mentor Graphics Corporation, all rights reserved.
Other Resources

Assertion-Based Design
Harry Foster, Adam Krolnik, David Lacey
Springer, 2004

Creating Assertion-Based IP
Harry Foster, Adam Krolnik
Springer, 2008

2014 Mentor Graphics Corporation, all rights reserved.

 Assertion-Based Verification
Introduction to ABV
Harry Foster
Chief Scientist Verification

info@verificationacademy.com | www.verificationacademy.com