Functional Verification Planning and

Management
Quantifying the path from specification to a fully functional design

Andy Piziali
Independent Consultant

Shankar Hemmady
Badri Gopalan
Agenda
 Introduction
 Coverage Basics
 Verification Planning Process
 Verification Plan and Coverage Automation
 Summary

December 7, 2009
Agenda
 Introduction
 Coverage Basics
 Verification Planning Process
 Verification Plan and Coverage Automation
 Summary

December 7, 2009
Introduction
 Problems facing design teams today
 Unpredictable design and verification schedules
 Concurrent execution bugs introduced by multi-core designs
 Functional bug escapes into production designs
 Inefficient application of design and verification resources

 Solutions
 Good design specifications
 Rigorous verification planning
 Automated verification management
 Live, executable verification plan

December 7, 2009
Agenda
 Introduction
 Coverage Basics
 Verification Planning Process
 Verification Plan and Coverage Automation
 Summary

December 7, 2009
Design and Verification Cycle
 Write functional specification
 Write verification plan
 Design and implement DUV (design under verification)
and VE
 Verify DUV
 Analyze results and adapt environment

December 7, 2009
Coverage Measurement and Analysis
 Simulation is based on limited execution samples
 Cannot run all possible scenarios
 Need to know that all areas of the DUV are verified
Solution: Coverage measurement and analysis

 The main ideas behind coverage

 Features (of the specification and implementation) are identified
 Coverage models written to quantify their behavioral spaces

 Main coverage techniques

 Code coverage: coverage models that are based on the implementation
code
 Functional coverage: coverage models that are based on the functionality
of the design
December 7, 2009
 Commonly Used Coverage
 Functional coverage – any coverage using explicit behavioral metrics

 Code coverage – coverage using implicit metrics of RTL or software

 Assertion coverage
1. Coverage using implicit metrics of an assertion
2. RTL lines that are within the logic cone of an assertion
3. Functional coverage implemented using an assertion
4. Density distribution of assertions throughout source code

 Formal coverage
 Full proof: full state space coverage; proof tied to constraints
 Partial proof: either restricted (full state space) or bounded (limited state
space) proof. Origin of state space coverage chosen by user.

 Transaction coverage – implicit or explicit coverage of transactions

December 7, 2009
Agenda
 Introduction
 Verification Basics
 Verification Planning Process
 Verification Plan and Coverage Automation
 Summary

December 7, 2009
Verification Planning Process
 Verification plan structure
 Verification planning methodology
 Verification scope
 Verification solution
 Verification planning automation
 Verification planning example

December 7, 2009
 Verification Plan Structure
1. Introduction ........................................ what does this document contain?
2. Functional Requirements ................... opaque box design behaviors
2.1 Functional Interfaces ........................ external interface behaviors
2.2 Core Features ................................. external design-independent behaviors
3. Design Requirements ........................ clear box design behaviors
3.1 Design Interfaces ............................ internal interface behaviors
3.2 Design Cores .................................. internal block requirements
4. Verification Views .............................. time-based or functional feature groups
5. Verification Environment Design ....... functional specification of the verif env
5.1 Coverage ....................................... coverage aspect functional specification
5.2 Checkers ........................................ checking aspect functional specification
5.3 Stimuli ............................................ stimulus aspect functional specification
5.4 Monitors ......................................... data monitors functional specification

December 7, 2009
 Verification Planning Methodology
Feature extraction Additional verification views

Coveragemodel
Coverage
Coverage modeldesign
model design
design

First order Second order Verification plan inputs

Verification management
Implementation and tracking tool

Coverage model VE implementation Status and progress

Execution engine
Controlling the
Executing engine
December 7, 2009
Verification Scope: Specification
Analysis
 Design intent is captured in the specifications
 Features need to be extracted from the specifications
 Top-down analysis
 Required for large specifications
 Need to distill behavioral requirements
 Intent abstraction gap bridged by the human mind in brainstorm
 Bottom-up analysis
 Suitable for small specifications
 Document analyzed chapter by chapter
 Amenable to analysis automation
 Reuse existing verification plans

December 7, 2009
Problem Scope Captured in Coverage
Model Designs
 Top-level model design
 Write semantic description
 Select attributes and values
 Choose model structure

 Detailed model design

 What to sample for each attribute?
 Where in the verification environment should we sample?
 When should each attribute be sampled?

December 7, 2009
Extracting Design Features
 Design features are behavioral requirements of the device
 They should state what the device is intended to do
 NOT how a behavior will be verified

 Design features have data-based and time-based behaviors

 A device should handle different data values
 A device should have a certain temporal behavior

December 7, 2009
What Should Be Captured?
1. An abstract name for the feature
2. A semantic description of the feature
3. Optional
 Responsibilities
 References to specification, industry standards, tables,
figures, requirements database, ...
 Project schedule information
 Estimate of the expected verification effort
 Short identification tag

December 7, 2009
Coverage Model Design
 Coverage model:
 Coverage space that captures input/output relationships
 A set of legal and interesting coverage points in the coverage
space
 DUV functional requirements of:
 Inputs
 Outputs
 Input-to-output relationships
 Model fidelity
 Abstraction gap
 Omitted functional relationships
December 7, 2009
First- and Second-Order Coverage
Models
 First-Order Model
 Coarse model of a device feature
 Represented by a table row in the verification plan

 Second-Order Model
 More precise model of a device feature
 Represented by a standalone table in the verification
plan

December 7, 2009
 Device Specification
 DUV: wood burning stove
 Specification
 logs: 3 to 6 in the stove
 thermostat: 200˚ to 800˚ in 100˚
increments
 thermometer: 100˚ to 1,000˚
 damper: open or closed
 catalytic converter: ignited or
not ignited
 doors: open or closed
 top lid: open or closed
December 7, 2009
First-Order Coverage Model
Description and Attributes
 Feature name: operating conditions
 Semantic description: Record all operating
conditions of the wood stove defined by logs, thermostat
and damper
 Attributes and values
 logs: 3, 4, 5, 6
 thermostat: 200˚ to 800˚ in 100˚ increments
 damper: open, closed

December 7, 2009
First-Order Coverage Model Top-Level Design

Feature Attribute or Values Monitor

Sampling Time
Operating @logs_loaded 1
Conditions
logs 3, 4, 5, 6 1

thermostat 200˚, 300˚, 1

400˚, 500˚,
600˚, 700˚, 800˚
damper open, closed 1

logs, thermostat, C{}

damper
December 7, 2009
First-Order Coverage Model Size
Attribute Values Number of
Points

logs 3, 4, 5, 6 4

thermostat 200˚, 300˚, 400˚, 500˚, x7

600˚, 700˚, 800˚
damper open, closed x2

Total: 56

December 7, 2009
First-Order SystemVerilog Coverage
Model Implementation: Attributes

class WoodStove;
rand int logs;
rand int stat;
rand enum {OPEN, CLOSED} damper;
endclass: WoodStove

December 7, 2009
First-Order SV Coverage Model
Implementation: Base Coverage Group
covergroup wood_stove_cg;
coverpoint logs {bins l[] = {[3:6]};}
coverpoint stat {bins s[] =
{200, 300, 400, 500, 600, 700, 800};}
coverpoint damper;
cross logs, stat, damper;
endgroup

December 7, 2009
Second-Order SystemVerilog Coverage
Model Implementation: Model Structure
covergroup wood_stove_cg;
...
cross logs, stat, damper {
ignore_bins EB =
((!binsof(logs) intersect {3,4} ||
!binsof(stat) intersect {200,300} ||
!binsof(damper) intersect {OPEN}) &&
(!binsof(logs) intersect {3,4} ||
!binsof(stat) intersect {400}) &&
(!binsof(logs) intersect {5} ||
!binsof(stat) intersect {500}) &&
(!binsof(logs) intersect {6} ||
!binsof(stat) intersect {600}) &&
(!binsof(logs) intersect {6} ||
!binsof(stat) intersect {700,800} ||
!binsof(damper) intersect {CLOSED}));
}
endgroup

December 7, 2009
Assertion-Based Coverage
 Assert statements
 Did all my assertions execute?
 Any unverified assertions?
 How many passed? Failed? Incomplete?

 Cover statements
 Functional conditions that must be exercised for complete
verification
 Concurrent properties that have the same syntax as concurrent
assertions

December 7, 2009
Assertion-Based Coverage Examples
 Data/address values at key times
 Reset during every state
 Overlapping writes to dual port RAM
 Design specific data requirements

 Ranges of vectors during transactions

 Legal & illegal ranges

 Corner cases
 Buffer full or empty
 FIFO overflow

December 7, 2009
 Assertion Coverage Implementation:
SVA Cover Examples

“Did we ever see: Areq2 followed within 10 cycles by Areq3 followed

within 10 cycles by Areq0?

SEQ_REQ2_REQ3_REQ0: cover property (@(posedge Aclk)

(!Arst) throughout Areq2 ##[0:10] Areq3 ##[0:10] Areq0);

“How many times were 3 bits high in my vector?”

c2: cover property (@(posedge clk) !(resetting)

throughout ($countones(my_vector) == 3));

December 7, 2009
Agenda
 Introduction
 Coverage Basics
 Verification Planning Process
 Verification Plan and Coverage Automation
 Summary

December 7, 2009
Verification Planning Automation
 Use the plan like a GPS to answer questions
 Where are we? What is the next waypoint?
 How far are we from functional closure?
 When will we reach functional closure?
 Verification plan requirements in executable form
 Written in an Excel spreadsheet or MS Word document
 Translated to HVPL for verification environment linkage
 An example of a VPA toolkit: Synopsys VMM Planner
 Spreadsheet or Hierarchical Verification Plan (HVP) input
 Executable verification plan back-annotated by VMM Planner

December 7, 2009
Executable Verification plan example:
VMM Planner

Plan in Spreadsheet
MS Excel Spreadsheet Report

Plan in Word Doc

Word Doc Report
MS Word
VMM
Planner
Text File
Editor Plan in HTML
HVP language Report

VMM Planner
GUI Editor

Coverage Other
Database measurable
Data

December 7, 2009
 User Writes HVP Spreadsheet Using MS Excel

Components

Interface

 “Components” and “Interface” are feature names

 “cache.*” and “core.*” are references to HLVL coverage groups
 “cache_test” identifies a test whose coverage should be tracked
 MSWord and HTML are alternative report formats

December 7, 2009
VMM Planner: Back-Annotation From
Verification Sources

December 7, 2009
Functional Coverage Automation

 Functional coverage implementation is time and resource-intensive

 Automate functional coverage where possible

 Assertion-based coverage from checkers
 Register and memory map coverage
 Low power verification coverage
 Verification IP for coverage
 Formal-driven automatically extracted properties

December 7, 2009
Functional Coverage Automation:
SVA Checker Library
 SVA checker library contains built-in coverage
 In addition to performing function checks, each checker can collect and report
coverage
 Detailed statistics and transaction information
 Has a FIFO been filled and emptied
 Did an arbiter receive a grant for each channel
 Have all variations of legal transactions occur
 Did maximum and minimum latencies occur

 Controlled though parameters

 coverage_level_1, _2, _3 parameters

December 7, 2009
Functional Coverage Automation:
Register Functional Coverage
Example: VMM Register Abstraction Layer (RAL)
• Provides automatically generated, extensible coverage model
• Register bits coverage model: large, best used at the block level
• Address map coverage model: a subset of bit model, used in system level or top
level
• Field value coverage model: Should be used in system level or top level

• User-written
• Configuration coverage model
• Control register value, sequence, transition, cross coverage

December 7, 2009
Functional Coverage Automation:
Coverage Inferred From SV Test Bench

 Rule based generation of coverage model from test

bench constraints

 Auto-inferred coverage model for

 Coverage of random variables
 Condition coverage over constraints
 Set membership coverage
 Cross coverage

December 7, 2009
 Functional Coverage Automation:
Inferred Implicit Coverage For SystemVerilog

constraint c { cp1: coverpoint (p || (q && r) {

if (p || (q && r)) {x inside {[0:5]}; bin b1 = {1} iff (p);
} bin b1 = {1} iff (q && r);
}
cp2 : coverpoint x {
bin b1 = {0};
bin b2 = {[1:4]};
bin b3 = {5};
}
cc1 : cross cp1, cp2;

December 7, 2009
Functional Coverage Automation:
Extracting Power FSM Model from UPF specification

Clock ISO_ Reset

Gated Enable On

Core
All On
Off All On Core
Off

Power FSM with Power FSM with voltage and

voltage states logic states

December 7, 2009
Functional Coverage Automation:
Low Power-Annotated VMM Verification Plan

Automatic
Low Power
Coverage

December 7, 2009
 Functional Coverage Automation:
VIP Functional Coverage
VIPs typically provide built-in coverage and sub-plans. For example:
 Synopsys Designware AXI VIP
 State coverage for different transaction features
 E.g., length, sequence type, transfer size, response
 Transaction flow coverage
 E.g., Ordering of handshakes, etc., within transactions
 E.g., Interleaving between transactions, etc.
 Error messages
 Cross coverage of important relationships
 Meta coverage for delays, etc.,

Issues to consider:
 Hierarchical planning with “import” capabilities for sub-plans
 Disabling specific items in sub-plan based on VIP configuration
 Creation of useful cross-points using items in the VIP and top-level items
December 7, 2009
Agenda
 Introduction
 Coverage Basics
 Verification Planning Process
 Verification Plan and Coverage Automation
 Summary

December 7, 2009
Summary
 The verification process is more demanding
 Increasing complexity
 Shorter time to market
 Failure can be disastrous

 Well-planned and controlled verification process can:

 Expose and address risks and challenges earlier
 Remove uncertainties from the process
 Improve verification (and design) quality
 Increase productivity

December 7, 2009