Constraint Random Verification with Python and Cocotb

Philipp Wagner
 Agenda

› A very quick cocotb introduction

› What is constraint random verification?
› How to use constraint random with cocotb: different approaches
› A look at code examples

2 www.aldec.com Constraint Random Verification with Python and Cocotb

 About Philipp

› Full stack hardware-software engineer. A little bit of

everything*: architecture, RTL, verification,
integration, ...
› Co-maintainer of cocotb
› Director at the FOSSi Foundation
› Contact me
› imphil on GitHub
› mail@philipp-wagner.com
› @MrImphil on Twitter

* earworm recommendation: Welcome to the Internet - Bo Burnham

3 www.aldec.com Constraint Random Verification with Python and Cocotb

 A very quick cocotb introduction

4 www.aldec.com Constraint Random Verification with Python and Cocotb

 Why cocotb?

Verification is software
With cocotb, you write verification like software.
In Python.

5 www.aldec.com Constraint Random Verification with Python and Cocotb

 cocotb overview

your testbench (Python) RTL simulator

e.g. Riviera-PRO, Active-HDL and many others

cocotb library cocotb your RTL code (SV, VHDL)

VPI, FLI, VHPI

6 www.aldec.com Constraint Random Verification with Python and Cocotb

 fifo.sv: the device under test (DUT)

module fifo #(
parameter Width = 8,
parameter Depth = 32
) (
input logic clk,
input logic rst,

input logic [(Width-1):0] din,

input logic wr_en,
output logic full,

output logic [(Width-1):0] dout,

input logic rd_en,
output logic empty
);

7 www.aldec.com Constraint Random Verification with Python and Cocotb

 Single FIFO write: setup

@cocotb.test()
async def test_fifo_manual(dut):
# clock with 1 ns period
cocotb.start_soon(Clock(dut.clk, 1, units='ns').start())

# reset (2 cycles)
dut.din.value = 0
dut.wr_en.value = 0
dut.rd_en.value = 0

dut.rst.value = 1
await ClockCycles(dut.clk, 2)
dut.rst.value = 0

# cont. on next slide

8 www.aldec.com Constraint Random Verification with Python and Cocotb

 Single FIFO write: check
# continued from previous slide

# Write a single word into the FIFO

dut.din.value = 100
dut.wr_en.value = 1
await RisingEdge(dut.clk)
dut.wr_en.value = 0

if dut.empty.value.integer != 0:
await FallingEdge(dut.empty)

dut.rd_en.value = 1
await RisingEdge(dut.clk)
dut.rd_en.value = 0

# one cycle latency

await RisingEdge(dut.clk)

expected_output = 100

assert dut.dout.value.integer == expected_output

9 www.aldec.com Constraint Random Verification with Python and Cocotb

 Run it!

› Create a Makefile (see cocotb documentation)

› make SIM=riviera clean sim GUI=1

10 www.aldec.com Constraint Random Verification with Python and Cocotb

11 www.aldec.com Constraint Random Verification with Python and Cocotb
 We’ve seen a directed test

› “Hardcoded” test
› Great for bringup (“designer tests”)
› Great for understanding the test
› Great to ensure that bugs stay fixed

Are you sure you thought about

all the interesting corner cases?

12 www.aldec.com Constraint Random Verification with Python and Cocotb

 What is constraint-driven test generation?

13 www.aldec.com Constraint Random Verification with Python and Cocotb

 Constraint random verification: The basics

› Automatic test generation

› Don’t specify test values directly.
› Instead, specify constraints (e.g., min/max values)
› A solver generates random test values based on those constraints.

› Benefits
› Write exhaustive test benches more efficiently.
› Finds corner cases you haven’t thought about.

14 www.aldec.com Constraint Random Verification with Python and Cocotb

 As known from SystemVerilog

class Bus;
rand bit[15:0] addr;
rand bit[31:0] data;
constraint word_align {addr[1:0] == 2'b0;}
endclass

Bus bus = new;

repeat (50) begin
if ( bus.randomize() == 1 )
$display ("addr = %16h data = %h\n", bus.addr, bus.data);
else
$display ("Randomization failed.\n");
end

code example taken from the SystemVerilog-2017 LRM

15 www.aldec.com Constraint Random Verification with Python and Cocotb

 Challenges with constraint random test generation

› Huge amounts of (generated) tests!

› “When are we done?”

16 www.aldec.com Constraint Random Verification with Python and Cocotb

 What is functional coverage?

17 www.aldec.com Constraint Random Verification with Python and Cocotb

 Coverage: The answer to “Are we done yet?”
(Objectively) measure the quality of the verification.

Types of coverage Types of code

› Code coverage › RTL
› Line, statement, toggle, FSM, ...
› Tests/verification code
› Functional coverage

Looking further: How to collect other types of coverage

Code coverage can be automatically collected by your simulator.

→ https://docs.cocotb.org/en/stable/simulator_support.html#coverage

Python test/verification coverage is done through Coverage.py.

→ https://docs.cocotb.org/en/stable/building.html#envvar-COVERAGE

18 www.aldec.com Constraint Random Verification with Python and Cocotb

 Functional coverage

“Functional coverage is a user-defined metric that measures how much of the

design specification, as enumerated by features in the test plan, has been
exercised. It can be used to measure whether interesting scenarios, corner
cases, specification invariants, or other applicable design conditions—captured
as features of the test plan—have been observed, validated, and tested.”
SystemVerilog-2017 LRM

Example: FIFO

Especially interesting cases: read while empty, write while full

› Level: full, empty
› Direction: read, write
› Crosses: level x direction

19 www.aldec.com Constraint Random Verification with Python and Cocotb

 Functional coverage: SystemVerilog terminology

› Coverage point (coverpoint): a variable or expression to collect coverage from.

› Bins: groups of “semantically similar” states or transitions.
Example: “FIFO is 2/3 full”, “FIFO is 1/3 full”
› Cross coverage: combinations between multiple coverage points.
› Coverage group (covergroup): a collection of all of the above + sampling
information + configuration.

20 www.aldec.com Constraint Random Verification with Python and Cocotb

 How to use constraint random verification with cocotb
A look at the different approaches

21 www.aldec.com Constraint Random Verification with Python and Cocotb

 Cocotb, constraints and coverage

› cocotb does not ship with or mandate a particular approach

› Extensions through Python decorators and embedded DSLs
@decorator
def my_function():
pass

At least two options

› cocotb-coverage by Marek Cieplucha and contributors
› PyVSC by Matthew Ballance and contributors

22 www.aldec.com Constraint Random Verification with Python and Cocotb

 PyVSC
SystemVerilog-style constraints and coverage in Python

23 www.aldec.com Constraint Random Verification with Python and Cocotb

 PyVSC

› Functional coverage and constraint random test generation for Python

› Usable with cocotb, but also with other Python-based verification approaches
› Aims for feature parity with SystemVerilog
› Permissively licensed (Apache 2 license)

https://pyvsc.readthedocs.io

24 www.aldec.com Constraint Random Verification with Python and Cocotb

25 www.aldec.com Constraint Random Verification with Python and Cocotb
 cocotb-coverage
Functional Coverage and Constrained Randomization
Extensions for Cocotb

26 www.aldec.com Constraint Random Verification with Python and Cocotb

 cocotb-coverage

› Constraints are Python functions and solved using a pure-Python SAT solver
› No covergroups, trees of coverpoints
› Inspired by SystemVerilog, but extended with own ideas
› Permissively licensed (BSD license)

https://github.com/mciepluc/cocotb-coverage for code and documentation, including

DVCon papers and talks

M. Cieplucha and W. Pleskacz, “New architecture of the object-oriented functional

coverage mechanism for digital verification,” in 2016 1st IEEE International Verification
and Security Workshop (IVSW), July 2016, pp. 1–6.

27 www.aldec.com Constraint Random Verification with Python and Cocotb

28 www.aldec.com Constraint Random Verification with Python and Cocotb
 cocotb-coverage: view coverage

› cocotbcoverageview
https://github.com/JoseIuri/cocoTBCoverageView

› UCIS export (in development)

https://github.com/mciepluc/cocotb-coverage/pull/46

29 www.aldec.com Constraint Random Verification with Python and Cocotb

 Randomization and seeds

› Both PyVSC and cocotb-coverage use Python’s random module

› To fix a seed: Set the RANDOM_SEED environment variable in cocotb
(uses random.seed() behind the scenes)

› Many functions to obtain random values:

› random.randint()
› random.getrandbits()
› …

https://docs.python.org/3/library/random.html
https://docs.cocotb.org/en/stable/building.html#envvar-RANDOM_SEED

30 www.aldec.com Constraint Random Verification with Python and Cocotb

 Summary

› cocotb brings the power of Python to hardware verification

› Functional coverage helps to see progress against the test plan
› Constraint random test generation eases testbench development, and finds
cornercases.
› Multiple ways to do CRV in cocotb exist
› cocotb-coverage: closer to Python
› PyVSC: domain-specific language, very close to SystemVerilog

31 www.aldec.com Constraint Random Verification with Python and Cocotb

 Constraint Random Verification with Python and Cocotb
Questions & Answers
RECAP ON POINTS DISCUSSED:
›
A very quick cocotb introduction › We invite you to ask questions,
›
Constraint random test generation which the host will put to Philipp.
›
Functional coverage
›
cocotb-coverage
›
PyVSC
› To submit a question please use the
Question Panel on the righthand
side of your screen.

You will be emailed a link to a recording of today’s webinar and you will be able to download
these slides. For upcoming webinars please visit www.aldec.com/events

32 www.aldec.com Constraint Random Verification with Python and Cocotb

 Corp. Headquarters Solutions
2260 Corporate Circle , Suite 400 Riviera-PRO™
Henderson, NV 89074 Advanced Verification Platform
Active-HDL™
Ph +1.702.990.4400 FPGA Design and Simulation
Fax +1.702.990.4414
sales@aldec.com ALINT-PRO™
Design Rule Checking
HES-DVM™
HW/SW Validation Platform

Traininghttps://www.aldec.com/training TySOM™
High Performance Embedded Development
Spec-TRACER™
Support https://www.aldec.com/support Requirements Lifecycle Management
DO-254/CTS™
FPGA Level In-Target Testing
Blog https://www.aldec.com/company/blog
HES™
SoC/ASIC Prototyping
RTAX/RTSX
Prototyping Microchip™ Rad-Tolerant Devices

33 www.aldec.com Constraint Random Verification with Python and Cocotb