Reproduction Package for TACAS 2024 Submission `Btor2-Cert: A Certifying Hardware-Verification Framework Using Software Analyzers'

Abstract

This artifact is a reproduction package for the manuscript "Btor2-Cert: A Certifying Hardware-Verification Framework Using Software Analyzers", submitted to TACAS 2024 (submission ID: 1930). It is published with the DOI 10.5281/zenodo.10013059 on Zenodo.

The artifact consists of source code, precompiled tool executables, and input data used in the evaluation of the manuscript, as well as the raw and processed results obtained from the experiments. Specifically, it contains (1) the source code of the proposed witness translator and the validator Btor2-Val, (2) the executables of the Btor2-to-C task translator Btor2C and the compared software verifiers and validators, (3) the benchmark suite of the word-level hardware-verification Btor2 tasks, and (4) the raw and processed data, including the verification witnesses and the translated files, generated from the evaluation.

The artifact aims to facilitate reproducing the evaluation reported in the manuscript. In particular, it supports the following claims: (1) The witness translator can translate software witnesses to Btor2 witnesses. (2) The validator Btor2-Val is effective and efficient compared to other mature validators. (3) The certifying verifier Btor2-Cert produced certified results on some Btor2 tasks that the state-of-the-art hardware model checkers failed to solve. For more information, please refer to the manuscript.

For reviewers to conveniently test the artifact, we selected a subset of the benchmark tasks that can be analyzed in roughly 2 hours if 1 CPU core and 4 GB of memory are allocated to the TACAS VM. Instructions for a kick-the-tires review pass can be found in the section TL;DR of the README file. To fully reproduce the experiments reported in the manuscript, 2 CPU cores, 16 GB of memory, and roughly 2 months of CPU time are required.

Contents

This artifact contains the following items:

• README.html: this documentation
• LICENSE: license information of the artifact
• Btor2-Cert_TACAS24_paper_1930.pdf: a preprint of the submitted manuscript
• verifiers/: a directory containing the evaluated software verifiers, which are used for generating verification witnesses, and the hardware verifier ABC, which is used as a reference for comparison
• validators/: a directory containing the evaluated validators, which are used for validating the witnesses produced by software verifiers
  • btor2-val/: a directory containing the proposed witness translator and validator for Btor2
• btor2c/: a directory containing the Btor2-to-C task translator Btor2C
• btor2-cert: the proposed certifying hardware verifier based on software analyzers, which translates Btor2 tasks to the programming language C, verifies the translated C programs with software verifiers and generates software witnesses, translates software witnesses to Btor2 witnesses, and finally validates Btor2 witnesses with Btor2-Val
• examples/: a directory containing the example verification tasks presented in the manuscript
• benchmarks/: a directory containing word-level hardware-verification tasks used in our evaluation
• bench-defs/: a directory containing the benchmark definitions for the experiments (used by BenchExec, a framework for reliable benchmarking)
• data-submission/: a directory containing the raw and processed data produced from our full evaluation (used in the manuscript, under paper-results/) and from a demo experiment (prepared for this reproduction package, under demo-results/)
• packages/: a directory containing the necessary Debian and Python packages to set up the environment for the experiments in the TACAS '23 Artifact Evaluation VM
• scripts/: a directory containing utility scripts
• Makefile: a recipe for assembling commands to run experiments and process data