zk-Bench: A Toolset for Comparative Evaluation and Performance Benchmarking of SNARKs

Abstract

Zero-Knowledge Proofs (ZKPs), especially Succinct Non-interactive ARguments of Knowledge (SNARKs), have garnered significant attention in modern cryptographic applications. Given the multitude of emerging tools and libraries, assessing their strengths and weaknesses is nuanced and time-consuming. Often, claimed results are generated in isolation, and omissions in details render them irreproducible. The lack of comprehensive benchmarks, guidelines, and support frameworks is a major barrier in the development of ZKP applications.

In response to this need, we introduce zk-Bench, the first benchmarking framework and estimator tool for performance evaluation of public-key cryptography, with a specific focus on practical assessment of general-purpose ZKP systems. To simplify navigating the complex set of metrics and qualitative properties, we offer a comprehensive evaluation platform, which enables the dissection and analysis of tools for ZKP development to uncover trade-offs throughout the development stack; from low-level arithmetic libraries, to high-level tools for SNARK development.

Using zk-Bench, we (i) collect data across 13 different elliptic curves implemented across 9 libraries, (ii) evaluate 5 tools for ZKP development and (iii) provide a tool for estimating cryptographic protocols, instantiated for the $\mathcal {P}\mathfrak {lon}\mathcal {K}$ proof system, achieving an accuracy of 6–32% for ZKP circuits with up to millions of gates. By evaluating zk-Bench for various hardware configurations, we find that certain tools for ZKP development favor compute-optimized hardware, while others benefit from memory-optimized hardware. We observed performance enhancements of up to 40% for memory-optimized configurations and 50% for compute-optimized configurations, contingent on the ZKP development tool.

Appendices

A Detailed Goals for zk-Bench

Completeness: zk-Bench covers quantitative metrics for components shown in Fig. 3, including low-level arithmetic operations, public-key operations, proof system performance, and circuit execution. We support 9 arithmetic libraries covering 13 elliptic curves and 5 tools for ZKP development in $\sim 3\,300$ lines of Go, 2 600 lines of Rust, 2 000 lines of Python, and 1 800 lines of JavaScript.

Modularity: We use and enhance benchmarks from upstream libraries, contributing modifications back to the source. Popular benchmarking libraries provide accurate timing measurements. Our modular framework allows external developers to extend it easily: each arithmetic operation and circuit is tested separately, with isolated benchmarks. Minimal effort is needed to extend our circuit benchmarks. For example, integrating Bellman requires 81 lines of Rust and 40 lines of Python, while adding a new circuit needs 30 lines of Rust. Integrating a DSL like Circom requires 210 lines of scripting and 40 lines of Python, with no extra lines needed for adding circuits beyond the circuit itself.

Reproducibility: Benchmarks are automated, public⁷ and executed on standardized cloud hardware. This ensures consistent and comparable results, facilitating reliable performance evaluations.

Pragmatism: Our objective is to evaluate conditions that mirror practical deployments: we set optimization flags whenever possible, and use all available machine cores. To our knowledge, no standard formats exist for recording raw benchmark data. We develop a unified format for future use by other developers.

Table 2. (a) Overview of arithmetic libraries and elliptic curves currently present in our benchmarking framework. (b) Overview of ZKP tools and proof systems as included in our benchmarking framework, and curves/fields that we consider in the results section. *We use Circom with Snarkjs and Rapidsnark.

Fig. 10.

Overview of common SNARK and STARK frameworks. (F)rontend and (B)ackend of related frameworks are grouped together. Updated 08/01/2023.

Fig. 11.

Proof Size for the Exponentiate circuit as observed on m6i.8xlarge. The observed proof size for circom/snarkjs and circom/rapidnsark is the same.

Fig. 12.

Comparison of memory consumption and execution time for CPU-optimized (c6i.12xlarge) vs. RAM-optimized (r6i.8xlarge) machines for proving Exponentiate circuits. Dark colors indicate higher execution time or memory usage for the CPU-optimized machine than the RAM-optimized machine.

Fig. 13.

Memory consumption of Exponentiate circuit on m6i.8xlarge. Circom’s memory consumption for verification is constant, as its API requires deserialization of the relatively small verification key, public, and proof. Other ZKP tools and libraries require deserialization of files that grow with circuit size. Memory consumption for Starky is missing due to the lack of serialization APIs.

B Related Work

eBATS [54] is the largest effort in benchmarking public-key cryptographic operations, focusing on Diffie-Hellman (scalar multiplication), KEM, and sign operations (multiplication and addition) rather than modern cryptography and runtime estimation. Baghery et al. [5] examine the setup phase of certain zk-SNARKs, and Botrel et al. [18] offer comparative insights into public-key operations in ZKP libraries such as gnark and arkworks. Unlike these works, we provide a comprehensive framework for both arithmetic operations and zk-SNARKs. The drive to standardize ZKPs led to a proposal for a benchmarking framework [12], laying foundational concepts that we expand upon in zk-Bench.

Practitioners are increasingly interested in benchmarking public-key cryptography libraries and tools for ZKP development. Housni evaluates the performance of MSMs and pairings in various elliptic curve implementations [28]. Separately, Bloemen benchmarked implementations of polynomial commitments [16]. We extend these efforts by offering an automated framework to systematically benchmark and compare library implementations across various operations.

Celer Network published a blog post [53] focusing on benchmarking the time and memory costs of proving SHA-256 circuits in various ZKP tools. In contrast, our work provides an extensible framework that supports arbitrary circuits, libraries, and tools, thoroughly benchmarking all phases involved in ZKPs. While we also used SHA-256 circuits where a trusted team provided an implementation, the results can depend heavily on circuit optimization. Therefore, we also benchmarked a straightforward exponentiation circuit across the evaluated tools. Additionally, Anoma has developed a framework [3] for benchmarking different ZKP compilation strategies, focusing on execution time. Lastly, Delendum has developed a framework for benchmarking zkVMs [26].

C Limitations

Hardware Considerations. Our evaluation focused on commodity machines, excluding GPUs, FPGAs, and mobile devices. To fully grasp real-world applicability, future work should include benchmarks on these platforms.

Naïve Public-Key Operations. Points in an elliptic curve can be represented in multiple ways (depending on the curve, affine points – in Weirstrass, Montgomery, or Edwards form – or projective – in standard, Jacobi, or Chudnovsky coordinates). Additionally, different algorithms might have different constraints: no heap allocations (that’s the case for curve25519), constant time, etc. Comparing different algorithms on the same ground is outside the scope of our current arithmetic benchmark. A broader study incorporating these optimizations can elucidate the trade-offs involved. We view an exploration of the impact of optimizations across ZKP tools as promising future work.

Scope of Evaluated Circuits. Our ZKP backend currently evaluates a limited set of test vectors, with performance varying significantly based on circuit implementation (cf. Sect. 5). Given the diversity and complexity of potential ZKP circuits, our chosen circuits may not fully represent all performance scenarios. However, zk-Bench ’s extensibility allows for the easy integration of additional benchmarks for more comprehensive future evaluations.