LL-GNN: Low Latency Graph Neural Networks on FPGAs for High Energy Physics

Graph neural networks (GNNs) have shown remarkable successes in wide applications with graph-structured data, such as recommendation systems [14, 49, 50], molecule property prediction [15, 46], and various High Energy Physics (HEP) applications, such as jet tagging [29], charged particle tracking [13, 23], and calorimeter energy measurements [31]. In the domain of high energy physics, particularly in large-scale experimental setups such as CERN’s Large Hadron Collider (LHC), the effectiveness of GNNs has been recognized. GNNs can extract and process complex patterns from high-dimensional data, which aligns well with the requirements of such environments where data from particle collisions are highly interconnected. However, incorporation of GNNs into real-time data processing in these settings presents significant challenges, primarily due to the enormous data volume and strict low latency requirements.

Real-time data processing from high-energy proton collisions at the LHC is challenging because the particle detectors around the LHC ring produce hundreds of terabytes of data per second [8, 11] from collisions happening every 25 ns. In the next phase of the LHC, the upgraded High-Luminosity LHC (HL-LHC) experiments expect an explosion of data due to cutting edge detectors with improved resolution and increased area as well as volume. The large data volumes produced from the detectors are reduced by an embedded real-time processing system, known as the trigger, which keeps interesting collision events while discarding the others. The Level-1 Trigger (L1T), which uses only FPGAs, requires sub-microsecond application processing latency in the LHC [9]. If algorithm latency exceeds this strict limit, valuable data and significant events are lost.

This inherent requirement for ultra-low latency, combined with the computational intensity of GNNs, presents a unique challenge in the field of high energy physics. Indeed, GNNs require large amounts of computation and suffer from irregular memory accesses, leading to significant inference latency. This hinders their effective deployment in real-time, ultra-low latency applications like the L1T system in an LHC experiment as suggested by Moreno et al. [29]. Accelerating GNN inference using reconfigurable accelerators such as FPGAs is essential in the LHC as it enables sophisticated processing in real-time with superior accuracy. Nonetheless, most existing FPGA-based GNN accelerators employ a single-engine architecture with fixed hardware to sequentially process layers or sub-layers (blocks) like GPUs, and the networks are processed in a recurrent fashion [24, 26, 27, 28, 47, 53]. However, they are not efficient for GNN execution when handling small graphs that necessitate ultra-low latency and high throughput, such as in scientific applications for particle identification. Furthermore, none of them is designed to operate under a hard latency constraint of one microsecond (\(1\mu\)s).

To address the gap, this work proposes a domain-specific Low Latency (LL)-GNN hardware architecture based on a layer-wise tailor-made pipeline to accelerate the GNNs for particle detectors. The layer-wise architecture has been used to speed up CNNs [5, 39, 55] and RNNs [32, 37], but there remains a lack of research focused on accelerating GNNs. This work uses the GNN-based JEDI-net algorithm [29] as an end-to-end application. JEDI-net delivers state-of-the-art accuracy for particle identification and has sparked a growing interest in the use of GNNs in trigger systems. Our work introduces an outer-product based matrix-matrix multiplication (MMM) approach for calculating the GNN aggregation function, which is further enhanced by exploiting the structured adjacency matrix. In addition, the proposed architecture adopts a column-major order instead of conventional row-major order for the data layout of intermediate results traversed along the hardware datapath, which avoids irregular memory access for intermediate results in GNNs. Moreover, we present a custom code transformation for MMMs of GNN feature transformation based on the character of interaction-network based GNNs that utilize a fully connected graph as input. This approach avoids the costly matrix multiplications between the adjacency matrix and the input feature matrix, thereby enhancing computation efficiency.

Furthermore, instead of running GNNs in a coarse-grained pipeline as shown in our prior work [34, 35], we propose fusion of sub-layers as much as possible, transforming multiple coarse-grained pipeline stages into a single stage with a fine-grained pipeline inside. The fusion eliminates unnecessary handshake acknowledgments and dual buffers between coarse-grained pipeline stages, effectively reducing end-to-end latency. To further optimize latency, a Finite State Machine (FSM)-based code structure is proposed to handle imperfect loops that may emerge after fusion. Finally, we present a GNN-specific algorithm-hardware co-design approach to optimize the algorithm and hardware simultaneously, exploring the optimal configuration to enhance overall performance. Our LL-GNN design successfully achieves sub-microsecond latency, which makes the algorithm compatible with the HL-LHC conditions where there is a hard ultra-low latency constraint, while also enabling improved accuracy.

To the best of our knowledge, this is the first FPGA-based design of GNNs with sub-microsecond latency for particle identification for the detectors at the CERN HL-LHC experiments. This work would help to improve next-generation trigger systems, enabling powerful algorithms to process massive LHC data effectively.

We make the following contributions in this article:

Although the GNN coefficients are HEP specific, the proposed optimizations and techniques have broader implications for other GNN applications beyond particle identification. Most of the techniques and optimizations introduced in Section 3 and Section 4.1\(\sim\)4.4 are general and can be beneficial to a range of GNN applications, not just limited to the JEDI-net. These optimizations include strategies for streamlining matrix operations, hardware pipelining, and ways to optimize FPGA resources for maximum performance, minimizing design latency, and maximizing acceleration efficiency. Moreover, the techniques devised for low latency optimization can benefit numerous other applications on FPGAs, especially those that require real-time processing.

This section introduces the hardware architecture and multiple optimizations to accelerate the interaction network based GNNs. We adopt a 3-step optimization strategy, named “divide, conquer and fusion”, to achieve low latency for GNN inferences. During the “divide” and “conquer” steps, we split GNNs into multiple sub-layers and perform dedicated optimizations, such as outer-product based MMM (Section 3.1) approach and leveraging the structured sparsity of adjacency matrices (Section 3.3) for GNNs. Then, in the “fusion” step (Section 3.5), multiple sub-layers are fused together to eliminate boundaries and buffers between different pipeline stages, resulting in low latency and high hardware efficiency.

Based on the new low latency hardware architecture incorporating fusion, this section presents a two-level parallelism scheme (Section 4.1) to improve the parallelism and reduce latency. Furthermore, a GNN-specific co-design approach is introduced (Section 4.4) to optimize the algorithm and hardware simultaneously. By combining these different optimizations, our designs achieve good performance and efficiency.

This section presents the evaluation results of the GNN-based JEDI-net on FPGAs demonstrating the scalability of the proposed optimization for GNNs.

Several studies have explored the use of GNNs for particle physics applications, such as jet tagging (identification) [29], charged particle tracking [23], and calorimeter energy measurements [31]. More can be found in a survey [43]. To achieve low latency, FPGAs are utilized. The work in [13] extends the hls4ml [11] tool to translate GNNs into FPGA firmware automatically for charged particle tracking. GarNet [22], a GNN-based algorithm, is proposed for calorimeter energy regression.

Numerous studies also focus on general GNN accelerations [16, 17, 18, 28, 44, 53, 58]. AWB-GCN [17] is based on a column-wise-product architecture with runtime re-balancing for GCN acceleration. Their upgraded version I-GCN [18] presents Islandization, a new runtime graph restructuring algorithm, to improve data locality. BoostGCN [53] presents a novel hardware-aware Partition-Centric Feature Aggregation (PCFA) scheme for pipelined GCNs. Lin et al. [28] introduce GCN acceleration using HLS and hardware-friendly optimizations. G-NMP [44] presents a Near-Memory Processing (NMP) solution for accelerating GNNs to handle the irregular memory access. Garg et al. [16] explore various dataflow choices for sparse and dense GNNs on spatial accelerators. The work in [4] designs a taxonomy of parallelism in GNNs. Sohrabizadeh et al. [40] present StreamGCN for accelerating GCN specialized for GNN streaming processing with small graphs. Chen et al. [7] introduce a heterogeneous pipeline architecture for GNNs on high bandwidth memory (HBM) enabled FPGAs. Abi-Karam et al. [2] propose GenGNN framework to deliver ultra-fast GNN inference and support a diverse set of GNN models. The results indicate that their designs achieve latency at the millisecond level. They also propose FlowGNN [38] which can flexibly support the majority of message-passing GNNs. Kang et al. [24] propose a GCN accelerator named GROW with Gustavson’s algorithm to architect a sparse-dense GEMM accelerator with row-wise product. Sun et al. [42] propose MultiGCN which balances network latency and network bandwidth for GCNs in multi-node systems. Yang et al. [48] present a dynamically reconfigurable accelerator for GNNs named DRGN. EGCN [21] is proposed using tiled matrix multiplication to reduce Off-Chip Memory Access. Furthermore, the I-GCN [18] adopts 4,096 multiplier-accumulator (MAC) units running at 350 MHz. Its peak performance is 2,703 Giga Operations Per Second (GOPS). Each MAC is counted as 2 operations since it includes both multiplication and addition. BoostGCN [53] has a peak performance of 1,792 GOPS, running at 250 MHz. GenGNN [2] utilizes 1,344 DSP blocks with 16-bit fixed-point data representation, resulting in a peak performance of 806 GOPS. Their later work, FlowGNN [38] has a peak performance of 1,499 GOPS. Both of them run at 300 MHz. Our design (J4), running at 200 MHz, can not only provide a sub-microsecond latency but also an effective performance of 3,025 GOPS which is higher than I-GCN, BoostGCN, GenGNN, and FlowGNN. The custom MMMs, which do not contain any multiplications or additions, are excluded, with the exception of MMM3 that includes a small number of additions. Finally, none of these previous designs target a sub-microsecond scenario. Although throughput at the application level can serve as a fair metric, the use of various applications for benchmarking makes it more complicated.

There are also previous studies about algorithm and hardware co-design for GNNs [51, 56, 57, 58]. The work in [56] presents a GNN and accelerator automatically co-search framework to maximize both task accuracy and acceleration efficiency. Zhou et al. [58] propose model-architecture co-design with a light-weight algorithm for temporal GNN inferences on FPGAs. You et al. [51] propose GCoD framwork, involving a two-pronged accelerator. Zhong et al. [57] propose an algorithm-hardware co-design scheme with a reuse-aware sampling method to accelerate GNN inference in mini-batch scenarios. Some previous studies focus on accelerating GNN training [27, 30, 41, 52]. GraphACT [52] introduces an FPGA-based accelerator with a subgraph-based algorithm for GCNs training. Su et al. [41] present an efficient graph sampling accelerator on HBM enabled FPGAs for training GNNs. Lin et al. [27] propose HP-GNN which maps GNN training on the CPU-FPGA platform automatically. DietGNN [30], a crossbar-aware pruning technique, is proposed to accelerate the training of large-scale GNNs.

Most of the previous designs utilize a fixed hardware for all the layers of GNNs and process them sequentially. This is not efficient for GNN inference execution when targeting small graphs with requirements of ultra-low latency (such as \(\lt 1\mu s\)) and high throughput for scientific applications, such as particle identification. This work focuses on layer-wise architecture for streaming processing of input graphs. We propose multiple novel optimizations to achieve high throughput and sub-microsecond level latency. These previous studies are orthogonal to our proposed approach and hardware architecture. Their techniques could be complementary to our approach, which could be extended in the future to achieve even lower latency.

This article presents a novel approach for minimizing the latency for processing GNNs on FPGAs, using JEDI-net algorithm as an end-to-end application. It involves optimizing the matrix operations and hardware pipeline to support next-generation low-latency collider trigger systems, the key to many fundamental physics experiments including particle identification. Results show up to 9.0 times reduction in latency over the existing GPU-based JEDI-net implementation and up to 16.7 times reduction over state-of-the-art work. Future work includes exploring the use of new FPGA resources such as the AI Engines [1] and the AI Tensor Blocks [25], automating the proposed co-design approach using techniques such as meta-programming [33, 45], and incorporating the proposed techniques into the design and implementation of the data processing architecture for next-generation collider trigger systems.