Asymmetric Data Acquisition System for an endoscopic PET-US detector

Asymmetric Data Acquisition System for an Endoscopic PET-US Detector Carlos Zorraquino, Ricardo Bugalho, Manuel Rolo, Jose C. Silva, Viesturs Vecklans, Rui Silva, Catarina Ortigão, Jorge A. Neves, Stefaan Tavernier, Pedro Guerra and João Varela Abstract–According to current prognosis studies of pancreatic cancer, survival rate nowadays is still as low as 6% mainly due to late detections. Taking into account the location of the disease within the body and making use of the level of miniaturization in radiation detectors that can be achieved at the present time, EndoTOFPET-US collaboration aims at the development of a multimodal imaging technique for endoscopic pancreas exams that combines the benefits of high resolution metabolic information from Time-Of- Flight (TOF) Positron Emission Tomography (PET) and anatomical information from ultrasound (US). A system with such capabilities calls for an applicationspecific high-performance Data Acquisition System (DAQ) able to control and readout data from two different detectors. The system is composed of two novel detectors designed within the collaboration; a PET head extension for a commercial US endoscope placed internally close to the Region-Of-Interest (ROI) and a PET plate placed over the patient’s abdomen in coincidence with the PET head. These two detectors will send asymmetric data streams that need to be handled by the DAQ system. The approach chosen to cope with these needs goes through the implementation of a DAQ capable to perform multilevel triggering and which is distributed across two different ondetector electronics and the off-detector electronics placed inside the reconstruction workstation. Present contribution gives an overview on the design of this innovative DAQ system and, based on results obtained by means of final prototypes of the two detectors and DAQ, we conclude that a distributed multi-level triggering DAQ system is suitable for endoscopic PET detectors and it shows potential for its application in different asymmetric scenarios. I. INTRODUCTION EndoTOFPET-US collaboration [1] was conceived to THE cope with the need of a tool to study newest specific Manuscript received June 16, 2014. This work, as part of PicoSEC MCNet Project, is supported by a Marie Curie Early Initial Training Network Fellowship of the European Community’s Seventh Framework Programme under contract number (PITN-GA-2011-289355-PicoSEC-MCNet). And EndoTOFPETUS has received funding from the European Union 7th Framework Program (FP7/2007-2013) under Grant Agreement No. 256984. Carlos Zorraquino, Ricardo Bugalho, Manuel Rolo, Viesturs Veckalns, Rui Silva, Catarina Ortigão, and Jorge Neves are with the Laboratório de Instrumentação e Física Experimental de Partículas, Lisboa, 1000-149 PT (emails: carlos.zorraquino.gaston@cern.ch, ricardo@lip.pt, mrolo@lip.pt, viesturs@lip.pt, rsilva@lip.pt, ortigao@lip.pt and janeves@lip.pt). Jose Carlos Silva and João Varela are with both the Laboratório de Instrumentação e Física Experimental de Partículas, LIP, Lisboa, 1000-149 PT and the European Organization for Nuclear Research, CERN, Geneve, 1217 SWZ (e-mails: Jc.Silva@cern.ch and joao.varela@cern.ch). Pedro Guerra is with the Universidad Politécnica de Madrid, Madrid, 28040 SP (e-mail: pedro.guerra@upm.es). Stefaan Tavernier is with the Vrije Universiteit Brussel, Brussel, 1050 BE (e-mail: stefaan.tavernier@vub.ac.be). biomarkers for pancreas and prostate. The result is an innovative endoscopic multimodal PET-US imaging system. This novel scanner aims to push current limitations of whole body PET detectors, making possible early detections of millimetric lesions. The standout characteristic of this system is its Region-Of-Interest (ROI) specific configuration, which breaks with the traditional whole body scheme thanks to the incorporation of a PET detector head in an endoscopic probe (see Fig. 1). Fig. 1. The EndoTOFPET-US scanner composed of an abdominal PET plate in coincidence with a miniaturized PET extension head on the tip of an ultrasound probe which is placed close to the ROI (in the duodenum in this case which is the closest accessible region with respect to the pancreas). Several technological challenges are addressed in the design of this novel scanner: 1 mm image spatial resolution, unprecedented 200 ps Coincidence Time Resolution (CTR) for enhanced background rejection, online tracking of both detectors and image reconstruction with partial volume information from an asymmetric geometry [2] [3]. A system with such characteristics demands a highperformance application-specific Data Acquisition (DAQ) system able to configure, control and readout data simultaneously from two different detectors. II. DESIGNING A DAQ FOR ENDOSCOPIC PET The EndoTOFPET-US DAQ [4] [5], can be decomposed in three different subsystems attending to their functionalities and location, as it is seen in Fig. 2: 1. 2. 3. On-detector plate electronics, this submodule implements the segment of the DAQ contained in the abdominal PET detector plate; On-detector probe electronics, this submodule corresponds with the segment of the DAQ contained in the endoscope extension PET detector probe; Off-detector electronics, which refers to the segment of the DAQ contained in the image reconstruction workstation. Fig. 2. DAQ system architecture. Top left picture shows the DAQ ondetector plate electronics making emphasis on the different components, its arrangement, link rates and triggering stages. On a similar way, same characteristics are depicted for the on-detector probe electronics in the top right of the figure and off-detector electronics in the bottom. Thanks to the approach of DAQ distribution over three subsystems, global complexity is balanced and it gives the possibility to perform and manage centrally a multi-level triggering scheme. As part of the multi-level triggering design, data rate is progressively reduced from the single gamma event detection to the actual coincidence information required by the image reconstruction software. Therefore, the required bandwidth in the interfaces between the different system components is considerably reduced and the several triggering levels can be independently configured for fine adjustment in order to: reject gamma events detection produced due to noise on one hand and to avoid rejection of meaningful but non purely time coincident gamma events on the other hand. Moreover, in an asymmetric system topology like the one of EndoTOFPET-US, where the two detectors present disparate sensitivities, the detectors readout has to face completely unbalanced data input streams and thus specific needs are required in terms of gamma events buffering, shorting and temporal coincidence classification. III. EXTERNAL PLATE The EndoTOFPET-US external plate is a pixelated PET detector panel placed over the patient’s abdomen. It is composed of a total of 256 matrices, with 4x4 of LYSO:Ce scintillating crystals per matrix, each of which is coupled to a coupled to a Hamamatsu TSV-MPPC (S12643-050CN) array of 4x4 Silicon Photomultiplier (SiPM) pixels. A modular approach is chosen to implement the plate electronics, leading to the division of the panel into four independent submodules. Each of the 4 quarters contains a Front-End-Board (FEB) called FEB/D (D, because it interfaces with the DAQ), which in turn are connected to 8 FEB/A each (A, because it contains two ASICs). As a result, each FEB/A connects 8 MPPC arrays and thus the FEB/D FPGA (a Xilinx Kintex 7) reads a total of 1024 MPPC channels. MPPC arrays readout is performed by means of a low power, low noise and high bandwidth ASICs with 64-channel implemented by the consortium. The ASIC is able to trigger on the first photoelectron, while rejecting dark counts (L0Plate Trigger Stage), at a maximum rate of 160 Kevents/s per channel. The ASIC digital interface encapsulates all the accepted gamma events in a digital frame. Each frame assembles events captured through its 64 channels within 6.4 µs time windows. For each event the ASIC includes energy and time information. For EndoTOFPET-US application, where the Field-of-View (FoV) is so small and detector sensitivity is so low, special emphasis has been placed on the optimization of timing resolution with the goal of improving the quality of the final image. For this reason, the ASIC has been designed to be able to provide gamma events fine time information with a 50 ps time binning. Taking into account the risk of using a novel technology for MPPCs readout, two different ASICs have been implemented by the collaboration in parallel to later decide which one is going to be used in the final system prototype based on their performance under the system conditions: TOFPET ASIC [6] and STiC2 ASIC [7]. The DAQ has been designed to be flexible in the processing of the ASIC data for the two different models, FEB/A is chip specific and FEB/D is able to accept FEB/A_TOFPET or FEB/A_STiC boards by selecting the corresponding firmware. The FEB/D FPGA is responsible for the implementation of the functionalities needed for the DAQ segment tied to the On-detector plate electronics: 1. Concentration and retransmission of gamma events. Readout of 16 ASICs centralizing into a super-frame all gamma events detected on the associated 1024 MPPC channels during a 6.4 µs time window and its transmission to the off-detector DAQ electronics using Multi-Gigabit Transceivers (MGT); 2. System configuration. Retransmission of configuration commands from the off-detector DAQ to the different devices on the FEB/D and FEB/As (namely ASICs, high-voltage Digital to Analog Converters and temperature sensors); 3. System monitoring. Readout of FPGA internal parameters and FEB/D and FEB/As parameters to assess system performance online. The communication protocol implemented on the ASICFEB/D_FPGA interface provides point-to-point serial communication at a maximum data rate of 640 Mb/s. It is based on 8B/10B codification and it includes extra error detection mechanisms. On the other hand, the communication protocol chosen on the interface FEB/D_FPGA-DAQ_FPGA is based on the AURORA 8B/10B protocol (Xilinx, 2100 Logic Drive San Jose, CA) working at 1.6 Gb/s over a HDMI physical link. The implementation includes extra error detection mechanisms (L1-Plate Trigger Stage). The overall resulting event transmission rate goes up to 64/128 Mevents/s for full/compact event format respectively. The external plate electronics and their corresponding segment of the DAQ have been validated and characterized with the experimental setup shown in Fig. 3. This setup uses the following modules: 1. The final prototype of the off-detector DAQ card; 2. A TOFPET ASIC test board implemented for the characterization of this chip; 3. A ML605 Xilinx Virtex6 development kit including a custom made mezzanine board that provides 4 microHDMI IOs through the kit FMC connector. Connecting boards 2 and 3 by a FMC2FMC flex connector together they conform a FEB/D + 1xFEB/A equivalent capable of validating the configuration and readout of two ASICs (readout of 128 MPPCs channels). Fig. 3. External plate test setup. Top right corner of the figure shows the final prototype of the off-detector DAQ card making use of the 4 HDMI IOs simultaneously to communicate with the FEB/D + 1xFEB/A equivalent conformed by the Xilinx Virtex6 development kit shown in the bottom left of the picture and the TOFPET ASIC test board in the center of the picture. The reason for using a development kit instead of real FEB/D + FEB/As prototypes to conduct these measurements is that, after ASIC production, ASIC characterization on TOFPET ASIC test board is needed for an optimum design of FEB/D and FEB/A boards. Therefore, we had an available experimental setup for DAQ validation before FEB/D and FEB/As production, which is used for ASIC characterization and DAQ validation in parallel. Off-detector DAQ electronics accepts up to four links through HDMI connections. Experimental tests making use of the four present links transmitting simultaneously from the ondetector to the off-detector electronics, demonstrate that 1.6 Gb/s can be used simultaneously on each link maintaining an error free data transmission during 24 hours continuous acquisitions. For this link quality measure, a specific firmware version has been implemented for the Virtex6 where 4 MGTs are instantiated to externally loopback the information transmitted by the four MGTs present on the off-detector DAQ card where (Bit Error Rate) BER is computed. One last import remark about the On-detector DAQ electronics is that it has been designed under the premise of flexibility and reusability. Therefore, the FEB/D firmware accepts two different system topologies for system integration ease: either each FEB/D is connected independently to the DAQ or FEB/Ds are paired in a master-slave configuration reducing the number of physical links between the on-detector plate electronics and the off-detector DAQ to two HDMI cables. In a similar way, for future applications of this DAQ, FEB/Ds can be daisy-chained to increase the number of channels in the detector. IV. INTERNAL PROBE The EndoTOFPET-US internal probe is a miniaturized PET detector head extension at the tip of an ultrasound probe meant to be placed close to the ROI. In its larger version, for prostate exams, it is composed of 2 (18x9 fibers) LYSO:Ce scintillating crystal matrices. And in its smaller version, for pancreas exams, it comprises one of these matrices. Electronics in the endoscopic probe requires heavy miniaturization, and thus a compact solution for the photodetection and data processing has been implemented by the collaboration [8], which is named Multi-Digital Silicon Photomultiplier (MD-SiPM). This device presents the same granularity as the probe crystal matrix providing a 1 to 1 fiberchannel coupling and it offers single SPAD readout, while rejecting dark counts (L0-Probe Trigger Stage), for each of its 162 channels. Within the same device, it collects and encapsulates into a digital frame all the gamma events detected during a 6.4 µs time window. For each event, it provides gamma events’ time and energy information. These, digital frames are transmitted to a small and ultra low power ice40 FPGA (Lattice Semiconductor, Moore Ct, Hillsboro USA). This FPGA processes and filters MD-SiPM data (L1Probe Trigger Stage) on the on-detector probe electronics sending events towards the off-detector DAQ electronics with a rate up to 320kHz through a LVDS pair contained in the DVI cable that connects the off-detector DAQ with the probe. This data path from probe to off-detector DAQ card is shared for the transmission of both gamma events data and system monitoring/control data, thus reducing the number of links between on-detector and off-detector DAQ segments. The DAQ-probe communication protocol has been designed to be a scalable, lightweight, link-layer protocol whose main objectives are: 1. Move gamma events data from the probe FPGA to the DAQ across one or more serial lanes; 2. Send configuration commands from PC to the probe FPGA via DAQ across a serial lane; 3. Monitor and control communication’s parameters. The resulting communication protocol provides a reliable communication link between probe and DAQ card thanks to data integrity features such as: error detection, DC balanced transmission, non discrete spectrum, clock recovery, data alignment, data encoding and devices synchronization. Additionally, the protocol is independent of data packet content for fast processing and flexibility. The probe MD-SiPM and its corresponding segment of the DAQ has been validated and characterized using the experimental test setup shown in Fig. 4. This setup uses the following modules: 1. The final prototype of the off-detector DAQ card; 2. A MD-SiPM test board implemented for testing and characterization of the chip; 3. A ML507 Xilinx Virtex5 development kit. Connecting boards 2 and 3 through the kit expansion headers, together they conform a complete probe equivalent. MD-SiPM board has an aperture on the backside (covered by a black tissue in this picture) to allow chip characterization with laser or crystals + radioactive source. However, this particular setup has been used to perform electrical characterization and thus gamma events are triggered in the chip by an external electrical triggering pulse sent from the ML605 FEB equivalent. Fig. 4. Probe test setup. The right part of the picture shows the detectors IOs front panel of the Off-Detector DAQ card inside the image reconstruction workstation. It is connected through a DVI cable containing LVDS pairs to provide a CLK for the probe hardware, a synchronization signal and a data path for the readout of the MD-SiPM. The picture in the left side of the figure shows the probe equivalent hardware composed of a Xilinx Virtex5 development kit (left side green box) and by a prototype of the MD-SiPM (right side green box). The picture shows as well the electrical trigger link coming from the ML605 FEB equivalent kit. In a similar way as for the case of the external plate electronics, for the internal probe electronics we have a first prototype used for chip characterization which is needed for optimum design of the final prototype electronics and which can be used at this stage for DAQ validation as well. Therefore, this setup was used to carry out experimental tests first to validate the DAQ segment tied to the internal probe and then to get link quality measures. In the latter ones we have demonstrated that, operating the link at 320 Mb/s, error free data transmission can be achieved during 24 hours continuous acquisitions. In order to perform these measurements, a specific firmware version was implemented for the off-detector DAQ card to produce MD-SiPM like data towards the Xilinx Virtex5 kit that processes this data (in the same way as if it were real MD-SiPM data) and send it back to the off-detector DAQ card where BER is computed. By means of this setup in combination with the FEB setup depicted in section III, synchronization of the two detectors with the DAQ has been tested. The procedure to test detectors synchronization by means of electrical triggering was the following: 1. Off-detector DAQ electronics sends a synchronization signal to the two detectors; 2. Each detector resets its internal counters upon reception of synchronization signal to set time 0 to the same time instant in both detectors; 3. Events are electrically triggered by means of an external test pulse, which in this setup comes from the FEB equivalent ML605 development kit; 4. Synchronization is checked by observation of a constant difference in the time stamps of the events coming from the two detectors. V. OFF-DETECTOR ELECTRONICS Off-detector DAQ electronics are implemented in a PCIe enabled board integrated within the image reconstruction workstation (see Fig. 5). Fig. 5. Off-detector DAQ card. In the bottom of the picture it can be noticed the PCIe x4 connector. It can be seen on the left side of the picture (DAQ card front panel) how the probe IO interface is implemented as a mezzanine board allowing for the off-detector DAQ card HDMI exclusive use for future applications. The board is connected to the FEB/Ds through 4 HDMI fast point-to-point links and to the endoscopic probe over a DVI connection. On each of these detectors interfaces (either for a FEB/D or for the probe) the off-detector DAQ electronics provides: • A reference clock. • A synchronization signal. A dedicated data path for the transmission of configuration commands from the off-detector DAQ to the on-detector electronics. • A shared data path from the on-detector electronics towards the off-detector DAQ, which is shared between frequent gamma events data and spare system monitoring/control data. The interface PC - DAQ board is based on a PCIe motherboard interconnection and it has been proven to provide error free data transmission at 4 Gb/s as experimental tests confirm. The procedure to acquire this quality measure was the following: 1. Off-detector DAQ card internally generates data packets with the same format as in a real detector readout scenario; 2. Generated data packets are processed and transferred to the PC via the PCIe interface; 3. PC DAQ readout software reads the generated data packets and check data integrity. A Xilinx Virtex-4 FPGA has been chosen to implement the functionality required for the off-detector DAQ electronics. This FPGA is responsible for: 1. Parallel and asymmetric readout of probe and plate data; 2. Configuration of the different on-detector electronics; 3. Parallel online monitoring of the different on-detector electronics; 4. Temporal coincidence classification of the gamma events and retransmission towards the image reconstruction workstation when DAQ is set to work on coincidences operating mode; 5. Merging/shorting of the gamma events coming from the different detectors and its retransmission towards the image reconstruction workstation when DAQ is set to work on singles operating mode. For the parallel readout of the two detectors data, special considerations need to be taken into account. Not only in terms of asymmetric buffering capabilities and data rate adjustments, but also a symmetric error handling mechanism needs to be implemented in order to preserve detector uniformity. Namely, if a frame coming from a certain detector and containing the gamma events detected during the 6.4 µs time window is lost, then all the incoming gamma events from the other detectors within the same time window have to be discarded in order to avoid introducing artifacts in the image reconstruction algorithm. According to the philosophy of this distributed and multilevel system, the gamma events temporal coincidence classification is performed in two folds. First, an early coarse classification stage is performed online in the off-detector DAQ FPGA, which classifies as temporal coincident all events lying within the same 12.5 ns time window (L2 Trigger Stage). Thanks to this first classification stage, off-detector DAQ electronics can procure a significantly reduced data rate towards the image reconstruction workstation. The final and fine stage of temporal coincidence classification is performed later on by the image • View publication stats reconstruction workstation software (L3 Trigger Stage). The main benefit of separating coincidence filtering in two stages is that we ensure that only meaningful gamma events data will reach the reconstruction algorithms, i.e. gamma events derived from Compton or optical cross-talk effects will pass the first filtering stage while totally uncorrelated gamma events will not and afterwards the image reconstruction software package will be able to process them according to its nature. VI. CONCLUSIONS An Asymmetric Data Acquisition system specific for an endoscopic PET-US scanner has been designed, implemented and tested. The system is capable to configure, to monitor on real-time and to perform the readout of two different detectors, which lead to an asymmetric readout scenario. The presented DAQ is distributed and balanced across the different system components giving us the opportunity to implement a multi-level triggering scheme that allows fine central adjustment and progressively reduces data rate preventing the system to filter out meaningful data which is highly precious in this scanner scenario where FoV is limited and sensitivity is so low. Due to the design intrinsic system flexibility/expandability, this DAQ could be easily scalable and its compatibility on future applications (such as small animals scanners or particle therapy online radiation monitoring) is guaranteed. As a proof of concept the system has been successfully tested by means of detectors and DAQ card prototypes leading us to the conclusion that a distributed and multi-level triggering DAQ system is suitable for endoscopic PET detectors. REFERENCES [1] EndoTOFPET-US Proposal: “Novel multimodal endoscopic probes for simultaneous PET/ultrasound imaging for image guided interventions”, FP7/2007-2013 under Grant Agreement No. 256984. [2] Aubry, N. et al. (EndoTOFPET-US Collaboration), “EndoTOFPET-US: A novel multimodal tool for endoscopy and positron emission tomography”, Journal of Instrumentation Volume 8, Issue 4, April 2013, Article number C04002DOI: 10.1088/1748-0221/8/04/C04002. [3] C. Zorraquino on behalf of EndoOFPE-US collaboration, “EndoTOFPET-US a High Resolution Endoscopic PET-US Scanner used for Pancreatic and Prostatic Clinical Exam”, Mediterranean Conference on Medical and Biological Engineering and Computing, 2013. [4] R. Bugalho, C. Zorraquino Gastón et al., “EndoTOFPET-US Data Acquisition System”, Journal of Instrumentation Volume 8, Issue 2, February 2013, Article number C02049. DOI: 10.1088/17480221/8/02/C02049. [5] C. Zorraquino et al., “EndoTOFPET-US DAQ, designing the Data Acquisition System of a High Resolution Endoscopic PET-US Detector”, NSS/MIC 2013 IEEE. [6] M. D. Rolo et al., "A 64-Channel ASIC for TOFPET Applications", 2013 JINST 8 C02050. [7] T. Harion, W. Shen, K. Briggl, H. Chen, A. Gil, V. Stankova, and H.-C. Schultz-Coulon, “STiC2 - Characterization Results of a SiPM, Readout ASIC for Time-of-Flight Applications”, NSS/MIC 2013 IEEE. [8] S. Mandai and E. Charbon, “Multi-Channel Digital SiPMs: Concept, Analysis and Implementation”, NSS/MIC 2012 IEEE.