Hana Krichene, Mouna Baklouti, Mohamed Abid Philippe Marquet, Jean-Luc Dekeyser WSEAS TRANSACTIONS on CIRCUITS and SYSTEMS G-MPSoC: Generic Massively Parallel Architecture on FPGA HANA KRICHENE University of Lille 1 INRIA Lille Nord Europe Lille - France ENIS School CES Laboratory Sfax - Tunisia hana.krichene@inria.fr MOUNA BAKLOUTI MOHAMED ABID ENIS School CES Laboratory Sfax - Tunisia mouna.baklouti@enis.rnu.tn mohamed.abid@enis.rnu.tn PHILIPPE MARQUET JEAN-LUC DEKEYSER University of Lille 1 INRIA Lille Nord Europe Lille - France Philippe.Marquet@univ-lille1.fr jean-luc.dekeyser@univ-lille1.fr Abstract: Nowadays, recent intensive signal processing applications are evolving and are characterized by the diversity of algorithms (filtering, correlation, etc.) and their numerous parameters. Having a flexible and programmable system that adapts to changing and various characteristics of these applications reduces the design cost. In this context, we propose in this paper Generic Massively Parallel architecture (G-MPSoC). G-MPSoC is a System-on-Chip based on a grid of clusters of Hardware and Software Computation Elements with different size, performance, and complexity. It is composed of parametric IP-reused modules: processor, controller, accelerator, memory, interconnection network, etc. to build different architecture configurations. The generic structure of GMPSoC facilitates its adaptation to the intensive signal processing applications requirements. This paper presents G-MPSoC architecture and details its different components. The FPGA-based implementation and the experimental results validate the architectural model choice and show the effectiveness of this design. Key–Words: SoC, FPGA, MPP, Generic architecture, parallelism, IP-reused 1 Introduction cuit layout modifications. In this work, Xilinx Virtex6 ML605 board is used to implement the G-MPSoC architecture and to evaluate the experimental results. The remainder of the paper is structured as follows: Section 2 discusses some related works; section 3 describes the proposed architecture and its execution model; the implementation methodology on FPGA board is detailed in Section 4; then, the experimental results are discussed in Section 5; and finally, Section 6 concludes the paper and proposes some perspectives. The intensive signal processing applications are increasingly oriented to specialized hardware accelerators, which allow rapid treatment for specific tasks. However, these applications are characterized by repetitive tasks performing multiple data, which require massive parallelism for their efficient execution. To achieve high performance required by these applications, many massively parallel Systemon-Chips are proposed [1, 7, 5, 2]. Despite their effectiveness, these solutions are still dedicated to specific applications and it is generally difficult to make future changes to adapt new applications. To address this problem, this paper proposes a novel Generic Massively Parallel System-on-Chip, named G-MPSoC. This system, based on modular structure, allows building different configurations to cover a wide range of intensive signal processing applications. The architectural model of G-MPSoC can integrate software homogeneous computation units or hardware(accelerator)/software(processor) heterogeneous computation units. This generic feature allows the best partition of the specific tasks on the computational resources. To implement this system, the FPGA platform is targeted to exploit its reconfigurable structure, which facilitates the test of different G-MPSoC configurations with rapid re-design or cir- E-ISSN: 2224-266X 2 Related work Nowadays, the digital embedded systems migrate toward the massively parallel on-Chip design, due to its provided high performance. Among these systems, we note the General-Purpose Processing on Graphics Processing Units (GP-GPU) [2, 24], which is a massively parallel System-on-Chip based on a hybrid model between vector processing and hardware threading. It allows high performance by hiding the memory latency [1]. But, it loses efficiency with the large branch divergence between executing threads [3] when data dependency is demanded in intensive signal (image, sound, motion...) processing applications. In this gap, Platform 2012 (P2012) [1] is positioned 456 Volume 14, 2015 Hana Krichene, Mouna Baklouti, Mohamed Abid Philippe Marquet, Jean-Luc Dekeyser WSEAS TRANSACTIONS on CIRCUITS and SYSTEMS 3 with highly coupled building blocks based on cluster of extensible processors varying from 1 to 16 and shared the same memory. Despite its high scalability, this architecture still specialized for limited range of applications such as multi-modal sensor fusion and image understanding. To provide more flexibility, a heterogeneous extension to the P2012 platform is proposed with He-P2012 [6] architecture. In He-P2012, the clusters of PEs used in P2012 are tightly coupled with hardware accelerators. All of them share the same data memory. With this architecture, a programming model is proposed, allowing the dispatch of Hardware and Software tasks with the same delay. Enlarge the initial software platform by additional hardware blocks can increase the design complexity with multiple communication interfaces. Others architectures have adopted the same strategy of massively parallel shared memory architecture, such as STM STHORM [1], Kalray MPPA [7], Plurality HAL [8], the NVIDIA Fermi GPU [9], etc. Although their provided high performance, sharing the same memory unit can limit the system bandwidth and cause some data access congestion, which limit the system scalability. The autonomous control structure for massively parallel System-on-Chip is proposed with the MPPA [4] architecture and Event-Driven Massively Parallel Fine-Grained Processor Array [25], executing in MIMD fashion. They provide more flexibility with asynchronous execution than the previous centralized and synchronous architectures. But, the use of a completely decentralized processing structure makes the control task of data transfer between independent computation units difficult to achieve. Generic Massively Parallel Architecture based on Synchronous Communication Asynchronous Computation 3.1 G-MPSoC architecture overview The G-MPSoC architecture is composed of a Master Controller Unit (MCU) connected to its sequential instructions memory, called MCU-memory, and a grid of Slave Control Units (SCUs). Each SCU is connected to a cluster of 16 Computation Elements (CEs), known collectively as Node. The CE can be a Software Processing Element (PE) or a Hardware specialized Accelerator-IP. Each CE is connected to its local instructions and data memories, called Mi memory. The parameter i is relative to the CE number. The MCU and SCUs grid are connected through a bus with single hierarchical level and the SCUs are connected together through neighbourhood interconnection network. Fig. 1 shows the hardware implementation of the G-MPSoC. Bellow, we present the design of the G-MPSoC components. M0 M0 CE0 M15 M15 SCU (0,2) M0 M0 CE15 CE15 M0 CE0 M1 M1 CE1 M15 CE1 M15 SCU (2,1) CE15 M15 SCU (2,2) CE15 Nodes grid Figure 1: G-MPSoC architecture 3.1.1 MCU The MCU is the first execution and control level in G-MPSoC. It is a simple processor, which fetches and decodes program instructions, executes sequential instructions and broadcasts mask activity and parallel control instructions to SCUs. In a parallel execution, MCU remains in the idle state and controls the end signal to resume the main program execution. In GMPSoC, the MCU is based on modified FC16 [10], which is a stack processor with short 16-bits instructions. It is open source and fully implemented in VHDL language, which allows rapid prototyping in In the next section, we detail the major components of G-MPSoC. E-ISSN: 2224-266X CE15 M0 CE1 Network Connection M15 SCU (1,2) CE0 M1 CE15 CE1 M15 SCU (1,1) M0 SCU (2,0) M1 CE1 CE0 CE Connection CE0 M1 CE1 Memory Connection M0 CE0 M1 M15 To overcome the limits of the existing proposed massively parallel architectures on-chip, we define a Generic Massively Parallel SoC (G-MPSoC), allowing the execution of wide range of intensive signal processing applications. To meet the high performance requirements of these applications, this generic architecture can have different configurations going from simple homogeneous structure to clustered heterogeneous structure communicated through regular Network-on-Chip (NoC) and controlled by hierarchical master-slave control structure [14]. This design is implemented into FPGA platform, which allows rapid reconfiguration of the architecture according to the designer needs. This reconfigurability based on programmable logic elements facilitates the system scalability. M15 CE15 SCU (0,1) CE0 SCU (1,0) CE1 CE15 SCU (0,0) Global Control M1 CE1 CE15 MCU CE0 M1 CE1 MCU Memory M0 CE0 M1 457 Volume 14, 2015 Hana Krichene, Mouna Baklouti, Mohamed Abid Philippe Marquet, Jean-Luc Dekeyser WSEAS TRANSACTIONS on CIRCUITS and SYSTEMS FPGA with instructions set simple to expand. Some specific instructions are added to the FC16 instructions set, mainly: mains in idle state until the end of the parallel execution. It is a synchronising instruction, which depends on the end-execution signal value. • Mask instructions To activate the involved nodes in the parallel execution, the MCU executes mask instructions, as presented in table 1, and broadcasts the mask map and mask code to all SCUs. The mask is coded in 32 bits, to address an array of (16x16) nodes. The MCU is connected to the SCUs grid through a bidirectional bus to communicate the parallel control instructions from MCU to SCUs and the computation results from CEs to MCU via SCUs. The end signal connects all SCUs to inform the MCU of the end of the parallel execution. 3.1.2 SCUs array Table 1: Mask instructions selbf 0x0080 Mask Code (000) selbfand 0x0081 (001) selbfor 0x0082 (010) selbfxor 0x0083 (011) Definition The second execution and control level in G-MPSoC is presented by the SCUs array. Each SCU controls: the local node activity, the autonomous parallel execution in the CEs, the end execution and the synchronous communication in the interconnection network. As shown in fig. 2, it is composed of four modules, where each one of them independently performs a specific control function, as detailed in [14]. Activate SCUs selected by the mask. Activate SCUs in the intersection of the current mask and previous one. Activate SCUs in the union of the current mask and previous one. Activate SCUs in the union of the current mask and previous one except the intersection part. ortree end SCU_ORTREE ce_inst CMD(15:0) scu_inst SCU_ Activity data_rce ISCU(15:0) Local_Control scu_code go scu_data brdbfb 0x0085 (101) brdall 0x0086 (111) xnet_a SLCU_COM Figure 2: SCU design • SCU Activity module This module receives the mask activity and the parallel control instruction from the MCU. Then, according to the mask code and broadcast code, it sets the activity-flag and transfers the parallel control instruction to the Local Control module, respectively. We notice that the use of SCU Activity module in G-MPSoC architecture allows the sub-netting of the SCUs grid, which optimizes the data flow transfer and increases the parallel broadcast domains. Definition Broadcast parallel control instructions to active SCUs. Broadcast parallel control instructions to inactive SCUs. Broadcast parallel control instructions to all the SCUs. • Local Control module After sub-netting the network, the parallel control instructions are broadcast to a Local-Control • Wait end instruction When executing this instruction, the MCU re- E-ISSN: 2224-266X xnet_com SCU_ROUTER Network Interface 0x0084 COM A D SCU COM brdbf Mask Code (100) COM inst COM_Control Table 2: Mask instructions instruction OpCode A Dir • Broadcast instructions Coded in 32 bits, the broadcast instructions identify which area executes parallel control instructions (table 2). Once the mask is mapped into the SCUs grid and when the broadcast instruction is executed, the MCU fetches the parallel control instruction, and then broadcasts it to all selected SCUs followed by the broadcast code. r_w simd_scac E CE Interface MCU interface instruction OpCode 458 Volume 14, 2015 Hana Krichene, Mouna Baklouti, Mohamed Abid Philippe Marquet, Jean-Luc Dekeyser WSEAS TRANSACTIONS on CIRCUITS and SYSTEMS module. This module prepares the communication phase and controls the CEs execution. The main sub-module in Local-Control is the Instruction Decoder, which decodes the instructions received from the SCU Activity module. These instructions are coded in 32 bits, as detailed in [14]: the first 16 bits represent the control microinstruction (CMD) and the last 16 bits represent: the address of the parallel instructions block, the single parallel SIMD instruction (P INST) or the value of the communicated data. տ ↑ ր → ց ↓ ւ ← • SCU COM module The SCU components in G-MPSoC are connected in two-dimensional neighbourhood interconnection network via the SCU-COM module. It allows the SCU to communicate with its 8 neighbours using only 6 connections. The SCU-COM is composed of COM Control and SCU router sub-modules. The COM Control sub-module manages the data transfer according to the communication instructions. The SCU router sub-module itself is composed of two routers, as presented in the fig. 3. S Local E Table 3: X-net directions Direction Code R-SCUXnet North West 0 0 տ North 1 0 տ North East 2 1 ր East 3 1 ր South East 4 2 ց South 5 2 ց South West 6 3 ւ West 7 3 ւ R-Xnet 0 տ 1 ր 1 ր 2 ց 2 ց 3 ւ 3 ւ 0 տ Each SCU-Router can take 4 different directions, allowing the data transfer in 8 directions, as detailed in table 3. When the COM-Control decodes the communication instructions, it orders the COM-Router to open the selected ports of the couple (R-Xnet,RSCUXnet) to achieve the data transfer in the specified direction. A COM-Control handles the communication requests and transfers data from the local SCU to the neighbour SCU in a given direction. Once the communication is established, data will be stored in R COM register to be used by the requester CE. Direction DEMUX MUX • OR Tree module The barrier synchronization is a high latency operation in massively parallel systems. Several systems have implemented either dedicated barrier networks [12] or provided hardware support within existing data networks [13]. The ORTree is a mechanism of global OR, checking the state of the system parallel processing. It is composed of a tree of ”OR” gates, which compares the end execution signals of all the CEs in pairs. It is a barrier synchronization that allows the controllers to know if all activated CEs finished the computation. The G-MPSoC supports a hierarchical OR Tree structure. The first level is in the SCU component to test the end execution in cluster of CEs and the second one is in the SCUs grid to test the end execution in all nodes of the system. (a) (b) Figure 3: Architecture of the SCU-Router (a) R-SCUXnet - (b) R-Xnet All the communications take place in the same direction, so there is no messages congestion in the same data transfer port. This feature allows the simple design of the routing element: – R-SCUXnet manages directions and distances of communications. It is composed of a couple of 4:1 mux/demux, allowing synchronous data transfer. The modular structure of the SCU and the independent execution of the control functions allow the autonomous parallel computation while performing the parallel communication. – R-Xnet allows simultaneous connection of any pair of non-occupied ports according to the given direction. E-ISSN: 2224-266X 459 Volume 14, 2015 Hana Krichene, Mouna Baklouti, Mohamed Abid Philippe Marquet, Jean-Luc Dekeyser WSEAS TRANSACTIONS on CIRCUITS and SYSTEMS 3.1.3 X-net: neighbourhood interconnection network To maintain efficient execution in embedded systems and high performance through a selective broadcast, we propose an on-chip regular neighbourhood interconnection network inspired from the network used in MP-1 and MP-2 MasPar [22] machines: called Xnet Network-on-Chip. The X-net network uses various configurations according to the data-parallel algorithms needs. Therefore, we define different bus sizes (1 bit, 4 bits or 16 bits) and different network topologies 1D (linear and ring) and 2D (mesh and torus). To change from one configuration to another, the designer has to use specific parameters (topology and bus size values) to establish the appropriate connections. When the network parameters are selected, the X-net network is generated with the chosen configuration. To achieve the re-usability and reconfigurability, the X-net network directly connects each CE with its 8 nearest neighbours in bidirectional ports through the SCU component. In some cases, the extremity connections in X-net network are wrapped around to form a torus topology, which is used to facilitate the matrix computation algorithms. All SCUs have the same direction controls. In fact, each SCU can simultaneously send a data to the northern neighbour and can receive another data from its southern neighbour. The X-Net uses a bit-state signal to identify nodes that participate in communication. Inactive SCUs can be used as pipeline stages to achieve distant communication. This data transfer through networks occurs without conflicts and is achieved by COM S and COM R instructions that allow all the SCUs to communicate with their neighbours in a given direction at a given distance. cessor can be assigned to dedicated hardware modules, called IP accelerators. It allows executing a specific function more efficiently than with any processor. Regardless of the function to achieve, HW-CE is only sensitive to the trigger signal sent by the SCU controller. Once received, the HW-CE begins the execution, independently of the others CEs in the cluster. The external interface of the CE is the same whatever the nature of the component (HW-CE or SW-CE) to allow the rapid integration of the CE component in the G-MPSoC architecture. Each CE in the cluster is connected to its own data register R CEi located in SCU to store the intermediate results, needed for the communication process, and the final result that will be transmitted to the MCU. Each cluster of CEs is controlled by its SCU and only performs when receives the trigger signal. 3.1.5 Memories • Sequential memory: MCU-memory The MCU is connected to the program memory. It includes sequential instructions to be executed by MCU and parallel control instructions to be broadcast to SCUs. The data are stored in the stack of MCU processor. • Parallel distributed memories: Mi memory Each SW-CE is connected to its own local Mi memory. It is divided into an instructions memory, including parallel instructions blocks, and a data memory, including parallel data. Depending on the SW-CE nature, the designer chooses to include or not the data memory. For example, in the case where SW-CE is a stack processor, the use of data memory is not necessary. 3.1.4 Cluster of CEs The cluster of CEs is the third execution level in GMPSoC. It allows the parallel execution through either homogeneous CEs, using the same Software SW-CE, or heterogeneous CEs with different Software SW-CE and Hardware HW-CE. The G-MPSoC architecture is designed as parametric and reconfigurable system, which is able to target various applications through customized architecture. Indeed, the number of the nodes (SCUs, CEs and memory size) is parametric in G-MPSoC, allowing the easiest scalability of the system. All these nodes are connected via a reconfigurable interconnection network, which can have one topology among 4 (2D (mesh, torus) and 1D (linear, ring)), 1 direction among 8 (N, S, E, O, NE, NO, SE and SO) and 1 distance ranging between 1 and 16. To facilitate the modification and the reuse of the components that constitute the architecture, a generic implementation is defined, by which several G-MPSoC configurations can be built. To configure a G-MPSoC system from a top level design, the designer defines the entities of the modules that will be used in the system, • SW-CE: Processor Element (PE) Each PE executes its own instructions block independently from the others PEs. This autonomous execution requires the integration of a local instruction memory and a local Program Counter (PC) in each SW-CE. When the SCU orders the parallel computation, all active SWCEs start the execution of their local instructions blocks at the same address on different data. • HW-CE: IP Accelerator Some specific functions performed by the pro- E-ISSN: 2224-266X 460 Volume 14, 2015 Hana Krichene, Mouna Baklouti, Mohamed Abid Philippe Marquet, Jean-Luc Dekeyser WSEAS TRANSACTIONS on CIRCUITS and SYSTEMS specifies their parameters and interconnects them together. Indeed, G-MPSoC instance can range from a simple configuration composed of a MCU connected to a parametric grid of SCUs, where each SCU is connected to a single CE, to a complex configuration where SCUs are interconnected via a reconfigurable neighbourhood network and each SCU is connected to a cluster of heterogeneous CEs. Therefore, from a generic architecture a tailored solution can be built, according to the application requirements, in terms of resources: computation, memorization and communication. 3.2 network. The master-slave control mechanism provides a flexible parallel execution with the use of multiple control flows globally synchronized. This flexibility increases the system scalability. 4 The choice of the G-MPSoC implementation on FPGA is justified by the flexibility and the reconfigurability of this device. It allows the implementation of generic architecture that is effectively tailored to the application requirements. The number of nodes (SCU controllers + CEs calculators) and the local memories size are parametric. Indeed, a targeted application can need many SCUs with many heterogeneous CEs using short memories, or a small amount of SCUs and CEs with large memories. This architecture is implemented with VHDL language and targets the Xilinx Virtex6 ML605 FPGA [15]. To evaluate the GMPSoC performance, we use the ISim simulator and the ISE synthesis Xilinx tools. The G-MPSoC configuration includes a single MCU and a grid of nodes. Each node is a hierarchical unit that contains a SCU and its cluster of CEs. Each CE is another hierarchical unit that contains an IP-accelerator or a PE connected to its local memory. Another intermediate hierarchical level, connecting several SCUs, has been defined to facilitate the routing process. All processors, used in G-MPSoC architecture, have a pre-existent FPGA implementation (FC16 [10], HoMade [16]...). We add some signals to be adapted to the CE generic interface and some instructions like: the wait GO instruction, to wait for the trigger signal sent from the SCU, and the end CE instruction, to inform the SCU of the end of the parallel computation. MCU is a modified FC16 processor, where mask, broadcast and wait end instructions are added. Its interface with the SCUs grid is modified to support 32 bits buses and 3 bits mask/broadcast code bus. The IP-accelerator can be a predefined Xilinx IP or an implemented IP using the RTL description. The memories modules are implemented using the existing FPGA blocks memories. The connections between the components used in this design require several signals that consume a large area. To increase the system scalability, it is necessary to optimize the use of these signals without affecting the processing. We have proposed to break all final result connections from SCUs to MCU and we have only kept the closest signals to MCUs (i.e. the SCUs in the first column). For the others, shift operations are performed to transfer the final result to the SCUs in the first column. This method allows the decrease of the system area occupancy approximately Execution model of G-MPSoC 3.2.1 Synchronous Communication • One-to-all communication: broadcast with mask The broadcast with mask [11] is a technique of dividing a network into two or more subnetworks, where the execution of different instructions is autonomous. This mechanism starts by activating the SCUs involved in the execution according to the mask sent by the MCUs. Each SCU in the grid has a unique number composed of the reference couple (X,Y). This number represents its position relative to X line and Y column. The mask codes set the activity flag of each SCU. All active SCUs execute control instructions, while the others remain in an idle state. • Collective regular communication The communication in G-MPSoC is defined by the temporal and spatial regularity of its data transfer. It is regular, by which all the nodes have the same degree of connection with their neighbours and synchronously communicate in the same direction and distance. This regular communication is managed by Send and Receive instructions. It is synchronously performed from a single control flow and locally controlled by the slave controllers. Such communication is simple to design without the overhead of synchronization mechanism. 3.2.2 Asynchronous Computation The master-slave control mechanism [14] is based on two control levels: the first one (MCU) executes sequential instructions and sends parallel control instructions to the second control level: SCUs. The second level controls the parallel communication in the interconnection network and the parallel execution in the cluster of CEs. Each CE executes its instructions stream asynchronously from the others, while the SCUs manage the synchronous data transfer in the E-ISSN: 2224-266X G-MPSoC implementation 461 Volume 14, 2015 Hana Krichene, Mouna Baklouti, Mohamed Abid Philippe Marquet, Jean-Luc Dekeyser WSEAS TRANSACTIONS on CIRCUITS and SYSTEMS about 6.26%. 5 The Synthesis results predict the maximum frequency of a given configuration, ranged around 88.212MHz This frequency is relative to the used processors frequency (FC16 151.404MHz, HoMade 94.117MHz) and the longest critical path in the design. A good place&route of the designed components is necessary to reduce the length of this critical path and to accelerate the signal propagation. The generic feature of G-MPSoC is ensured not only by the modular and parametric structure of its different components, but also by its hierarchicaldistributed control, the diversity and the heterogeneity of its processing units, and the parametricity of its interconnection network. In this section and through several benchmarks, we validate the generic feature of the architecture and show the effectiveness of their implementation in terms of flexibility, scalability and high performance (execution time and bandwidth). To test these benchmarks, we designed the G-MPSoC architecture with VHDL language and targeted the Xilinx Virtex6-ML605 FPGA as prototyping platform. Table 4 gives the resources occupancy for the GMPSoC components on the targeted FPGA. Depending on the used CE size, G-MPSoC configuration can include 100 nodes with FC16 processor, 49 nodes with HoMade processor or more than 256 nodes with muladd accelerator. This number of component can be increased with the use of another FPGA generation with more provided hardware resources to reach the defined architecture with 256 nodes and 16 CEs per cluster (i.e. 4096 CEs). The table 4 shows the power consumption of these modules. For all of them, the power characteristics are reported as around 400mW. This low power consumption makes the choice of the integrated CE only depending on size and speed. For simple operation, it is better to integrate specialized accelerator IPs than processors, in order to raise the architecture size and to accelerate the parallel execution. Table 4 also shows that the control structure in G-MPSoC system does not clutter the occupied area. In particular, the SCU module contains four submodules that not require too much logic elements, as shown in table 5. Thereby, the total area-cost of GMPSoC system does not burst with the use of masterslave control structure. In fact, based on the work in [14], the experimental results show that for 100 nodes with clusters of a single CE (FC16 processor) connected to 4KB local memory, the grid of SCUs occupy about 38% of the total consumed on-chip logic area. For an array of 16 SCUs, it is around 16%. In fact, if the number of G-MPSoC nodes increases, the area occupancy linearly increases, but the incremental cost of adding SCU functionality to G-MPSoC control system quickly becomes small. 5.1 E-ISSN: 2224-266X LUTs 4 352 101 3 Benchmark 1: Red-Black checkerboard Master-slave control in G-MPSoC architecture is based on a hierarchical-distributed structure, which allows having several parallel processing areas selected by the sub-netting mechanism. Thereby, the MCU can manage these areas and can switch from one to another using different activity masks, during the execution of a parallel program. This flexibility to switch between the processing areas allows the simultaneous execution of both conditional structure blocks (if...then...else...). This feature facilitates the activity process and avoids having idle processing nodes when executing the conditional structure in massively parallel architecture. To highlight this feature and its impact on the proposed architecture performance, we tested the RedBlack checkerboard application with the broadcastwith-mask [11] and traditional one-to-all [19] methods. This application is used to solve partial differential equations (Laplace equation with Dirichlet boundary conditions, Poisson-Boltzmann Equation, etc), using massively parallel systems. It is based on the division of parallel processing nodes into red and black areas, where all the same colour area are performing the same instructions block. The code below is composed of a set of the added mask/broadcast instructions and a ”lit” FC16 instruction [10] to map the red-black mask into the processing grid, and then broadcast the parallel instruction to trigger the execution of the first instructions block. Using the inverted mask, the second area can be activated and the execution of the second instructions block can be triggered. The code of the broadcast-with-mask, detailed in listing 1, is implemented with only 6 instructions to subnet the (16 × 16) grid into red-black checkerboard as shown in fig. 4. Therefore, the nodes with the same colour have the same control flux and execute the same instructions blocks, independently of the others Table 5: SCU sub-modules: synthesis result on FPGA Virtex6 ML605 Sub-module Activity controller Parallel execution controller Communication controller End execution controller (ORTree) Experimental results Registers 2 80 50 0 462 Volume 14, 2015 Hana Krichene, Mouna Baklouti, Mohamed Abid Philippe Marquet, Jean-Luc Dekeyser WSEAS TRANSACTIONS on CIRCUITS and SYSTEMS Table 4: G-MPSoC components: synthesis result on FPGA Virtex 6 ML605 CE LUTs 1132 3560 26 1139 420 FC16 HoMade muladd accelerator MCU SCU Control module Slice registers 206 493 19 201 132 nodes in the neighbour sub-network. l i t 0xAAAAAAAA / / f i n d mask A ( 2 c y c l e s ) selbf / / s e n d mask A ( 1 c y c l e ) l i t 0 x55555555 / / f i n d mask B ( 2 c y c l e s ) selbfor / / s e n d mask B ( 1 c y c l e ) l i t 0 x06000010 / / find parallel control i n s t r u c t i o n (2 cycles ) brdbf / / send p a r a l l e l c o n t r o l i n s t r u c t i o n (1 cycle ) 0 1 0 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 0 Mask A Bandwidth (MB/s) 800 780 760 740 720 one-to-all model 700 broadcast-with-mask model 680 660 640 620 4 16 64 256 CEs Figure 5: Influence of broadcast models on bandwidth Fig. 5 shows that an increasing number of nodes integrated in the G-MPSoC architecture leads to a higher gap between the bandwidth provided by the broadcast-with-mask method and one-to-all method. This result shows the importance of broadcast-withmask technique in the massively parallel systems scalability, without causing bottlenecks. However, the traditional one-to-all method is recommended with small systems because of its efficiency in terms of area cost and bandwidth. Thereby, the designer has to make the right choice between the system size and the broadcast technique to ensure the high performance needed for the parallel execution. 1 Mask B Red/Black mask 5.2 Benchmark 2: Matrix multiplication The processing units (CEs) in the G-MPSoC architecture can have several forms. They can be homogenous with the same type of processors or heterogeneous with clusters of different processors or clusters of dif- Nodes grid Figure 4: Red-Black mask E-ISSN: 2224-266X Power(W) 0.462 0.509 0.436 0.483 0.536 820 0 0 Fmax(Mhz) 151.404 94.117 624.220 153.520 256.937 840 As shown in listing 1, the red-black mask can be rapidly mapped into a grid of (16 × 16) nodes in 6 clock cycles, unlike the one-to-all method [19], that needs several clock cycles to map this mask into the processing nodes. With the one-to-all method, there must be a relationship between identity and the activity bit to activate the nodes with peer identities and disable the ones with odd identities, or to enable nodes with identity lower than a specific value and disable the others. So that to map the Red-Black mask into a grid of (16 × 16) nodes, 16 operations are required: 8 for odds and 8 for peers, executed in 20 clock cycles. Thus, we notice that the broadcast-with-mask method allows the rapid grid sub-netting on several processing areas, twice shorter than with the traditional one-to-all method. 0 FPGA occupancy <1% 2% <1% <1% <1% The parallel instructions broadcast using these two different activity control methods requires different implementations of the control structure in GMPSoC. The synthesis result given by ISE tool [17] and presented in table 6 shows the large bandwidth (∼12% higher than the one-to-all) provided by the G-MPSoC using broadcast-with-mask despite the additional consumed logic elements. This can be explained by the fact that the proposed control structure for parallel activity and parallel broadcast management is based on local controllers (SCUs), which optimize the data flow transfer. Listing 1: Broadcast-with-mask code for Red-Black checkerboard 1 DSP48E1s 1 1 - 463 Volume 14, 2015 Hana Krichene, Mouna Baklouti, Mohamed Abid Philippe Marquet, Jean-Luc Dekeyser WSEAS TRANSACTIONS on CIRCUITS and SYSTEMS Table 6: Synthesis results on FPGA Virtex6 of G-MPSoC with different activity control methods Broadcast with mask Broadcast one-to-all LUTs 257747 54% 242768 51% FPGA occupancy Slice registers 410904% 410564% memories 102567% 102567% Performance Bandwidth(MB/s) 786 697 Fmax(Mhz) 205.999 182.846 Power(W) 0.421 0.447 G-MPSoC system with different directions and distances and using the Send/Receive instructions. This application is performed via the X-net network, using the neighbourhood/distant communications and the broadcast-with-mask method to facilitate the network sub-netting, as shown in fig. 6. ferent processors and accelerators. Each one performs a specific function or blocks of parallel instructions. To highlight the genericity and the flexibility of the CEs, we have tested the matrix multiplication example with two different G-MPSoC configurations with a grid of (8×8) nodes, where each node is composed of: • Conf.1: SCU + software CE (FC16 processor). • Conf.2: SCU + software CE (FC16 processor) + hardware CE (muladd accelerator to perform the multiplication and the addition operations). The matrix multiplication is one of the basic computational kernels in many data parallel applications. It is presented by the equation (1): cij = n X aik bkj avec 1 ≤ i, j ≤ n (1) k=1 Figure 6: Summing application steps To perform (8×8) matrix multiplication into system with (8×8) nodes, the execution needs 16 multiplications, 16 additions and 30 communications. The FC16 processor performs the multiplication in 19 clock cycles [10], whereas muladd accelerator performs the multiplication in one cycle. That is why the configuration based on clusters of (FC16 + muladd) is more efficient than the architecture based on only SW-CE (FC16). We notice that the conf.2 is the most suitable for matrix multiplication. The generic feature of G-MPSoC architectural model allows changing the system structure from a homogeneous configuration to a heterogeneous configuration according to the performance provided by the CEs and the algorithm requirements. The use of parametric modules significantly facilitates the generation of processing nodes with rapid modification of system configuration. Thereby, the G-MPSoC system is flexible, scalable and quickly adapts to applications changes. Performing the Summing application, in traditional SIMD system, needs several clock cycles, especially for node activity step and communication step. In addition, doing distant communication in [18] requires external link. Therefore, the use of the X-net network and the broadcast-with-mask mechanism improves the communication performance in G-MPSoC architecture, where X-net communication costs d (distance) cycles: the delay of data transfer between source and destination (1 cycle for neighbour communication). 200000 FPGA resources 175000 150000 125000 100000 75000 LUTs 50000 Registres 25000 0 4 16 64 256 4 mesh 16 64 torus 256 4 16 64 bus (16+1 bits) 5.3 4 16 64 256 torus bus (1bit) number of PEs (Topology/bus size) Benchmark 3: Parallel Summing To validate the flexible communication in G-MPSoC architecture, we chose the summing application [18] using a grid of 4×4 nodes. The aim of this application is to test the synchronous communication in E-ISSN: 2224-266X 256 mesh Figure 7: FPGA occupancy with different size of GMPSoC configurations 464 Volume 14, 2015 Hana Krichene, Mouna Baklouti, Mohamed Abid Philippe Marquet, Jean-Luc Dekeyser WSEAS TRANSACTIONS on CIRCUITS and SYSTEMS 600 5.4 550 450 Bandwidth (MB/s) Benchmark 4: FIR filter The digital Finite Impulse Response (FIR) [21] filters are widely used in digital signal processing to reduce certain undesired aspects. A FIR structure is described by the differential equation (2): 500 400 350 300 250 200 150 y(n) = 100 50 h(k) × x(n − k) (2) k=0 0 4 16 64 256 4 16 mesh 64 256 4 torus 16 64 256 4 mesh bus size (16+1) bits 16 64 256 It is a linear equation, where X represents the input signal and Y presents the output signal. The order of the filter is given by the parameter N, and H(k) represents the filter coefficients. The number of inputs and outputs data is equal to n. To emphasize the advantage of communication networks in the execution of data-parallel programs, we have implemented the FIR filter with two methods: 1st method: 2D configuration This method is inspired from the work presented in [20]. It is an implementation of FIR filter as defined in its equation (2). The system is composed of a MCU, a grid of (4×4) nodes (SCU + PE) and an X-net network in torus topology. We assume that the FIR system takes 16 H(k) parameters and 64 X(n) inputs. The algorithm is described as the following steps: torus bus size 1 bit CEs number Figure 8: Influence of CEs number and bus sizes on bandwidth Fig. 7 and fig. 8 present synthesis and bandwidth results on different system topologies and bus sizes. We note a compromise between area and bandwidth. Indeed, the configuration integrating the X-net interconnect with (16+1) bits buses gives efficient data transfer with large bandwidth, but it occupies a large chip area (2 times higher), as shown in fig. 7. In addition, if the number of the nodes is multiplied by a factor of 16, the bandwidth decreases by factor of 2 and the FPGA area increases by a factor of 4, which is an acceptable rate. 1. Data initialization: each H(k) is stored in PE(i,j) local data memory according to the following function: k = 4 × i + j and all X(n) values are stored in each PE memory. 300 250 Latency (cycle) N −1 X 200 2. Multiplication of all the inputs with H(k) in each PE(i,j). bus(16+1)bits 150 bus(4+1)bits 100 bus 1 bit 3. Communication: as shown in fig. 10, all PEs do West communication to send first multiplication element to their neighbour; then only PEs in the last column do North-communication. 50 0 0 2 6 distance 10 14 4. Addition of this new value with the second local multiplication value in each PE(i,j). Figure 9: Influence of buses size on communication delay 5. Repeat 4) until obtain all the results which will be stored in PE(0,0) memory. We have also tested the communication delay with the use of the different bus sizes. Fig. 9 shows that the communication time is 17 times higher with 1-bit bus than with 17-bits bus. Despite this tedious communication, 1-bit data transfer allows the use of relatively simple buses with low hardware cost, which increase system scalability. These previous experimental results show the efficiency of the integration of reconfigurable interconnection network in G-MPSoC. Depending on the application needs, the designer can select the network parameters, which guarantee the less communication delay. E-ISSN: 2224-266X To perform this FIR filter algorithm, (64×2) communications are required. The multiplication and the addition operations are performed in parallel. 2nd method: 1D configuration This method is inspired from the work presented in [23]. In this implementation, the system is composed of a MCU, 16 slaves and X-net network in linear topology. We also assume that the FIR system takes 16 H(k) parameters and 64 X(n) inputs. All H(k) are stored in MCU and sent to PE(i) when 465 Volume 14, 2015 Hana Krichene, Mouna Baklouti, Mohamed Abid Philippe Marquet, Jean-Luc Dekeyser WSEAS TRANSACTIONS on CIRCUITS and SYSTEMS PE 00 PE 01 PE 02 PE 03 PE 10 PE 11 PE 12 PE 13 PE 20 PE20 PE 21 PE 22 PE 23 PE 30 PE30 PE 31 PE31 PE 32 PE32 PE 33 PE33 Depending on the application needs, the designer can select the most appropriate network configuration to his system. The different topologies that can support the X-net offer diverse choice of 1D or 2D configurations as well as at the size of the interconnect bus, to ensure data transfer rapidly and with low cost. 6 This paper presents a new generation of massively parallel System-on-Chip based on generic structure, called G-MPSoC platform. It is a configurable system, composed of clusters of hardware and/or software CEs, locally controlled by a grid of SCUs and globally orchestrated by the MCU. All the CEs can communicate between each other via the SCUs components that are connected through regular X-net interconnection network. G-MPSoC architecture is entirely described in VHDL Language, in order to allow rapid prototyping and testing, using the synthesis and simulation tools. The execution model of GMPSoC is detailed to highlight the advantages of the synchronous communication, based on broadcast with mask structure and regular communication network, and asynchronous computation, based on master-slave control structure. An FPGA Hardware implementation of G-MPSoC platform is also presented in this paper and validated through several parallel applications. Different configurations are tested going from a simple homogeneous structure to a complex heterogeneous structure. All configurations tested different X-net network topologies, different data buses sizes and different memories sizes. In this work, we define a generic massively parallel system on chip that is able to be quickly adapted to the applications requirements. The next work is to define a new weakly coupled massively parallel execution model for G-MPSoC platform, based on Synchronous Communication and Asynchronous Computation. This model is classified between the synchronous centralized SIMD model and the asynchronous decentralized MIMD model. It takes advantages of these two models to allow more performance needed for the execution of nowadays intensive processing applications. Figure 10: FIR filter implementation in 2Dconfiguration Table 7: 16-order FIR with inputs implementation the inputs (n) Com execution time (cycle) on SIMD mode 8 16 64 G-MPSoC Method 1 G-MPSoC Method 2 ESCA 126 270 1134 72 144 576 255 351 1158 reconfSIMD on-chip 332 744 - they are needed in the algorithm. The input X(n) are shifted inter-slaves using West-communication, as presented in fig. 11. The n outputs are calculated using multiplication and addition instructions, in the alternative way. To do this algorithm, only 63 communications are required but the algorithm is not totally parallel. PE013 PE014 PE015 MCU PE00 PE01 PE02 PE03 PE012 0 0 0 0 0 0 0 x(0) h(15) 0 0 0 0 0 0 x(0) x(1) h(14) 0 0 0 0 0 x(0) x(1) x(2) h(13) 0 .. 0 .. 0 .. 0 .. x(0) .. x(1) .. x(2) .. x(3) .. h(12) .. x(0) .. x(1) .. x(2) .. x(3) .. x(12) .. x(13) .. x(14) .. x(15) .. h(3) x(47) x(48) x(49) x(50) x(59) x(60) x(61) x(62) h(1) h(2) h(0) Figure 11: FIR filter implementation in 1Dconfiguration Experimental results The experimental results in table 7 show the time needed for data transfer in FIR filter application. As expected, the G-MPSoC architecture allows more rapid processing than both reconfigurable SIMD architecture on-Chip [23] and ESCA [20] architectures. We deduce that G-MPSoC architecture based on linear interconnection topology is the most effective for the FIR application. E-ISSN: 2224-266X Conclusion References: [1] D. Melpignano, L. Benini, , E. Flamand, et al. Platform 2012, a many-core computing accelerator for embedded SoCs: performance evaluation of visual analytics applications. Proc. Int. Conf. Design Automation Conference, New York, USA, 2012, pp. 1137–1142. 466 Volume 14, 2015 Hana Krichene, Mouna Baklouti, Mohamed Abid Philippe Marquet, Jean-Luc Dekeyser WSEAS TRANSACTIONS on CIRCUITS and SYSTEMS [2] SIMD < SIMT < SMT: parallelism in NVIDIA GPUs. http://yosefk.com/blog/simd-simt-smtparallelism-in-nvidia-gpus.html [3] PU-based Image Analysis on Mobile Devices. http://arxiv.org/pdf/1112.3110v1.pdf. [4] D.B. Thomas, L. Howes and W. Luk. A Comparison of CPUs, GPUs, FPGAs, and Massively Parallel Processor Arrays for Random Number Generation. Proc. Int. Conf. Field Programmable Gate Arrays, New York, USA, 2009, pp. 63–72. [5] F. Hannig, V. Lari, S. Boppu, et al. Invasive Tightly-Coupled Processor Arrays: A DomainSpecific Architecture/Compiler Co-Design Approach, ACM transactions on Embedded Systems, 13, 2014, pp. 133:1-133:29. [6] F. Conti, C. Pilkington, A. Marongiu, et al. HeP2012: Architectural Heterogeneity Exploration on a Scalable Many-Core Platform. Proc. Int. Conf. Application-specific Systems, Architectures and Processors, Zurich, Swiss, 2014, pp. 114-120. [7] Many-core Kalray MPPA, http://www.kalray.eu [8] The HyperCore Processor, http://www.plurality.com/hypercore.html [9] Next Generation CUDA Compute Architecture: Fermi WhitePaper, http://i.dell.com/sites/doccontent/sharedcontent/data-sheets/en/Documents/NVIDIAFermi-Compute-Architecture-Whitepaperen.pdf [10] R.E. Haskell and D.M. Hanna, A VHDL Forth Core for FPGAs, Journal of Microprocessors and Microsystems, 29, 2009, pp. 115-125. [11] H. Krichene, M. Baklouti, Ph. Marquet, et al. Broadcast with mask on Massively Parallel Processing on a Chip. Proc. Int. Conf. High Performance Computing and Simulation, Madrid, Spain, 2012, pp. 275–280. [12] C.E. Leiserson, Z.S. Abuhamdeh, D.C. Douglas, et al. The network architecture of the connection machine CM-5, Journal of Parallel and Distributed Computing, 33, 1996, pp. 145-158. [13] S.L. Scott. Synchronization and communication in the T3E multiprocessor. Proc. Int. Conf. Architectural Support for Programming Languages and Operating Systems, New York, USA, 1996, pp. 26–36. [14] H. Krichene, M. Baklouti, Ph. Marquet, et al. Master-Slave Control structure for massively parallel System-on-Chip. Proc. Int. Conf. Euromicro Conference on Digital System Design, Santander, Spain, 2013, pp. 917–924. E-ISSN: 2224-266X [15] ML605 Hardware User Guide - UG534 (v1.8), http://www.xilinx.com/support/documentation/ boards and kits/ug534.pdf. [16] HoMade processor, https://sites.google.com/site/homadeguide/home. [17] Xilinx website, https://www.xilinx.com. [18] M. Leclercq and P.Y. Aquilanti, X-Net network for MPPSoC, Master thesis, University of Lille 1, 2006. [19] M. Baklouti, A rapid design method of a massively parallel System on Chip: from modeling to FPGA implementation. PhD thesis, University of Lille 1 & University of Sfax, 2010. [20] P. Chen, K. Dai, D. Wu, et al. Parallel Algorithms for FIR Computation Mapped to ESCA Architecture. Proc. Int. Conf. Information Engineering, Beidaihe, China, 2010, pp. 123–126. [21] S.M. Kuo and W.S. Gan, Digital Signal Processors Architectures, implementation and application. Prentice Hall, 2005. [22] J.R. Nickolls, The design of the MasPar MP1: a cost effective massively parallel computer, IEEE Computer Society International Conference, Compcon Spring, San Francisco, USA, 1990, pp. 25–28. [23] J. Andersson, M. Mohlin and A. Nilsson, A reconfigurable SIMD architecture on-chip. Master Thesis in Computer System Engineering - Technical Report, School of Information Science, Computer and Electrical Engineering - Halmstad University, 2006. [24] Sh. Raghav, A. Marongiu and D. Atienza, GPU Acceleration for Simulating Massively Parallel Many-Core Platforms. Journal of Parallel and Distributed Systems, 26, 2015, pp. 1336–1349. [25] D. Walsh and P. Dudek , An Event-Driven Massively Parallel Fine-Grained Processor Array. Proc. Int. Conf. Circuits and Systems (ISCAS), Lisbon, Portugal, 2015, pp. 1346–13496. 467 Volume 14, 2015