Speaker
Roberto AMMENDOLA
(INFN Roma Tor Vergata)
Description
apeNET is a seven-years old project aimed at the acceleration of numerical simulations, mainly Lattice QCD, on commodity clusters of O(1000) nodes using a custom-designed 3D toroidal interconnect. In this paper we report on apeNET+, the new generation of our network adapters supporting wire speeds up to 34 Gbit/s per link, PCIe x8 gen 2, improved hardware RDMA support and enhanced communication primitives.
The project target is the development of a low latency, high bandwidth direct network, i.e. without external switching hardware, matching the wire performance of the commercial counter-parts and improving on the price/performance ratio when the cluster size increases.
apeNET is a network of point-to-point links with a 3D toroidal topology, where each processing node is directly connected to its first neighbours by six bi-directional full duplex links. We adopted a packet-based communication protocol with wormhole dimension-ordered routing and virtual channels. The network interface provides hardware support for the RDMA programming model.
A Linux kernel driver, a set of low-level RDMA APIs and a OpenMPI library driver are available, enabling painless porting of standard applications.
Preliminary electrical measurements are provided and status update of the hardware developments are reported.
Primary author
Roberto AMMENDOLA
(INFN Roma Tor Vergata)
Co-authors
Dr
Alessandro LONARDO
(INFN Roma)
Dr
Andrea BIAGIONI Andrea
(INFN Roma)
Dr
Andrea SALAMON
(INFN Roma Tor Vergata)
Dr
Davide ROSSETTI
(INFN Roma)
Dr
Francesca LO CICERO
(INFN Roma)
Dr
Francesco SIMULA
(University of Rome and INFN)
Dr
Gaetano SALINA
(INFN Roma Tor Vergata)
Mr
Giacomo CHIODI
(INFN Roma)
Dr
Laura TOSORATTO
(INFN Roma)
Dr
Nazario TANTALO
(INFN Roma Tor Vergata)
Dr
Ottorino FREZZA
(INFN Roma)
Dr
Pier PAOLUCCI
(INFN Roma)
Dr
Piero VICINI
(INFN Roma)
Mr
Riccardo LUNADEI
(INFN Roma)
Prof.
Roberto PETRONZIO
(University of Rome Tor Vergata and INFN)
