RealGraphOF: A High-Performance Graph Engine for Very Large Graph Analysis

Abstract

Recently, single-machine-based graph engines, utilizing external storage within a single machine, have been studied extensively for efficient graph analysis. Existing studies, however, do not consider the situation where the graph data does not fit even the capacity of external storage, being stored in storages of multiple remote servers. In this case, loading parts of the graph along with transferring them over the network degrades the processing performance significantly. From this motivation, we propose RealGraphOF, an improved version of the original RealGraph, that processes large-scale real-world graphs efficiently by exploiting external storages in remote servers through NVMe-over-Fabrics. RealGraphOF employs (1) local storage caching to reduce expensive network transfers and (2) user-space/asynchronous IO to obtain higher IO bandwidth by issuing IO requests more frequently. Experimental results on real-world datasets show that RealGraphOF outperforms dramatically state-of-the-art graph engines including naive RealGraphOF