iTurboGraph: Scaling and Automating Incremental Graph Analytics

Abstract

With the rise of streaming data for dynamic graphs, large-scale graph analytics meets a new requirement of Incremental Computation because the larger the graph, the higher the cost for updating the analytics results by re-execution. A dynamic graph consists of an initial graph G and graph mutation updates ΔG$ of edge insertions or deletions. Given a query Q, its results $Q(G)$, and updates for ΔG$ to G, incremental graph analytics computes updates ΔQ$ such that Q($G \cup ΔG)$ = $Q(G)$ $\cup$ ΔQ$ where $\cup$ is a union operator. In this paper, we consider the problem of large-scale incremental neighbor-centric graph analytics (\NGA ). We solve the limitations of previous systems: lack of usability due to the difficulties in programming incremental algorithms for \NGA and limited scalability and efficiency due to the overheads in maintaining intermediate results for graph traversals in \NGA. First, we propose a domain-specific language, ŁNGA, and develop its compiler for intuitive programming of \NGA, automatic query incrementalization, and query optimizations. Second, we define Graph Streaming Algebra as a theoretical foundation for scalable processing of incremental \NGA. We introduce a concept of Nested Graph Windows and model graph traversals as the generation of walk streams. Lastly, we present a system \SystemName, which efficiently processes incremental \NGA for large graphs. Comprehensive experiments show that it effectively avoids costly re-executions and efficiently updates the analytics results with reduced IO and computations.