Krill: A Compiler and Runtime System for Concurrent Graph Processing

Authors: Hongzheng Chen, Minghua Shen, Nong Xiao, and Yutong Lu (Sun Yat-sen University, Guangzhou, China)

Abstract: As a large number of emerging graph applications spread across different domains, the need for processing massive concurrent graph jobs (CGJs) is increasing. However, existing graph processing systems designed for a single job cannot efficiently tackle multiple CGJs, where they suffer from interfering memory access patterns and inefficient property management. In this paper, we introduce Krill, a compiler and runtime system for processing concurrent graph jobs. In the compiler, we propose leveraging the property buffer to easily write and manage property data. In the runtime system, we propose a novel technique named graph kernel fusion to reduce memory accesses, which fuses all the jobs and processes them as a whole. Experimental results show our system significantly reduces the number of memory accesses for CGJs by more than 6x compared with the baseline, and achieves up to 7.67x speedup with 3.84x shorter response latency compared with GraphM, the state-of-the-art graph system.


Presentation: file


Back to Technical Papers Archive Listing