An Efficient Topology Refining Scheme for Apache Flink

Abstract

In the past decade, there has been a boom in the volume of data and in the popularity of cloud applications with industry and academia keenly interested in big data analytics, streaming application, and social networking applications. This led to the emergence of real-time distributed stream processing systems such as Flink, Storm, Dataflow, and Samza. These systems process complex queries on streaming data sets to be distributed across multiple worker nodes in a cluster. Few of them provide adequate supports to adapt the topologies of stream processing tasks to changing input workload. We present an intelligent and efficient topology adjustment scheme which allow Flink framework to refine its topology on the basis of incoming workload. It is designed to increase the overall performance by making the refining of topology robust according to incoming workload streams on the fly, while maintaining SLA constraints. Apache Flink distributed processing engine is used as testbed in the paper. Our preliminary results indicate that the proposed system outperforms the existing default framework.