A Backpressure Mitigation Scheme in Distributed Stream Processing Engines

Abstract

There is a significant rise in the adaptation of streaming applications in the past decade by individuals researchers and organizations in both industry and academia. These applications are all based on the modern data stream processing systems that implement resource allocation and management in order to provide an uninterrupted track of queries over incoming input distributed data streams. More than a few stream processing engines exists to handle these distributed streaming applications. These distributed applications have open challenges like backpressure. In this paper, we introduce a backpressure mitigation mechanism for the distributed stream processing systems. The proposed backpressure mitigation technique is a generic one and is feasible to be implemented on top of a number of popular streaming frameworks. We use Flink as a testbed for this work and use its available APIs.