Hierarchical recursive resource sharing for containerized applications

Abstract

Applications are increasingly containerized using techniques, such as LXC and Docker. Scientific workflow applications are no exception. In this paper, we address the problem of resource contention between concurrently running containerized scientific workflows. To this end, we design and implement Hierarchical Recursive Resource Sharing (HRRS), which structures multiple concurrent containers in a hierarchy that automatically and dynamically regulates their resource consumption based on their level/tier in the hierarchy. The hierarchy is recursively updated as the top-tier container completes its execution with the second-tier container becoming the top-tier container inheriting the resource consumption priority. We have evaluated the performance of HRRS using multiple large-scale scientific workflows containerized by Docker. The experimental results show the significant reduction of resource contention as evident in performance improvement of 49%, 160% and 18% compared with sequential execution, concurrent execution with fair resource share and execution with submission interval, respectively.