Navigator: Dynamic multi-kernel scheduling to improve GPU performance

Abstract

Efficient GPU resource-sharing between multiple kernels has recently been a critical factor on overall performance. While previous works mainly focused on how to allocate resources to two kernels, there has been limited amount of work on determining which workloads to concurrently execute among multiple workloads. Therefore, we first demonstrate on a real GPU system how the selection of concurrent workloads can have significant impact on overall performance. We then propose GPU Navigator - a lookup-table-based dynamic multi-kernel scheduler that maximizes overall performance through online profiling. Our evaluation shows that GPU Navigator outperforms a greedy policy by 29.3% on average.