Not All Tasks Are Created Equal: Adaptive Resource Allocation for Heterogeneous Tasks in Dynamic Workflows

Workshop:WORKS21: 16th Workshop on Workflows in Support of Large-Scale Science

Authors: Thanh Son Phung and Douglas Thain (University of Notre Dame); Kyle Chard (University of Chicago, Argonne National Laboratory (ANL)); and Logan Ward (Argonne National Laboratory (ANL))

Abstract: Users running dynamic workflows in distributed systems usually have inadequate expertise to correctly size the allocation of resources (cores, memory, disk) to each task due to the difficulty in uncovering the obscure yet important correlation between tasks and their resource consumption. Thus, users typically pay little attention to this problem of allocation sizing and either simply apply an error-prone upper bound of resource allocation to all tasks, or delegate this responsibility to underlying distributed systems, resulting in substantial waste from allocated yet unused resources. In this paper, we will first show that tasks performing different work may have significantly different resource consumption. We will then show that exploiting the heterogeneity of tasks is a desirable way to reveal and predict the relationship between tasks and their resource consumption, reduce waste from resource mis-allocation, increase tasks’ consumption efficiency, and incentivize users’ cooperation. We have developed two info-aware allocation strategies capitalizing on this characteristic and will show their effectiveness through simulations on two modern applications with dynamic workflows and five synthetic datasets of resource consumption. Our results show that info-aware strategies can cut down up to 98.7% of the total waste incurred by a best-effort strategy, and increase the efficiency in resource consumption of each task on average anywhere up to 93.9%.

Back to WORKS21: 16th Workshop on Workflows in Support of Large-Scale Science Archive Listing

Back to Full Workshop Archive Listing