RISE Seminar 1/31/20: Optimal Resource Allocation for Parallelizable Jobs, a talk by Ben Berg

January 31, 2020

Title: Optimal Resource Allocation for Parallelizable Jobs

Abstract:  Modern distributed computation frameworks have enabled the dynamic allocation of resources to parallelizable jobs.  When a job is parallelized across many servers, it will complete more quickly.   However, jobs typically receive diminishing returns from being allocated additional servers.  Hence, given a fixed number of servers, it is not obvious how to allocate servers across a set of jobs in order to minimize the overall mean response time.  A good allocation policy should favor shorter jobs,  but favoring any single job too heavily can cause the system to operate very inefficiently.   We derive the optimal allocation policy which minimizes mean response time across a set of jobs by balancing the trade-off between granting priority to short jobs and maintaining the overall efficiency of the system.

Bio:  Ben Berg is a fourth year Ph.D. student at Carnegie Mellon University where he is advised by Mor Harchol-Balter.