A Highly Efficient Data Locality Aware Task Scheduler for Cloud-Based Systems.
Abstract
Scheduling tasks in the vicinity of stored data can significantly diminish network traffic. Scheduling optimisation can improve data locality by attempting to locate a task and its related data on the same node. Existing schedulers tend to ignore overhead and tradeoff between data transfer and task placement, and bandwidth consumption, by only emphasising data locality without considering other factors. We present a novel data locality aware scheduler for balancing time consumption and network bandwidth traffic - DLAforBT - to improve data locality for tasks and throughput, with the optimal placement policy exhibiting a threshold-based structure. DLAforBT uses bipartite graph modelling to represent data placement, adopts a judgment mechanism and a precise prediction model to determine moving data or moving computation. It integrates an improved Dominant Resource Fairness (DRF) resource allocation to capture tenants' resource allocation and run as many jobs as possible. DLAforBT improves by 16% of data locality rate, and 25% of throughput. Index Terms-data locality, multi-tenancy, scheduling, bipartite graph modelling, cloud computing I. I NTRODUCTION With the increase in volumes of data, cloud computing has become more important for large-scale data intensive applications, e.g. search engines and online map services. These applications not only process but also generate very large amount of data, which brings more complicated challenges in the issues of data locality, computation placement and task scheduling. Bandwidth per node within a rack is much higher than bandwidth per node between racks. Avoiding off-switch data exchange is critical for cloud's performance. Putting a task near its required data can reduce network traffic and cost. To enhance data locality for clouds, there has been much research into scheduling tasks close to their data [1]-[4]. BOLAS in [2] models the scheduling process as a bipartitegraph matching problem to assign data block to the nearest task. Work in [3] studies the data locality problem by utilising data migration and hotspot file prediction for lowering task execution waiting delay. Such schedulers greedily search for a task and all