  In the past few years, we have envisioned an increasing number of businesses
start driving by big data analytics, such as Amazon recommendations and Google
Advertisements. At the back-end side, the businesses are powered by big data
processing platforms to quickly extract information and make decisions. Running
on top of a computing cluster, those platforms utilize scheduling algorithms to
allocate resources. An efficient scheduler is crucial to the system performance
due to limited resources, e.g. CPU and Memory, and a large number of user
demands. However, besides requests from clients and current status of the
system, it has limited knowledge about execution length of the running jobs,
and incoming jobs' resource demands, which make assigning resources a
challenging task. If most of the resources are occupied by a long-running job,
other jobs will have to keep waiting until it releases them. This paper
presents a new scheduling strategy, named DRESS that particularly aims to
optimize the allocation among jobs with various demands. Specifically, it
classifies the jobs into two categories based on their requests, reserves a
portion of resources for each of category, and dynamically adjusts the reserved
ratio by monitoring the pending requests and estimating release patterns of
running jobs. The results demonstrate DRESS significantly reduces the
completion time for one category, up to 76.1% in our experiments, and in the
meanwhile, maintains a stable overall system performance.

Performance