  In order to boost the performance of data-intensive computing on HPC systems,
in-memory computing frameworks, such as Apache Spark and Flink, use local DRAM
for data storage. Optimizing the memory allocation to data storage is critical
to delivering performance to traditional HPC compute jobs and throughput to
data-intensive applications sharing the HPC resources. Current practices that
statically configure in-memory storage may leave inadequate space for compute
jobs or lose the opportunity to utilize more available space for data-intensive
applications. In this paper, we explore techniques to dynamically adjust
in-memory storage and make the right amount of space for compute jobs. We have
developed a dynamic memory controller, DynIMS, which infers memory demands of
compute tasks online and employs a feedback-based control model to adapt the
capacity of in-memory storage. We test DynIMS using mixed HPCC and Spark
workloads on a HPC cluster. Experimental results show that DynIMS can achieve
up to 5X performance improvement compared to systems with static memory
allocations.

Performance;Distributed, Parallel, and Cluster Computing