Could you flip the order of eval_scales, i,e. eval_scales = (2, 1.5, 1.25, 1, 0.75, 0.5) and check the memory usage again? Maybe you see some memory fragmentation, which could be avoided by allocating the largest memory block first?