If you want to keep using pytorch as a generic thing then stop worrying about maximum performance or question about GPU consumption because this is not area for consumers.
Now if you want to increase your memory usage and have a package that can speeds up your training and better use your GPUs and memory consumption then you will need to start study that too.