You are right that the GPU memory usage would depend on the data on the GPU only and should not be influenced by a dataset in the host RAM.
Could you post a minimal, executable code snippet showing this behavior with a random dataset, so that we could debug it, please?