thanks for the update James.
Did you, by any chance, try to use zarr instead of hdf5?  Is is supposed to circumvent this very issue but I did not have the time to dig deeper into this, particularly since zarr does not work with hdf5 files directly but requires files to be stored with zarr as well.
Also, how does your GPU utilization look like with your code above? Do you spot any drops in workload after iterations?