I also tried to use .share_memory_() on the torch tensors of my data (as suggested by @apaszke and @trypag)  and then access themin the getitem method. This works and is very fast when the dataset is small but when increasing the dataset size at one point I get the following error which I -again- can’t understand. I still have plenty of ram free so this shouldn’t be the issue.
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\TestUser\Anaconda3\envs\py3_pt3_orig\lib\multiprocessing\spawn.py", line 106, in spawn_main
    exitcode = _main(fd)
  File "C:\Users\TestUser\Anaconda3\envs\py3_pt3_orig\lib\multiprocessing\spawn.py", line 116, in _main
    self = pickle.load(from_parent)
  File "C:\Users\TestUser\Anaconda3\envs\py3_pt3_orig\lib\site-packages\torch\multiprocessing\reductions.py", line 86, in rebuild_storage_filename
    storage = cls._new_shared_filename(manager, handle, size)
RuntimeError: Couldn't map view of shared file <torch_12164_1041449738>, error code: <87> at ..\src\TH\THAllocator.c:226

Could it be that nearly all my problems are windows related and I should just switch to linux at one point?