Setting num_workers=0 actually showed the real problem which simply was :
AttributeError: 'SVHN_Dataset' object has no attribute 'is_train_split'
I simply had a misspelling and that was causing the issue. basically as noted here 244 and here 58 :

this error only occurs when you try to do multiprocessing on some code with errors in it.

hope this is useful for somebody out there!