Thanks for the isolation advice. It did came down to the PIL Image opening using all the cores. I read that is a python issue more than PIL.
I had conda install pillow=6.2.1
And I did found a very similar issue open in github with Pillow:

  
      github.com/python-pillow/Pillow
  
  
    
  
	  
  

  
    
      Memory is slowly leaking 2
    

    
      
        opened 
        
          
        
        Jul 8, 2016
      
      


      
        
          
          dmpetrov
        
      
    
  



  Memory is slowly leaking when process many image files one by one.
Leak size:
100K images --> ~100MB
300K images --> ~200MB
1M images -->...



    Bug
    Memory
    NumPy
    Platform


  
  
    
    
  
  


I found a solution:
https://pillow.readthedocs.io/en/stable/installation.html?highlight=cpu 6
Environment variable:  MAX_CONCURRENCY=n . Pillow can use multiprocessing to build the extension. Setting  MAX_CONCURRENCY  sets the number of CPUs to use, or can disable parallel building by using a setting of 1. By default, it uses 4 CPUs, or if 4 are not available, as many as are present.
Also I did change all resizing and preprocessing outside torch.