Such a strange problem!  I put code in the Dataset to watch index 0, as all indexes in the cache are being affected.  With the first case, where we load cache with num_workers=0, everything is fine, loading the cache and then reading it back out.  With the second case, where num_workers=1 when loading the cache, this is what happens:
Current index 0, value of cache index 0 object is 35830685
Current index 1, value of cache index 0 object is 35830685
Current index 2, value of cache index 0 object is 35830685
Current index 3, value of cache index 0 object is 35830685
.
.
.
.
Current index 96, value of cache index 0 object is 35830685
Current index 97, value of cache index 0 object is 35830685
Current index 98, value of cache index 0 object is 35830685
Current index 99, value of cache index 0 object is 35830685
(now reading back out)
Current index 0, value of cache index 0 object is 0
Current index 1, value of cache index 0 object is 0
Current index 2, value of cache index 0 object is 0
Current index 3, value of cache index 0 object is 0
Current index 4, value of cache index 0 object is 0
.
.
.
.

So during the time the cache is being loaded, that index is not affected.  But it must get hosed when the load finishes? Because by the time you goto read it back its already messed up.