Solved by samarth-robo in post #2 
              
              
                Memory for network parameters: 
(256*5000 + 5000) * 4 * 2 = 10 Mbytes, where the factor of 2 is because the network has 1 tensor for weights and 1 tensor for gradients, and the additional 5000 is for biases. 
Memory for data: 
8192 * 512 * 4 * 2 = 32 Mbytes 
So by those rough calculations, the memor…