Solved by JuanFMontesinos in post #6 
              
              
                To me it seems strange you are using softmax + mse loss. 
You are computing some sort of probabilities but then you use a cartesian distance? Strange 
Are you using the same dataset or a different one? You have convolutions with 512 filters which leads to a huge amount of a parameters for a 3D nn. B…