Solved by ptrblck in post #3 
              
              
                You won’t be able to convert 512 channels to 3 given the formula from the docs. 
input [N, L=C*factor**2, H_in, W_in]
output [N, C, H_out=H_in*factor, W_out=W_in*factor]

For your setup you would need to calculate: 
3 * factor**2 = 512
factor**2 = 512/3
factor = sqrt(512/3)
factor ~= 13.07 # not an …