Solved by JuanFMontesinos in post #4 
              
              
                it’s a difficult question in case of optical flow. 
First of all, depending on which net are you using normalization you need vary a lot. If you are using both displacement components you have to apply the same mapping for both, not to affect the vector. You could use just the magnitude. 
You should…