Pinterest, Engineering, Monetization.
Consequently, the variance computed in step 1 becomes zero. While increasing the mini-batch size and shuffling at the row level can help mitigate this problem, they don’t fully solve it. To overcome the instability caused by Batch Norm, we at Pinterest have developed a solution called Resilient