Why does Group Normalization work? In their paper Group Normalization the author introduce GroupNorm(GN) as a replacement for BatchNorm. They show that LayerNorm(LN) and InstanceNorm(IN) are extreme cases of GN.
They also show that GN significantly outperforms LN and IN. My experiments also confirm this.
But I didn't find an intuition as to why that is the case. Why does GN which is the middle ground between LN and IN, outperforms them?


I don't think there's any formal reasoning as to why GN would outperform both LN and IN. So this is all based on my intuition.
The difference in LN, GN, and IN is essentially the number of groups you use to perform normalization over channel features. For LN, the number of groups is 1; and for IN, the number of groups is the number of input channels.
So in some sense, LN may not sufficiently allow for separated normalization of different "kinds" of channel features (say some groups of channels are dedicated for edge features, and some are for bulk features). On the other hand, IN does not allow any grouping of features, and implicitly assumes that every channel represents a distinct feature that has to be separately normalized.
In reality, I guess there is some underlying natural "grouping" of channel features that is between the two extremes, and GN takes advantage of this.
