Understanding deep learning requires rethinking generalization
I tend to think this is a result of classification&#x27;s information density being too low. You can &#x27;learn&#x27; a classification problem in the same way with a hash function: take a hash of each image, and memorize the hash and label. Then you only need to &#x27;learn&#x27; a very tiny amount of data relative to the size of the dataset to get zero loss. A series of random projections can also function as a crude hash function, and this is likely how NN memorization works.<p>Generative models, on the other hand, don&#x27;t allow this kind of data reduction trick. If you need to predict (say) every sample of an audio stream conditioned on the previous audio samples, you really do need to memorize the whole dataset (not just a hash of each item) to get to zero loss, because the information density of the output is still very high.<p>And then you&#x27;ve got BERT, which is basically a generative model for language, used for downstream tasks. The information-dense task probably helps with the memorization &#x27;problem&#x27; so good features are learned that adapt nicely to other tasks. (and as others have said, memorization may not be a problem in practice. Sometimes it&#x27;s actually the right thing to do.)