What makes deep learning model complexity so different that conventional measures are not sufficient? Currently, there are some efforts to define a complexity measure for deep learning models. Such as topological, spectral ergodic, see in-depth recent survey.
What makes deep learning model complexity so different that conventional measures are not sufficient? Conventional in the sense that we would use for model selection, AIC or BIC. In other words, Are AIC or BIC not good enough for overparameterized models?

I want to preface this by saying that I am not an expert in this area but I did read some related articles a while back.
I think a reason that classical techniques like AIC does not work for neural networks is due to the symmetry of the weight space. If I recall correctly then AIC is based on the fact that the in the limit of infinite data the MLE will converge to a specific parameter setting, but in neural networks you have symmetries so that different parameter configurations can in fact model the exact same function. An example of this symmetry is if you have a network with ReLU activation functions. In this case you can multiply the incoming weights and bias for a neuron with a factor 1/k where k>0 and the outgoing weights from that same neuron with k and the function will remain identical. So the MLE cannot converge to a single parameter setting in the limit and the AIC theory will not hold.
Another reason that classical techniques does not apply is that the large modern networks can memorize the data even when the labels are random as shown in "Understanding deep learning requires rethinking generalization". This means that classical measures like Rademachers does not yield that much information.
