  Based on our observation that there exists a dramatic drop for the singular
values of the fully connected layers or a single feature map of the
convolutional layer, and that the dimension of the concatenated feature vector
almost equals the summation of the dimension on each feature map, we propose a
singular value decomposition (SVD) based approach to estimate the dimension of
the deep manifolds for a typical convolutional neural network VGG19. We choose
three categories from the ImageNet, namely Persian Cat, Container Ship and
Volcano, and determine the local dimension of the deep manifolds of the deep
layers through the tangent space of a target image. Through several
augmentation methods, we found that the Gaussian noise method is closer to the
intrinsic dimension, as by adding random noise to an image we are moving in an
arbitrary dimension, and when the rank of the feature matrix of the augmented
images does not increase we are very close to the local dimension of the
manifold. We also estimate the dimension of the deep manifold based on the
tangent space for each of the maxpooling layers. Our results show that the
dimensions of different categories are close to each other and decline quickly
along the convolutional layers and fully connected layers. Furthermore, we show
that the dimensions decline quickly inside the Conv5 layer. Our work provides
new insights for the intrinsic structure of deep neural networks and helps
unveiling the inner organization of the black box of deep neural networks.

Computer Vision and Pattern Recognition