Can the dimension of the latent space in VAEs, be larger than the dimension of the data? I am experimenting with VAEs. There, there is a parameter that you need pass when you create the NN, which is the dimension of the latent space.
In the typical MNIST example we have the following data:
# Load digits data 
(X_train, y_train), (X_test, y_test) = keras.datasets.mnist.load_data()

# Print shapes
print("Shape of X_train: ", X_train.shape)
print("Shape of y_train: ", y_train.shape)
print("Shape of X_test: ", X_test.shape)
print("Shape of y_test: ", y_test.shape)

Shape of X_train:  (60000, 28, 28)
Shape of y_train:  (60000,)
Shape of X_test:  (10000, 28, 28)
Shape of y_test:  (10000,)

which after reshaping become:
# Reshape input data
X_train = X_train.reshape(60000, 784)
X_test = X_test.reshape(10000, 784)

# Print shapes
print("New shape of X_train: ", X_train.shape)
print("New shape of X_test: ", X_test.shape)

New shape of X_train:  (60000, 784)
New shape of X_test:  (10000, 784)

So after reshaping, we end up with 60k observations with 784 features/dimensions for the training data and 10k observations with 784 features/dimensions for the test data.
In all the examples that I show, they chose a dimension of the latent space which is smaller than 784.
My questions are:

*

*what does it mean if you specify the dimension of the latent space, to be larger than 784, lets say 1000 ?


*Also how does the dimension of the latent space affect the results ? Why (in this MNIST example) would someone choose 100 or 200 instead of 300 ?

A guiding principle of autoencoders is that they must obtain a bottleneck.
If they don't, they may simply learn the identity function (i.e. to perfectly replicate the input, but without building any meaningful internal representation of it), which is useless for sampling in the case of VAEs.
I think a practical example of this behaviour is shown in this question.
The dimension of the latent space does affect results, and finding the right dimension brings the tradeoff that Simon stated.
