Thanks for your answer!
Can you explain me the “the spatial size” of the encoder ? As I can understand you are trying to tell me that resnet are trained for specific input size 224x224, so in order to use a resnet I must have same input size. But as I can understand when we pass an image from an encoder, we do not have any more pixels. For example I have:
x.shape = ([128,3,32,32]), where 128 is batch size, 3 channels and 32x32 pixels each image
When I call rkhs_1 = encode(x) , then rkhs_1.shape = ([128,1024,1,1]) and I want to use this as input to resnet to train a classifier for cifar10.
If I had pixels as input what I am doing for resnet is to change the first layer:
model.conv1 = nn.Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
model.maxpool = nn.Identity()

but now I don’t have pixels, I have only a 1024 vector for each image