Hmmm, I do not think that I understand what you said.
To help you, my data is a matrix with shape 1000x2048. Every row of the matrix is the output from a intermediate layer of resnet50 MoCo model. First I create the matrix, so when I train my MLP, I do not use resnet50 at all. I use it only in the beginning in order to create the matrix.