It might be you are trying to backward through resnet as well. It is hard to guess without knowledge of how exactly you collect the features and what data argument represent in the train_model function: could be batch or full matrix.