ptrblck:

The output shape of [15, 1] is a bit weird, since it should be [batch_size, 17*batch_size] based on your model definition


I’m probably doing something wrong here. But this is my workflow:

Extract features features = pose_estimator.backbone(img_list.tensor.float()). The shape of the extracted features is torch.Size([1, 256, 272, 480]) via print(features[0].shape). Each image is added appended to a list called features_list.
Then I concatenate the images using:
cat_features = torch.cat([i[0].cpu() for i in features_list], 0). The shape is torch.Size([3, 256, 34, 60]), where the 3 represents that 3 image features were concatenated (i.e. batch size).
Then I apply cat_features = cat_features.transpose(1, 0).unsqueeze(0).to(device) to get in the format I need: torch.Size([1, 256, 3, 272, 480]), where now the batch is 1 is one as I want the concatenated features to be seen as a single batch by the model, and 3 represents the depth (i.e. 3 images).

I hope that so far it’s all making sense.
What I would like to do is use this image to predict future actions of pedestrians. So, for that, I also obtain keypoints using the keypoint_rcnn model. Each keypoint has a shape of torch.Size([1, 17, 3]) and when concatenating the keypoints of the 3 images as I did with the images in step 2, cat_kp = torch.cat(kypnt_instances[i]['pred_keypoints'], 0).to(device), I get a shape of torch.Size([51, 3]).
When I run my model, pred_label, pred_kp = model(cat_features), the shape of pred_kp is torch.Size([51, 1]), so when I try to calculate the loss kp_loss = kp_criterion(pred_kp, cat_kp), I get an error saying the red_kp and cat_kp are not the same size.
I think the issue is, that I want the model to see it as a batch size of 1, but output the keypoints as a batch of 3, 1 for each of the images. Which doesn’t make sense now that I think about it. Is there a way that I can combine/reshape cat_kp so that it is torch.Size([51, 1]) instead of torch.Size([51, 3]) without losing too much feature quality?
Sorry for the long post and I hope that it all makes some sense.