I am using the Keypoint RCNN to predict keypoints. Using those keypoints, I am able to create a skeletal for detected objects (please see example in image below). What I want to do now is collect the keypoints of the detected object over a number of frames. I want to then classify these frames/keypoints as whether the pedestrian is going to cross the road or not. I hope my explanation makes sense.
I am thinking to feed in only the keypoints into the model without the corresponding images. I want the network to be dependent on just the keypoints for prediction. Would this be something that could work?
image960×540 110 KB
The issue I am having is how to feed the list of keypoints into the model. I have some experience working with image classification, but not with 3D inputs (i.e. the list of keypoints). Do I use torch.nn.Conv3d for this task?
Any advise would be greatly appreciated.