I found a very similar post here that answers my questions

  
    
    
    How to process variable length sequence of images with CNN 
  
  
    Sure, each data sequence is of format [image_0, action_1, image_1, action_2, image_2, action_3, image_3…], and the task is to predict action sequence [action_1, action_,…, action_k] given image sequence [image_0, image_1,…, image_k]. The image sequence of a variable length k+1 in a batch, so I pad each sequence with zero images until sequence length is max_seq_len. The batched input is thus of shape (B, max_seq_len, C, H, W). 
My network uses a CNN model to embed each image into a feature vector…