I found my mistake, the correct way is described in

  
      
      GitHub
  
  
    

HHTseng/video-classification

Tutorial for video classification/ action recognition using 3D CNN/ CNN+RNN on UCF101 - HHTseng/video-classification