Thanks, I will look into it.
Actually what I have is a CNN model in Theano/Lasagne, another CNN model in pytorch, after training both these seperately, I would take the fully connected (Dense layer) features from both CNN and feed into LSTM pytorch model.