You could try to apply data augmentation on the input data. Depending on your dataset you could try to crop, change the brightness, color etc.
Also regularization might help, i.e. weight decay, Dropout, etc.
Are you training the whole network from scratch? If so, probably a pre-trained CNN will help, so that you just have to train the RNN.