Just some suggestions from me, and I hope they help 

In this case, where u r feeding 1 frame at a time, u can replace LSTM with LSTMCell, tho it’s not necessary.
U really should add some dropouts inside ur model because both LSTM and Linear suffers from over-fitting badly.