Yep fixed it! No need to pass save the LSTM hidden state for this problem. Thanks!
For anyone interested, not reseting the hidden state after each forward pass in each branch of the saimese network will cause additional issues. Since the forward networks have shared weights (ie are the same network) the starting hidden state for the forward pass for the second branch will be the ending hidden state for the forward pass of the first branch.

Capture.PNG737×411 101 KB