The LSTM was just a slightly modified PyTorch Sequence Tagging 6 one modified to be a bi-LSTM and using sigmoid instead of softmax, and the Ignite loop was just the Quickstart code. The only difference was I used an Adam optimizer instead of SGD. And everything worked again once I implemented the fix in my original post.
I ended up scrapping LSTMs for my project in favour of 1DCNNs which have been working great with Ignite, but I can dig around my commit history for the original code and try it again if need be!