Hi @cmdkev , thanks for using Ignite !
I think in your case, the training step is a bit different from the basic one used in create_supervised_trainer, I suggest to create the trainer and evaluator in the following and more flexible way:
from ignite.engine import Engine

model = BiLSTM()
loss_func = nn.MSELoss(reduction='mean')
optimizer = torch.optim.Adam(model.parameters(), lr=lr)
device = "cuda"

def training_step(engine, batch):
    model.train()
    optimizer.zero_grad()

    x, y = batch[0].to(device), batch[1].to(device)
    y_pred, hidden = model(x, trainer.state.hidden)
    loss = loss_func(y_pred, y)

    loss.backward()
    optimizer.step()

    trainer.state.hidden = hidden

    return loss.item()


trainer = Engine(training_step)
trainer.state.hidden = None

@torch.no_grad()
def evaluation_step(engine, batch):
    model.eval()

    x, y = batch[0].to(device), batch[1].to(device)
    y_pred, _ = model(x)
    return y_pred, y

evaluator = Engine(evaluation_step)


Remaining code can be the same.
Hope this helps and, please, let me know if it works or not.