If anyone is concerned about bi-directional LSTM and memory leak issues in pytorch - maybe a good place to start coding a new project is from the already working tutorial?
https://github.com/yunjey/pytorch-tutorial/blob/master/tutorials/07%20-%20Bidirectional%20Recurrent%20Neural%20Network/main.py 51