No, in my model too I create a new hidden variable with previous hidden data in every batch:
But just look at my training logs:
| epoch   1 |    10/ 1327 batches | lr 20.00 | ms/batch 4014.78 | loss  8.95 | ppl  7732.07
| epoch   1 |    20/ 1327 batches | lr 20.00 | ms/batch 3434.41 | loss  7.36 | ppl  1565.54
| epoch   1 |    30/ 1327 batches | lr 20.00 | ms/batch 3776.96 | loss  7.18 | ppl  1307.52
| epoch   1 |    40/ 1327 batches | lr 20.00 | ms/batch 3311.54 | loss  7.08 | ppl  1182.57
| epoch   1 |    50/ 1327 batches | lr 20.00 | ms/batch 3227.49 | loss  6.95 | ppl  1046.28
| epoch   1 |    60/ 1327 batches | lr 20.00 | ms/batch 3226.90 | loss  6.95 | ppl  1042.19
| epoch   1 |    70/ 1327 batches | lr 20.00 | ms/batch 2971.59 | loss  6.81 | ppl   904.40
| epoch   1 |    80/ 1327 batches | lr 20.00 | ms/batch 3106.82 | loss  6.87 | ppl   962.97
| epoch   1 |    90/ 1327 batches | lr 20.00 | ms/batch 3352.58 | loss  6.78 | ppl   883.79
| epoch   1 |   100/ 1327 batches | lr 20.00 | ms/batch 3032.48 | loss  6.77 | ppl   870.15
| epoch   1 |   110/ 1327 batches | lr 20.00 | ms/batch 2901.16 | loss  6.78 | ppl   878.80
| epoch   1 |   120/ 1327 batches | lr 20.00 | ms/batch 3002.09 | loss  6.72 | ppl   831.85
| epoch   1 |   130/ 1327 batches | lr 20.00 | ms/batch 2947.38 | loss  6.74 | ppl   849.17
| epoch   1 |   140/ 1327 batches | lr 20.00 | ms/batch 2962.97 | loss  6.77 | ppl   873.62
| epoch   1 |   150/ 1327 batches | lr 20.00 | ms/batch 3008.20 | loss  6.74 | ppl   844.83
| epoch   1 |   160/ 1327 batches | lr 20.00 | ms/batch 2940.90 | loss  6.71 | ppl   823.43
| epoch   1 |   170/ 1327 batches | lr 20.00 | ms/batch 3062.02 | loss  6.77 | ppl   868.79
| epoch   1 |   180/ 1327 batches | lr 20.00 | ms/batch 3022.68 | loss  6.78 | ppl   881.30
| epoch   1 |   190/ 1327 batches | lr 20.00 | ms/batch 2979.49 | loss  6.65 | ppl   776.37
| epoch   1 |   200/ 1327 batches | lr 20.00 | ms/batch 2969.84 | loss  6.66 | ppl   782.79
| epoch   1 |   210/ 1327 batches | lr 20.00 | ms/batch 2922.02 | loss  6.73 | ppl   840.52
| epoch   1 |   220/ 1327 batches | lr 20.00 | ms/batch 3018.08 | loss  6.64 | ppl   766.08
| epoch   1 |   230/ 1327 batches | lr 20.00 | ms/batch 3340.55 | loss  6.66 | ppl   776.91
| epoch   1 |   240/ 1327 batches | lr 20.00 | ms/batch 2847.63 | loss  6.65 | ppl   773.44
| epoch   1 |   250/ 1327 batches | lr 20.00 | ms/batch 2953.34 | loss  6.71 | ppl   818.55
| epoch   1 |   260/ 1327 batches | lr 20.00 | ms/batch 2964.03 | loss  6.68 | ppl   799.16
| epoch   1 |   270/ 1327 batches | lr 20.00 | ms/batch 2933.35 | loss  6.70 | ppl   813.85
| epoch   1 |   280/ 1327 batches | lr 20.00 | ms/batch 2907.14 | loss  6.68 | ppl   794.25
| epoch   1 |   290/ 1327 batches | lr 20.00 | ms/batch 3008.06 | loss  6.67 | ppl   790.62
| epoch   1 |   300/ 1327 batches | lr 20.00 | ms/batch 2966.33 | loss  6.59 | ppl   725.52
| epoch   1 |   310/ 1327 batches | lr 20.00 | ms/batch 2951.73 | loss  6.57 | ppl   712.75
| epoch   1 |   320/ 1327 batches | lr 20.00 | ms/batch 2856.64 | loss  6.51 | ppl   672.87
| epoch   1 |   330/ 1327 batches | lr 20.00 | ms/batch 3065.96 | loss  6.63 | ppl   760.79
| epoch   1 |   340/ 1327 batches | lr 20.00 | ms/batch 3290.83 | loss  6.70 | ppl   813.69
| epoch   1 |   350/ 1327 batches | lr 20.00 | ms/batch 3535.03 | loss  6.76 | ppl   859.84
| epoch   1 |   360/ 1327 batches | lr 20.00 | ms/batch 4087.89 | loss  6.65 | ppl   774.28
| epoch   1 |   370/ 1327 batches | lr 20.00 | ms/batch 5108.70 | loss  6.66 | ppl   778.76
| epoch   1 |   380/ 1327 batches | lr 20.00 | ms/batch 6547.46 | loss  6.65 | ppl   773.92
| epoch   1 |   390/ 1327 batches | lr 20.00 | ms/batch 7921.22 | loss  6.65 | ppl   774.29
| epoch   1 |   400/ 1327 batches | lr 20.00 | ms/batch 9174.89 | loss  6.63 | ppl   754.19
| epoch   1 |   410/ 1327 batches | lr 20.00 | ms/batch 10519.90 | loss  6.66 | ppl   777.76
| epoch   1 |   420/ 1327 batches | lr 20.00 | ms/batch 11718.43 | loss  6.66 | ppl   781.59
| epoch   1 |   430/ 1327 batches | lr 20.00 | ms/batch 12073.58 | loss  6.58 | ppl   721.32
| epoch   1 |   440/ 1327 batches | lr 20.00 | ms/batch 12529.14 | loss  6.64 | ppl   766.29
| epoch   1 |   450/ 1327 batches | lr 20.00 | ms/batch 14175.94 | loss  6.63 | ppl   755.54
| epoch   1 |   460/ 1327 batches | lr 20.00 | ms/batch 14835.17 | loss  6.63 | ppl   758.95
| epoch   1 |   470/ 1327 batches | lr 20.00 | ms/batch 16415.23 | loss  6.63 | ppl   757.16
| epoch   1 |   480/ 1327 batches | lr 20.00 | ms/batch 17147.11 | loss  6.55 | ppl   700.03
ms/batch is increasing and increasing…!!
The model is the same, same parameters, same everything. Really slow training.