Hi
I am trying to implement recurrence over an arbitrary layer:
Here is the forward function:
def forward(self, sequence, hidden):
  
    i = 0
    for x in sequence:

        if(i==0):
            hidden[i] = x # Initially it is the input
            print("Hidden", hidden[i].data)
        else:
            hidden[i] = hidden[i-1].clone() # otherwise the previous time-step's output
        
        for layer in self.highway_layers: # This the recurrence over layers
            hidden[i] = layer(hidden[i].clone())
            print("Hidden", hidden[i].data)
           
        i = i +1
             
    return hidden, hidden # since for rnn, hidden and output are the same, if you don't want to do softmax

The hidden values soon explode (they are roughly doubling? ) and finally I get Math overflow error in loss function.
Hidden
  84.0122   89.9400   81.6280  ...    87.6645   70.6815   59.7525
  89.6357   59.4279  -87.3051  ...    30.1372  -84.1233  -96.8704
 -75.7449   75.4689  -73.7856  ...    40.0317   51.0543   38.5197
             ...                ⋱                ...
   6.2067   50.8187  -55.4566  ...    19.7606   39.6369   46.3537
  91.9438   -2.5618  -94.3562  ...   -24.0399  -38.2352  -58.3377
  97.2880  -29.3549   50.7392  ...   -39.0659  -86.5748  -47.3998
[torch.FloatTensor of size 20x200]

Hidden
 168.0244  179.8800  163.2560  ...   175.3291  141.3631  119.5050
 179.2715  118.8557 -174.6103  ...    60.2745 -168.2467 -193.7408
-151.4898  150.9377 -147.5712  ...    80.0634  102.1087   77.0395
             ...                ⋱                ...
  12.4135  101.6373 -110.9131  ...    39.5212   79.2738   92.7073
 183.8876   -5.1236 -188.7125  ...   -48.0799  -76.4703 -116.6755
 194.5760  -58.7099  101.4783  ...   -78.1319 -173.1495  -94.7997

How to fix this issue…? Thanks for your help.