My model is a simple stack of FC and LSTM as follows.

class NETWORK(nn.Module):
def init(self, hparams):
super(NETWORK, self).init()
    self.fc = nn.Sequential(
        nn.Linear(self.inputDim, self.numNode), nn.ReLU(True), nn.Dropout(0.5),
        nn.Linear(self.numNode, self.numNode), nn.ReLU(True), nn.Dropout(0.5)
    )
    self.LSTM = nn.LSTM(input_size = self.numNode, hidden_size = int(self.numNode/2), num_layers = 1, bidirectional = self.bidirectional )
    self.outfc = nn.Sequential(
        nn.Linear(self.numLinear, self.outputDim))

def forward(self, inSequence, h, c):
    fcout = self.fc(inSequence)
    self.LSTM.flatten_parameters()
    lstmout, (h, c) = self.LSTM(fcout, (h, c))
    output = self.outfc(output)
    return output


I also train in FP16 environment w/ apex library from NVIDIA.
When I train wth FP32 training, everything goes well.
But when I train with FP16 training, LSTM output shows nan value.
Particularly, this NaN phenomena only occurs when I initialize hidden and cell state of LSTM w/ Normal distribution. (zero-mean, and variance value is between 0.0001~1.0000)
On the other hand, zero initialization of LSTM cell and hidden states doesnt  show this NaN phenomena. I want to initialize these states as small random numbers since it results in faster convergence of model.
Are there any solutions or hints to resolve this problem?