I check all parts in my model by printing out their execution time without  torch.cuda.synchronize().
One part with GRU and LayerNorm has a 100x more time cost than other part.
Code in this part is sth like:
v1 = self._gru(self._ln1(v1 + v0))
v2 = self._gru(self._ln2(v2 + v0))
v3 = self._gru(self._ln3(v3 + v0))

Here self._ln1 and self._ln2 and self._ln3 are instances of nn.LayerNorm
And self._gru is a Residual-GRU  with code
class ResidualGRU(nn.Module):
    def __init__(self, hidden_size, dropout, num_layers):
        super(ResidualGRU, self).__init__()
        self.enc_layer = nn.GRU(input_size=hidden_size, hidden_size=hidden_size // 2, num_layers=num_layers,
                                batch_first=True, dropout=dropout, bidirectional=True)
        self.enc_ln = nn.LayerNorm(hidden_size)

    def forward(self, input):
        output, _ = self.enc_layer(input)
        return self.enc_ln(output + input)

May I ask if GRU will cause sync? Or what’s wrong with these code.