Ok sir, I have one last doubt, I have noticed that if we change the shape of any variable in the forward pass, it takes up a lot of gpu memory so the solution I found online was to detach the variable as in example:

class FAN(nn.Module):
    def __init__(
        self,):
        super(FAN, self).__init__()
        self.encoder = Encoder()

    def forward(self, x):
        batch_size, seq_length, c, h, w = x.shape
        x = x.view(batch_size * seq_length, c, h, w).detach()
        x = self.encoder(x)
        return x

and

class FAN(nn.Module):
    def __init__(
        self,):
        super(FAN, self).__init__()
        self.encoder = Encoder()

    def forward(self, x):
        batch_size, seq_length, c, h, w = x.shape
        x = x.view(batch_size * seq_length, c, h, w)
        x = self.encoder(x)
        return x

Will the backpropogation graph in both cases is same? Or the first method is not correct and the weights will not get updated normally?
Thanks for such quick replies. Your help is appreciated