Hi again jpeg,
I don’t reset the hidden state after each frame - the below part of my code iterates through the frames.
for u in range(len(lstm_input)):
    hidden = lstm(lstm_input[u, :], hidden)
    mu  = hidden[0][0, -4:] # extract last elements of each hidden layer

What you proposed would indeed probably solve my problems, however I can’t do it as  described since I have to access the last 4 coordinates of the hidden state after each timestep. If I feed in a PackedSequence of T features at once, all of the intermediate hidden states are lost. I tried my best to put it in pseudocode once again:
class LSTM_(nn.Module):
    def __init__(self, input_size, hidden_size):
        super(LSTM_, self).__init__()
        self.hidden_size = hidden_size
        self.input_size = input_size
        self.lstm = nn.LSTMCell(4096 + num_coords, hidden_size)

    def forward(self, input, hidden):
        hidden = self.lstm(input, hidden)
        return hidden

    def initHidden(self):
        return (Variable(torch.zeros(1, self.hidden_size)).cuda(), 
                    Variable(torch.zeros(1, self.hidden_size)).cuda())
 
def train(input_lstm):
    """this function trains the LSTM for exactly clip_length frames, once"""
    hidden = lstm.initHidden()
    optimizer.zero_grad()
    
    # now, iterate over a clip
    for u in range(len(lstm_input)):                            
        input_ = Variable(input_lstm[u, :])
        hidden = lstm(input_, hidden)
        mu  = hidden[0][0, -4:] # access some coordinates of hidden state
        mu_list_for_later.append(mu.cpu().data.numpy())

    grad = function_of_the_intermediate_hidden_states(mu_list_for_later)
    
    print('gradient: ',grad) 
    grad = Variable(grad)
    mu.backward(grad)

    print('grad sum is')
    print(np.sum(f.grad.cpu().data.numpy()))
    optimizer.step()

lstm = LSTM_(4096 + num_coords, hidden_size)

hidden = lstm.initHidden()

optimizer = optim.Adam(list(lstm.parameters()), lr = lr)

if __name__ == '__main__': 
    for u in range(args.epochs):
        for i, video in tqdm(enumerate(dataloader)):
            for j in range(clips_per_video):                
                input_lstm = build_feature_vector_using_a_CNN()
                train(input_lstm) # do this for many different feature vectors

The problem remains:
The line
print('gradient: ',grad) 

prints a large gradient that doesn’t change much (which is fine).
However, the term
for f in lstm.parameters():
    print('grad sum is')
    print(np.sum(f.grad.cpu().data.numpy()))

vanishes in an exponentially declining manner.
I’m sure that I make a conceptual mistake here, but I fail to find it. Once again, many thanks for your answers, they already helped me a lot towards understanding how PyTorch works. Highly appreciated.