Yeah I misunderstood what you were asking about. If you had some blocking get_latest_sample function, this should work - mostly the same as offline training but using a input of size 1 (or some chosen chunk size). Importantly, keep the last input and hidden state around for future time steps (I also updated the above gist to pass hidden as an argument to rnn.forward)
last_targets = get_latest_sample()
hidden = None

while True:
    inputs = last_targets
    targets = get_latest_sample()
    outputs, hidden = model(inputs, hidden)

    optimizer.zero_grad()
    loss = criterion(outputs, targets)
    loss.backward()
    optimizer.step()
    
    last_targets = targets