If you are pushing tensors to the device, you have to reassign them:
clips = clips.to(device)
optical_flows = optical_flows.to(device)
...

Also, it’s recommended to call the model directly (net(clips, optical_flows)) instead of the .forward() method. This is unrelated to your current issue, but could avoid other problems in the future.