Hi, I run the following code for testing my model on a single GPU:
model.eval()
for i, x in enumerate(test_inputs):
    t0 = time.time()
    with torch.no_grad():
        y = model(x)
    print(f'forward time: {time.time() - t0:.4f}s')

The model is a simple MLP with 11 layers, #neurons in each layer is 1024. Inputs is a tensor of shape [50, 1024].
My problem is: For the first 23 loops, the forward time is like 0.0015s. While after the #23 loop, the forward time suddenly jumps to ~0.28s and stay at this level afterwards. I am wondering why the forward speed suddenly slows down at some point while the previous loops work really fast? I feel this is related to pytorch cuda cache. What would you suggest to resolve this problem? Thanks!
My environment:
OS: Ubuntu20.04
CUDA: 11.2
GPU: Tesla V100 16GB
Python: 3.9.5
Pytorch1.9.0