Hi, so I’ve been working on a model that uses a few different components and when benchmarking found some bizarre inconsistencies with the performance of the various models.
All models are just basic ConvNets with BatchNorm and ReLU layers.
Specs:

RTX 3090
CUDA 11.4
PyTorch 1.10.0
cuDNN 8.2.0

The initial setup looks somewhat like this:
model_1 = Model_1().cuda()
model_2 = Model_2().cuda()

x = torch.rand(1, 5, 3, 416, 416, device='cuda')

Now, the weird part is the performance issue that I experience with different combinations of the models.
Example 1:
for _ in range(100):
    model_2(model_1(x))

for _ in range(100):
    model_2(x)
# >> Second loop takes 0.053 seconds

Example 2:
for _ in range(100):
    model_2(x)
# >> This loop takes 0.47 seconds

Example 3:
for _ in range(100):
    model_2(x)

for _ in range(100):
    model_2(x)
# >> Second loop takes 2.5 seconds

Example 4:
for _ in range(100):
    model_1(x)
# >> This loop takes 1.72 seconds

for _ in range(100):
   model_2(x)
# >> This loop takes 0.01 seconds

for _ in range(100):
    model_1(x)
    model_2(x)
# >> This loop takes 8 seconds

Example 5:
for _ in range(100):
    model_1(x)
    model_2(x)
# >> This loop takes 5 seconds

I have no idea what is going on. Why am I experiencing an ~1.6x reduction in performance between Example 4 and Example 5 by just removing the two initial loops.
Why does model_2 take ~0.47 seconds to loop 100 times alone (Example 2) but when I add a loop before with model_2(model_1(x)) (Example 1) it takes ~0.05 seconds to loop 100 times alone. But then adding a loop with model_2(x) (Example 3) reduces the performance by ~5x.
What is going on here? How do I mitigate the issue?