If clearing the cache was hitting the performance, wouldn’t it decrease the performance on both of the computers? They are running the same code at the same time, and there is a huge performance difference both in terms of allocated VRAM which leads to reduced inference speed due to lack of gpu VRAM available.
I realized when the gpu VRAM is full while using Flask server, whenever a request is made that requires more space on VRAM, it queues the operation which leads to reduced inference speed.
So the issue is with RTX3070 allocating more VRAM compared to 2070 SUPER.