It will return to ~5s for a forward, but It is difficult to find how it works. In addition, when the time cost increases, the performance (i.e., mAP) of the model will be dramatically worse.
My code is available at https://github.com/SeanChenxy/TSSD-OTA/tree/0.40 6, but it may take a little time to run it. I have written how to run it and produce this issue, and I think the most likely wrong code snippets are class TSSD and class seqMultiBoxLoss. If it is convenient for you, I hope you could have a look at it. Thank you very much.