Hi Alban,
Thank you very much for reaching out.
I would love to, but currently I do have a repro on colab but I can not get that without my training set, which I access over my own gdrive. I will soon try to find a way. I am open to advices on that for sure.
Here is some good news though: I was able to eliminate the decrease in speed. Now my training is as fast as before. I have managed to fix the problem using one of my old environment.yml files. I had already checked the hardware, data, ubuntu and conda itself (by reinstalling all of them) but it was my environment after all.
What it all means to me is that there is a library (or are libraries) that causes the training to be 5x slower. I might be able to find that certain library/ies by gradually upgrading my environment. I will investigate further once I am done with my current project.
Here are links to my old 8 and new 12 environments.
Btw, I want to emphasize that the problem persisted when I tried to get a repro in colab. This might mean that something interesting going on here like, something in one of the up-to-date libraries might be slowing down the training of CSRNet.