If the slowdown is caused in your custom CUDA code by updating the CUDA toolkit, I would recommend to post this issue in the NVIDIA discussion board so that the compiler devs are aware of it.
EDIT: In case you are seeing the slowdown in 11.2, could you update to >=11.2U2 and check the performance again? We were aware of an increase in local memory usage for 11.2 and 11.2U1, which was fixed in newer versions.