You probably want to look into vectorizing your function better, i.e. ideally eliminate the loops over in and out channels (the batch one might be less urgent).
Unless you’ll go all the way to custom CUDA kernels (and that’s probably quite some effort) you’ll have a hard time solving this, even using C++.
Best regards
Thomas