For everybody who reads this and is looking for a good solution: Try “Differentiable Spatial to Numerical Transform” (DSNT)
Code: https://github.com/anibali/dsntnn 73
Paper: https://arxiv.org/pdf/1801.07372.pdf 47
This outperforms both the tanh and linear solution by far in my case. And it is really easy to use and adds no learnable parameters to your model.