I solved the problem by using the reparametrization trick with Gumbel Softmax. I took inspiration from this: https://gist.github.com/yzh119/fd2146d2aeb329d067568a493b20172f 174
Thanks for your help anyway