Not sure on speeding up backwards, but
(1) as Andrew Plassard @aplassard notes, you should probably use F.linear instead of  torch.matmul(vec, self.W) + self.b. Doing fused addmm is optimized for doing, well, matrix multiplication and addition whereas matmul ... + is not.
Then (2) define text_transform as its own method rather than a nested function. As is, that is defined every forward pass. In my experience, that can be a significant slowdown.
Then I also wonder if there’s a way to batch-vectorize your first list comprehension, text_transform(tor...). That could go a long way on forward and backward.