The operations don’t reside on cuda, the tensors do. When you move your model to CUDA using model.cuda() and pass into the model function tensors say input and target that are on cuda() all computations will take place on GPU I believe.
So you can define your model using just torch.mm 86 or torch.mul, just make sure that the tensors/variables that are passed into the model() are on cuda and that your model is also on cuda.
Hope this helps!