Using different activation functions sounds reasonable. However, I would recommend to check if your code would raise errors for invalid inplace operations before porting it to libtorch.
If so, you might assign the results to temporal tensors and concatenate them afterwards.