I think it should not be a problem until the very last activation of the network before classification, where you have to make sure the output is in the range you expect it to be (usually either [0,1] or [-1,1]). You can check that by looking at the output tensor’s .min and .max values.
Remember that the activation is there to introduce non-linearity in the network.