Thank you for your suggestions.
I am actually predicting float values rather than labels (like in regression) so I am just using the value provided by the linear layer rather than using any activation.
P.S. see the other replies for the results.