Answers.Body: <p>First of all: an activation function is usually placed after a linear operation and you can have a lot of them (maybe different) in your nn. That's why it would be better to say <span class="math-container">$\bf a$</span> activation function of the neural net and not <span class="math-container">$\bf the$</span> activation function. So if you meant by activation function the last operation which is going to make your outputs non-negative, then you're right, ReLU isn't a good choice. Usually when people need to output some positive values, they take exponent as the last operation. I used it several times and everything was just fine.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-12T16:08:14.353Z
Answers.LastActivityDate: 2020-09-12T16:08:14.353Z
Answers.UserDisplayName: Michał Słodki
Body: <p>I am working on a deep Q-learning project. My project is different than normal deep Q-learning. The rewards of my neural network must be positive because I need their values to importance sample actions. I know that I can't use ReLU as the activation function of my neural network. So the only suitable functions which I know are sigmoid, softmax and exponential function. I tried working with sigmoid and softmax but they generate wrong results and the loss function diverges.
  There are two terminal states in my model. Their rewards are 1 and 0. All other states don't have any immediate rewards.</p>
  
Tags: <neural-networks><reinforcement-learning>
Title: What are some suitable positive functions as activations of neural networks?
UserDisplayName: bitWise
ThreadText: # What are some suitable positive functions as activations of neural networks?
  
  (Asked by: username_0 on 2020-09-11)
  
  I am working on a deep Q-learning project. My project is different than normal deep Q-learning. The rewards of my neural network must be positive because I need their values to importance sample actions. I know that I can't use ReLU as the activation function of my neural network. So the only suitable functions which I know are sigmoid, softmax and exponential function. I tried working with sigmoid and softmax but they generate wrong results and the loss function diverges.
  There are two terminal states in my model. Their rewards are 1 and 0. All other states don't have any immediate rewards.
  
  **Tags:** neural-networks, reinforcement-learning
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-09-12. Score: 1 | Accepted answer
  
  First of all: an activation function is usually placed after a linear operation and you can have a lot of them (maybe different) in your nn. That's why it would be better to say $\bf a$ activation function of the neural net and not $\bf the$ activation function. So if you meant by activation function the last operation which is going to make your outputs non-negative, then you're right, ReLU isn't a good choice. Usually when people need to output some positive values, they take exponent as the last operation. I used it several times and everything was just fine.
  
  
  ---
  
  