<p>First of all: an activation function is usually placed after a linear operation and you can have a lot of them (maybe different) in your nn. That's why it would be better to say <span class="math-container">$\bf a$</span> activation function of the neural net and not <span class="math-container">$\bf the$</span> activation function. So if you meant by activation function the last operation which is going to make your outputs non-negative, then you're right, ReLU isn't a good choice. Usually when people need to output some positive values, they take exponent as the last operation. I used it several times and everything was just fine.</p>

CC BY-SA 4.0
2020-09-12T16:08:14.353Z
2020-09-12T16:08:14.353Z
Michał Słodki
<p>I am working on a deep Q-learning project. My project is different than normal deep Q-learning. The rewards of my neural network must be positive because I need their values to importance sample actions. I know that I can't use ReLU as the activation function of my neural network. So the only suitable functions which I know are sigmoid, softmax and exponential function. I tried working with sigmoid and softmax but they generate wrong results and the loss function diverges.
There are two terminal states in my model. Their rewards are 1 and 0. All other states don't have any immediate rewards.</p>

<neural-networks><reinforcement-learning>
What are some suitable positive functions as activations of neural networks?
bitWise
# What are some suitable positive functions as activations of neural networks?

(Asked by: username_0 on 2020-09-11)

I am working on a deep Q-learning project. My project is different than normal deep Q-learning. The rewards of my neural network must be positive because I need their values to importance sample actions. I know that I can't use ReLU as the activation function of my neural network. So the only suitable functions which I know are sigmoid, softmax and exponential function. I tried working with sigmoid and softmax but they generate wrong results and the loss function diverges.
There are two terminal states in my model. Their rewards are 1 and 0. All other states don't have any immediate rewards.

**Tags:** neural-networks, reinforcement-learning


---

## 1 answers

### Answer by username_1 on 2020-09-12. Score: 1 | Accepted answer

First of all: an activation function is usually placed after a linear operation and you can have a lot of them (maybe different) in your nn. That's why it would be better to say $\bf a$ activation function of the neural net and not $\bf the$ activation function. So if you meant by activation function the last operation which is going to make your outputs non-negative, then you're right, ReLU isn't a good choice. Usually when people need to output some positive values, they take exponent as the last operation. I used it several times and everything was just fine.


---

