Body: <p>I find the logistic map absolutely fascinating. Both in itself (because I love fractal) and because it is observed in nature (see: <a href="https://www.youtube.com/watch?v=ovJcsL7vyrk" rel="nofollow noreferrer">https://www.youtube.com/watch?v=ovJcsL7vyrk</a>).</p>
  <p>I'm wondering if anyone tried it as an activation function in some way or another with any kind of success.</p>
  <p>I like it because it has some kind of &quot;I'm not sure what to do&quot; above ~3.0 and the less confidence the more chaotic the response is. It gives the possibility to explore some other solution to escape a local optimum (not sure I use this word correctly).  And below 3 it's still a nice and smooth activation function like, eg, a tanh.</p>
  <p>Eg : the reward i got isn't the reward i expect, and the higher the difference the more i'll explore other solution. But it's still gradual, from 1 choice, to 2 choice, 4, 8, 16, ... until it become chaotic. (giving the possibility to experiment some pseudo-random choice). And below this threshold it still act as a usable &quot;good old&quot; activation function.</p>
  <p>Another good side is that it's gpu-friendly and don't need many iteration for this application since a little bit of uncertainty (even below the threshold) isn't undesirable. see : <a href="https://upload.wikimedia.org/wikipedia/commons/6/63/Logistic_Map_Animation.gif" rel="nofollow noreferrer">https://upload.wikimedia.org/wikipedia/commons/6/63/Logistic_Map_Animation.gif</a></p>
  <p>Edit : so, ok, i tested it on my extremely naive racetrack. (feedforward, no feedback, no error, no fitness, only genetic selection for the car that didn't crash). It does work, for sure. I don't see any advantage in practive but with such a naive NN, there isn't much i can tell.</p>
  <p>My implementation :</p>
  <pre><code>def logi(r):
      x = .6  # the initial population doesn't matter so i took .6
      for _ in range(random.randrange(10,40)):
          x = r * x * (1 - x)
      return x
  </code></pre>
  <p>The activation take 8% of my laptop cpu (while is was invisible on my radar with leaky leru)</p>
  <p><a href="https://i.sstatic.net/XX7zz.gif" rel="nofollow noreferrer"><img src="https://i.sstatic.net/XX7zz.gif" alt="Logistic map bifurcation diagram" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-23T17:25:00.063Z
Comments.Text: Can you explain more in detail how you would use the "logistic map" as an activation function? Maybe it would be a good idea if you briefly describe what the "logistic map" is, how it is defined, and why you think it could be used as an activation function.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-23T23:22:51.797Z
Comments.Text: i'm not quite sure, i'm still struggling with my code and don't have the leisure to experiment like this, yet. But the general idea is to allow the NN to have some level of chaotic behavior. Eg : the reward i got isn't the reward i expect, and the higher the difference the more i'll explore other solution. But it's still gradual, from 1 choice, to 2 choice, 4, 8, 16, ... until it become chaotic. (giving the possibility to experiment some pseudo-random choice). And below this threshold it still act as a usable "good old" activation function.
Comments.UserDisplayName: ker2x
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-23T23:24:08.320Z
Comments.Text: The formula is still simple https://ipython-books.github.io/121-plotting-the-bifurcation-diagram-of-a-chaotic-dynamical-system/ but a bit slow since it is an iterated function.
Comments.UserDisplayName: ker2x
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-23T23:30:41.157Z
Comments.Text: Another good side is that it's gpu-friendly and don't need many iteration for this application since a little bit of uncertainty (even below the threshold) isn't undesirable. see : https://en.wikipedia.org/wiki/Logistic_map#/media/File:Logistic_Map_Animation.gif
Comments.UserDisplayName: ker2x
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-23T23:31:47.643Z
Comments.Text: I suggest that you edit your post to include more details that explain your idea.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-23T23:45:05.660Z
Comments.Text: i edited my post. as for what the logistic map is, the veritasium video i linked explain it better than i can. Biological Neuron also have some behavior that look like a logistic map
Comments.UserDisplayName: ker2x
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-24T00:51:59.130Z
Comments.Text: Wait, it's not clear here what you mean by this "(feedforward, no feedback, no error, no fitness, only genetic selection for the car that didn't crash)". So, you have a MLP with this activation function. What does this MLP get as input and produce as output? What exactly are you changing with "genetic selection"?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-24T01:04:53.060Z
Comments.Text: I have a selection process (car that didn't crash + multiple mutated clone (child) of the fastest car) and randomly change (multiplied by a decreasing learning rate) the weight and bias of a randomly chosen layer based on some mutation chance. I'll have to change the code to chose a random neuron instead of a random layer.
Comments.UserDisplayName: ker2x
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-24T01:09:40.543Z
Comments.Text: The input are some kind of lidar (raycasting from -90 to 90°) to detect the racetrack + speed of the car + lateral velocity (+/- G lateral force, for drifting)
Comments.UserDisplayName: ker2x
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-24T01:12:06.023Z
Comments.Text: this is similar, afaik, to what seems to be implemented in the game "gridworld" https://store.steampowered.com/app/396890/Gridworld/
Comments.UserDisplayName: ker2x
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-24T01:24:40.297Z
Comments.Text: I suggest that you apply your idea (of using that function as an activation) to a simple neural network in a simple task. Start with a dataset like Iris Flowers Dataset. You're trying to combine multiple things, each of which may not work, consequently, it will be hard to make something work.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-24T07:16:08.187Z
Comments.Text: or someone with the necessary expertise to evaluate/test/quantify an activation function could do it ?
Comments.UserDisplayName: ker2x
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-24T08:00:05.207Z
Comments.Text: my main concern is that it may simply amount to having a "noisy input" and nothing else.
Comments.UserDisplayName: ker2x
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-24T10:19:40.643Z
Comments.Text: So, you said that the input to your neural network is "some kind of lidar (raycasting from -90 to 90°) to detect the racetrack + speed of the car + lateral velocity (+/- G lateral force, for drifting)", but what is the **output** of this neural network supposed to represent? I still don't understand your setup and what problem you're trying to solve. You should edit your post to start by saying: 1. what problem you're trying to solve, 2. which techniques are you using and **how** exactly you're using to solve that problem, 3. where this logistic map comes into play exactly in this process.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-24T10:21:40.553Z
Comments.Text: I know that this is not the main part of the question, but I still don't understand where you're using this logistic map. Your implementation suggests that this logistic map gets $r$ as input, but $r$ is just a hyper-parameter that defines the specific logistic map curve, and not the real input to a neuron of a NN (or is $r$ supposed to be that?). Where exactly are you using this logistic map? Are you using it as an activation function of a neural network or something else? Are all your activation functions in the NN equal to the logistic map? What $r$ value are you using for each of them?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-24T10:41:36.647Z
Comments.Text: Also, what are these numbers " from 1 choice, to 2 choice, 4, 8, 16,"? Are the numbers you use for $r$ or what? If yes, then I don't understand at all how you're using it as an activation function of the neurons of your neural network (NN).
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-24T11:10:12.230Z
Comments.Text: that's how the logistic map work. After N iteration, whatever the initial condition is, it converge to a single point if r (the input) is <  ~3, then it oscillate between 2 point if ~3 < r < ~3.44. then 4 points if ~3.44 < r < ~3.54, then 8, 16, ... and above ~3.56 it become chaotic; then there are some "island" for r where it oscillate again, then chaotic again, ... It works as an activation function since you can map an input (r) to an output (x). it is, however, extremely sensitive to initial condition above a certain threshold (that's why it's chaotic)
Comments.UserDisplayName: ker2x
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-24T11:11:26.343Z
Comments.Text: the video i linked in the post explain it all. it's an extremely simple formula.
Comments.UserDisplayName: ker2x
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-24T11:15:07.880Z
Comments.Text: my own problem/playground to test this does not matter. The point of the post is to know if anyone else used/tried/tested/evaluated the logistic map as a valid activation function (instead of using relu, tanh, etc ...) and what they think of it. (i know it works on my playground, but does it always ? is it better in some case or another ? i'm not good enough to test and tell)
Comments.UserDisplayName: ker2x
Tags: <neural-networks><activation-functions>
Title: Has the logistic map ever been used as an activation function?
UserDisplayName: ker2x
ThreadText: # Has the logistic map ever been used as an activation function?
  
  (Asked by: username_0 on 2020-11-23)
  
  I find the logistic map absolutely fascinating. Both in itself (because I love fractal) and because it is observed in nature (see: https://www.youtube.com/watch?v=ovJcsL7vyrk).
  I'm wondering if anyone tried it as an activation function in some way or another with any kind of success.
  I like it because it has some kind of "I'm not sure what to do" above ~3.0 and the less confidence the more chaotic the response is. It gives the possibility to explore some other solution to escape a local optimum (not sure I use this word correctly).  And below 3 it's still a nice and smooth activation function like, eg, a tanh.
  Eg : the reward i got isn't the reward i expect, and the higher the difference the more i'll explore other solution. But it's still gradual, from 1 choice, to 2 choice, 4, 8, 16, ... until it become chaotic. (giving the possibility to experiment some pseudo-random choice). And below this threshold it still act as a usable "good old" activation function.
  Another good side is that it's gpu-friendly and don't need many iteration for this application since a little bit of uncertainty (even below the threshold) isn't undesirable. see : https://upload.wikimedia.org/wikipedia/commons/6/63/Logistic_Map_Animation.gif
  Edit : so, ok, i tested it on my extremely naive racetrack. (feedforward, no feedback, no error, no fitness, only genetic selection for the car that didn't crash). It does work, for sure. I don't see any advantage in practive but with such a naive NN, there isn't much i can tell.
  My implementation :
  def logi(r):
      x = .6  # the initial population doesn't matter so i took .6
      for _ in range(random.randrange(10,40)):
          x = r * x * (1 - x)
      return x
  
  The activation take 8% of my laptop cpu (while is was invisible on my radar with leaky leru)
  
  **Tags:** neural-networks, activation-functions
  > **Comments:**
  > **username_1**: Can you explain more in detail how you would use the "logistic map" as an activation function? Maybe it would be a good idea if you briefly describe what the "logistic map" is, how it is defined, and why you think it could be used as an activation function.  
  > **username_0**: i'm not quite sure, i'm still struggling with my code and don't have the leisure to experiment like this, yet. But the general idea is to allow the NN to have some level of chaotic behavior. Eg : the reward i got isn't the reward i expect, and the higher the difference the more i'll explore other solution. But it's still gradual, from 1 choice, to 2 choice, 4, 8, 16, ... until it become chaotic. (giving the possibility to experiment some pseudo-random choice). And below this threshold it still act as a usable "good old" activation function.  
  > **username_0**: The formula is still simple https://ipython-books.github.io/121-plotting-the-bifurcation-diagram-of-a-chaotic-dynamical-system/ but a bit slow since it is an iterated function.  
  > **username_0**: Another good side is that it's gpu-friendly and don't need many iteration for this application since a little bit of uncertainty (even below the threshold) isn't undesirable. see : https://en.wikipedia.org/wiki/Logistic_map#/media/File:Logistic_Map_Animation.gif  
  > **username_1**: I suggest that you edit your post to include more details that explain your idea.  
  > **username_0**: i edited my post. as for what the logistic map is, the veritasium video i linked explain it better than i can. Biological Neuron also have some behavior that look like a logistic map  
  > **username_1**: Wait, it's not clear here what you mean by this "(feedforward, no feedback, no error, no fitness, only genetic selection for the car that didn't crash)". So, you have a MLP with this activation function. What does this MLP get as input and produce as output? What exactly are you changing with "genetic selection"?  
  > **username_0**: I have a selection process (car that didn't crash + multiple mutated clone (child) of the fastest car) and randomly change (multiplied by a decreasing learning rate) the weight and bias of a randomly chosen layer based on some mutation chance. I'll have to change the code to chose a random neuron instead of a random layer.  
  > **username_0**: The input are some kind of lidar (raycasting from -90 to 90°) to detect the racetrack + speed of the car + lateral velocity (+/- G lateral force, for drifting)  
  > **username_0**: this is similar, afaik, to what seems to be implemented in the game "gridworld" https://store.steampowered.com/app/396890/Gridworld/  
  > **username_1**: I suggest that you apply your idea (of using that function as an activation) to a simple neural network in a simple task. Start with a dataset like Iris Flowers Dataset. You're trying to combine multiple things, each of which may not work, consequently, it will be hard to make something work.  
  > **username_0**: or someone with the necessary expertise to evaluate/test/quantify an activation function could do it ?  
  > **username_0**: my main concern is that it may simply amount to having a "noisy input" and nothing else.  
  > **username_1**: So, you said that the input to your neural network is "some kind of lidar (raycasting from -90 to 90°) to detect the racetrack + speed of the car + lateral velocity (+/- G lateral force, for drifting)", but what is the **output** of this neural network supposed to represent? I still don't understand your setup and what problem you're trying to solve. You should edit your post to start by saying: 1. what problem you're trying to solve, 2. which techniques are you using and **how** exactly you're using to solve that problem, 3. where this logistic map comes into play exactly in this process.  
  > **username_1**: I know that this is not the main part of the question, but I still don't understand where you're using this logistic map. Your implementation suggests that this logistic map gets $r$ as input, but $r$ is just a hyper-parameter that defines the specific logistic map curve, and not the real input to a neuron of a NN (or is $r$ supposed to be that?). Where exactly are you using this logistic map? Are you using it as an activation function of a neural network or something else? Are all your activation functions in the NN equal to the logistic map? What $r$ value are you using for each of them?  
  > **username_1**: Also, what are these numbers " from 1 choice, to 2 choice, 4, 8, 16,"? Are the numbers you use for $r$ or what? If yes, then I don't understand at all how you're using it as an activation function of the neurons of your neural network (NN).  
  > **username_0**: that's how the logistic map work. After N iteration, whatever the initial condition is, it converge to a single point if r (the input) is <  ~3, then it oscillate between 2 point if ~3 < r < ~3.44. then 4 points if ~3.44 < r < ~3.54, then 8, 16, ... and above ~3.56 it become chaotic; then there are some "island" for r where it oscillate again, then chaotic again, ... It works as an activation function since you can map an input (r) to an output (x). it is, however, extremely sensitive to initial condition above a certain threshold (that's why it's chaotic)  
  > **username_0**: the video i linked in the post explain it all. it's an extremely simple formula.  
  > **username_0**: my own problem/playground to test this does not matter. The point of the post is to know if anyone else used/tried/tested/evaluated the logistic map as a valid activation function (instead of using relu, tanh, etc ...) and what they think of it. (i know it works on my playground, but does it always ? is it better in some case or another ? i'm not good enough to test and tell)  
  
  ---
  
  ## 0 answers
  
  