Body: <p>I am looking for an example in which it is simply impossible to use some sort of gradient descent to train a neural network. Is this available? </p>
  
  <p>I have read quite some papers about gradient-free optimization tools, but they always use it on a network for which you can also use gradient descent. I want to have a situation in which the only option to train the network is by, for example, a genetic algorithm. </p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-06-19T08:46:49.293Z
Comments.Text: What level of example are you looking for? Creating a neural network architecture that cannot be trained using gradient descent is very easy. Describing a problem that requires finding an optimal function, where it is possible to apply a NN, but impossible to use gradients, is a lot harder. It can be the case that using gradients is not as efficient as other methods, but here you are asking for "simply impossible", which is tougher
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-06-19T08:53:49.507Z
Comments.Text: Ideally I would like to have a neural network design that can be applied to, for example, the MNIST dataset but where gradient descent would not be possible to train the network.
Comments.UserDisplayName: user248463
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-06-19T09:16:27.673Z
Comments.Text: OK, that's easy. Just use a non-linearity somewhere in the network which cannot be usefully differentiated. A step function in the last layer should do it (or in any other layer). It's artificial, and probably lower-performing than the equivalent network with that layer removed. But it should achieve your goals
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-06-19T17:46:03.007Z
Comments.Text: @DuttaA This is wrong, unless by "basic perceptron learning algorithm" you mean something else. In that case, your terminolgy is at least misleading. The perceptron can and is trained using GD. See e.g. https://stats.stackexchange.com/a/411216/82135.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-06-19T19:11:50.100Z
Comments.Text: @DuttaA A perceptron is a neural network with one input layer and one output layer. That's what a perceptron is.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-06-19T19:42:58.427Z
Comments.Text: @DuttaA The word perceptron is often used to denote such neural network. If the original paper used a Hebbian rule to learn, it is a different story. There are actually several variants of the perceptron.
Comments.UserDisplayName: nbro
Tags: <neural-networks><gradient-descent>
Title: Neural networks when gradient descent is not possible
UserDisplayName: user248463
ThreadText: # Neural networks when gradient descent is not possible
  
  (Asked by: username_0 on 2019-06-19)
  
  I am looking for an example in which it is simply impossible to use some sort of gradient descent to train a neural network. Is this available?
  
  I have read quite some papers about gradient-free optimization tools, but they always use it on a network for which you can also use gradient descent. I want to have a situation in which the only option to train the network is by, for example, a genetic algorithm.
  
  **Tags:** neural-networks, gradient-descent
  > **Comments:**
  > **username_1**: What level of example are you looking for? Creating a neural network architecture that cannot be trained using gradient descent is very easy. Describing a problem that requires finding an optimal function, where it is possible to apply a NN, but impossible to use gradients, is a lot harder. It can be the case that using gradients is not as efficient as other methods, but here you are asking for "simply impossible", which is tougher  
  > **username_0**: Ideally I would like to have a neural network design that can be applied to, for example, the MNIST dataset but where gradient descent would not be possible to train the network.  
  > **username_1**: OK, that's easy. Just use a non-linearity somewhere in the network which cannot be usefully differentiated. A step function in the last layer should do it (or in any other layer). It's artificial, and probably lower-performing than the equivalent network with that layer removed. But it should achieve your goals  
  > **username_2**: @DuttaA This is wrong, unless by "basic perceptron learning algorithm" you mean something else. In that case, your terminolgy is at least misleading. The perceptron can and is trained using GD. See e.g. https://stats.stackexchange.com/a/411216/82135.  
  > **username_2**: @DuttaA A perceptron is a neural network with one input layer and one output layer. That's what a perceptron is.  
  > **username_2**: @DuttaA The word perceptron is often used to denote such neural network. If the original paper used a Hebbian rule to learn, it is a different story. There are actually several variants of the perceptron.  
  
  ---
  
  ## 0 answers
  
  