Answers.Body: <p>It is difficult to prove a negative, but I do not think there is any classifier (neural network or otherwise) that fully matches to your idea.</p>
  <p>I suspect that you will not be able to take the idea of random connections and loops at run time, and make a useful classifier out of it. That's not to say the idea is completely without merit, sometimes it is good to explore blue sky ideas and just see what happens. However, I think it might be a frustrating excercise to build anything on top of your idea without some basic foundation work first. I recommend that you look into the <a href="https://towardsdatascience.com/introduction-to-logistic-regression-66248243c148" rel="nofollow noreferrer">theory and implementation of logistic regression</a> as a starting point, which is a good stepping stone to understanding neural networks.</p>
  <p>There are some neural network components and architectures that make use of random behaviour at the activation level:</p>
  <ul>
  <li><p><a href="https://www.cs.toronto.edu/%7Ehinton/absps/JMLRdropout.pdf" rel="nofollow noreferrer">Dropout</a>. This is a method used during training which zeroes outputs from randomly selected neurons. It often gives an effective boost to neural network stability (acting to prevent overfitting to input data) and can improve accuracy of classifiers too due to behaving similarly to having multiple simpler classifiers.</p>
  </li>
  <li><p><a href="https://en.wikipedia.org/wiki/Boltzmann_machine" rel="nofollow noreferrer">Boltzmann machines</a>, and <a href="https://en.wikipedia.org/wiki/Restricted_Boltzmann_machine" rel="nofollow noreferrer">restricted Boltzmann machines (RBMs)</a> output 0 or 1 randomly from each &quot;neuron&quot; unit, with the probability decided by sum of inputs. They are used to create <em>generative</em> models, not classifiers though. Another difference is that the randomness is applied both during training and during inference, whilst dropout is most often applied to augment training. Early on in the days of deep learning, RBMs were used to pre-train layers in a deep neural network. This was effective, but other simpler methods were discovered later and are nowadays preferred in most cases.</p>
  </li>
  <li><p>A variant of dropout call called <a href="https://arxiv.org/pdf/1506.02142.pdf" rel="nofollow noreferrer">Monte Carlo dropout</a> is used at inference time. This can be used to measure <em>uncertainty</em> in a model's individual predictions, which is otherwise hard to obtain.</p>
  </li>
  <li><p>Although not quite as freeform as your random connections on a per neuron basis. If you <a href="https://medium.com/@bingobee01/a-review-of-dropout-as-applied-to-rnns-72e79ecd5b7b" rel="nofollow noreferrer">applied dropout to a recurrent neural network</a>, that might be quite close to your idea, because the existence of loops between neurons in each time step would be random. This could be applied in language modelling and classifiers for sequence data. The same motivations apply here as for dropout in simpler feed forward classifiers - it can in theory make a classifier more robust against noise in the inputs and more accurate.</p>
  </li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-29T18:29:46.463Z
Answers.LastActivityDate: 2020-08-30T10:33:18.080Z
Answers.LastEditDate: 2020-08-30T10:33:18.080Z
Answers.UserDisplayName: Neil Slater
Body: <p>I am trying to make a classifier.</p>
  <p>I am new to AI (even if I know the definition and all such a bit) , and also I have no idea of how to implement it properly by myself even if I know a bit of Python coding (in fact, I am fifteen years old !), but my passion for this has made me ask this (silly, probably) question.</p>
  <p>Are there neural networks where nodes are randomly selected from among a set of nodes (in random orders and a random number of times)? I know this is from ML (or maybe deep learning, I suppose), but I have no idea how to recognize such a thing from the presently available algorithms. It will be great if you all could help me, because I am preparing to release an API for programming a model which I call the 'Insane Mind' on GitHub, and I want some help to know if my effort was fruitless.</p>
  <p>And for reference, here's the code :</p>
  <pre><code>from math import *
  from random import *
   
  class MachineError(Exception):
      '''standard exception in the API'''
      def __init__(self, stmt):
          self.stmt = stmt
  def sig(x):
      '''Sigmoid function'''
      return (exp(x) + 1)/exp(x)
  
  class Graviton:
      def __init__(self, weight, marker):
          '''Basic unit in 'Insane Mind' algorithm
             -------------------------------------
             Graviton simply refers to a node in the algorithm.
             I call it graviton because of the fact that it applies a weight
             on the input to transform it, besides using the logistic function '''
          self.weight = weight # Weight factor of the graviton
          self.marker = marker # Marker to help in sorting
          self.input = 0 # Input to the graviton
          self.output = 0 # Output of the graviton
          self.derivative = 0 # Derivative of the output
  
      def process(self, input_to_machine):
          '''processes the input (a bit of this is copied from the backprop algorithm'''
          self.input = input_to_machine
          self.output = (sig(self.weight * self.input) - 1)/(self.marker + 1)
          self.derivative = (sig(self.input * self.weight) - 1) * self.input *self.output * (1- self.output) 
          return self.output
      
      def get_derivative_at_input(self):
          '''returns the derivative of the output'''
          return self.derivative
  
      def correct_self(self, learning_rate, error):
          '''edits the weight'''
          self.weight += -1 * error * learning_rate * self.get_derivative_at_input() * self.weight
          
  class Insane_Mind:
  
      def __init__(self, number_of_nodes):
          '''initialiser for Insane_Mind class.
             arguments : number_of_nodes : the number of nodes you want in the model'''
          self.system = [Graviton(random(),i) for i in range(number_of_nodes)] # the actual system
          self.system_size = number_of_nodes # number of nodes , or 'system size'
          
      def  output_sys(self, input_to_sys):
          '''system output'''
          self.output = input_to_sys
          for i in range(self.system_size):
              self.output = self.system[randint(0,self.system_size - 1 )].process(self.output)
          return self.output
      
      def train(self, learning_rate, wanted):
          '''trains the system'''
          self.cloned = [] # an array to keep the sorted elements during the sorting process below
          order = [] # the array to make out the order of arranging the nodes
          temp = {} # a temporary dictionary to pick the nodes from
          for graviton in self.system:
              temp.update({str(graviton.derivative): graviton.marker})
          order = sorted(temp)
          i = 0
          error = wanted - self.output
          for value in order:
              self.cloned.append(self.system[temp[value]])
              self.cloned[i].correct_self(learning_rate, error)
              error *= self.cloned[i].derivative
              i += 1
          self.system = self.cloned
  </code></pre>
  <p>Sorry for not using that <code>MachineError</code> exception anywhere in my code (I will use it when I am able to deploy this API).</p>
  <p>To tell more about this algorithm, this gives randomized outputs (as if guessing). The number of guesses vary from 1 (for a system with one node), 2 (for two nodes) and so on to an infinite number of guesses for an infinite number of nodes.</p>
  <p>Also, I wanna try and find how much it can be of use (if this is something that has never been discovered, if it is something that can find a good place in the world of ML or Deep Learning) and where it can be used.</p>
  <p>Thanks in advance.</p>
  <p>Criticisms (with a clear reason) are also accepted.</p>
  
Tags: <neural-networks><machine-learning>
Title: Are there neural networks where nodes are randomly selected from among a set of nodes (in random orders and a random number of times)?
UserDisplayName: Spectre
ThreadText: # Are there neural networks where nodes are randomly selected from among a set of nodes (in random orders and a random number of times)?
  
  (Asked by: username_0 on 2020-08-29)
  
  I am trying to make a classifier.
  
  I am new to AI (even if I know the definition and all such a bit) , and also I have no idea of how to implement it properly by myself even if I know a bit of Python coding (in fact, I am fifteen years old !), but my passion for this has made me ask this (silly, probably) question.
  
  Are there neural networks where nodes are randomly selected from among a set of nodes (in random orders and a random number of times)? I know this is from ML (or maybe deep learning, I suppose), but I have no idea how to recognize such a thing from the presently available algorithms. It will be great if you all could help me, because I am preparing to release an API for programming a model which I call the 'Insane Mind' on GitHub, and I want some help to know if my effort was fruitless.
  
  And for reference, here's the code :
  
  ```
  from math import *
  from random import *
   
  class MachineError(Exception):
      '''standard exception in the API'''
      def __init__(self, stmt):
          self.stmt = stmt
  def sig(x):
      '''Sigmoid function'''
      return (exp(x) + 1)/exp(x)
  
  class Graviton:
      def __init__(self, weight, marker):
          '''Basic unit in 'Insane Mind' algorithm
             -------------------------------------
             Graviton simply refers to a node in the algorithm.
             I call it graviton because of the fact that it applies a weight
             on the input to transform it, besides using the logistic function '''
          self.weight = weight # Weight factor of the graviton
          self.marker = marker # Marker to help in sorting
          self.input = 0 # Input to the graviton
          self.output = 0 # Output of the graviton
          self.derivative = 0 # Derivative of the output
  
      def process(self, input_to_machine):
          '''processes the input (a bit of this is copied from the backprop algorithm'''
          self.input = input_to_machine
          self.output = (sig(self.weight * self.input) - 1)/(self.marker + 1)
          self.derivative = (sig(self.input * self.weight) - 1) * self.input *self.output * (1- self.output) 
          return self.output
      
      def get_derivative_at_input(self):
          '''returns the derivative of the output'''
          return self.derivative
  
      def correct_self(self, learning_rate, error):
          '''edits the weight'''
          self.weight += -1 * error * learning_rate * self.get_derivative_at_input() * self.weight
          
  class Insane_Mind:
  
      def __init__(self, number_of_nodes):
          '''initialiser for Insane_Mind class.
             arguments : number_of_nodes : the number of nodes you want in the model'''
          self.system = [Graviton(random(),i) for i in range(number_of_nodes)] # the actual system
          self.system_size = number_of_nodes # number of nodes , or 'system size'
          
      def  output_sys(self, input_to_sys):
          '''system output'''
          self.output = input_to_sys
          for i in range(self.system_size):
              self.output = self.system[randint(0,self.system_size - 1 )].process(self.output)
          return self.output
      
      def train(self, learning_rate, wanted):
          '''trains the system'''
          self.cloned = [] # an array to keep the sorted elements during the sorting process below
          order = [] # the array to make out the order of arranging the nodes
          temp = {} # a temporary dictionary to pick the nodes from
          for graviton in self.system:
              temp.update({str(graviton.derivative): graviton.marker})
          order = sorted(temp)
          i = 0
          error = wanted - self.output
          for value in order:
              self.cloned.append(self.system[temp[value]])
              self.cloned[i].correct_self(learning_rate, error)
              error *= self.cloned[i].derivative
              i += 1
          self.system = self.cloned
  
  ```
  
  Sorry for not using that `MachineError` exception anywhere in my code (I will use it when I am able to deploy this API).
  
  To tell more about this algorithm, this gives randomized outputs (as if guessing). The number of guesses vary from 1 (for a system with one node), 2 (for two nodes) and so on to an infinite number of guesses for an infinite number of nodes.
  
  Also, I wanna try and find how much it can be of use (if this is something that has never been discovered, if it is something that can find a good place in the world of ML or Deep Learning) and where it can be used.
  
  Thanks in advance.
  
  Criticisms (with a clear reason) are also accepted.
  
  **Tags:** neural-networks, machine-learning
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-08-29. Score: 2 | Accepted answer
  
  It is difficult to prove a negative, but I do not think there is any classifier (neural network or otherwise) that fully matches to your idea.
  
  I suspect that you will not be able to take the idea of random connections and loops at run time, and make a useful classifier out of it. That's not to say the idea is completely without merit, sometimes it is good to explore blue sky ideas and just see what happens. However, I think it might be a frustrating excercise to build anything on top of your idea without some basic foundation work first. I recommend that you look into the [theory and implementation of logistic regression](https://towardsdatascience.com/introduction-to-logistic-regression-66248243c148) as a starting point, which is a good stepping stone to understanding neural networks.
  
  There are some neural network components and architectures that make use of random behaviour at the activation level:
  
  * [Dropout](https://www.cs.toronto.edu/%7Ehinton/absps/JMLRdropout.pdf). This is a method used during training which zeroes outputs from randomly selected neurons. It often gives an effective boost to neural network stability (acting to prevent overfitting to input data) and can improve accuracy of classifiers too due to behaving similarly to having multiple simpler classifiers.
  * [Boltzmann machines](https://en.wikipedia.org/wiki/Boltzmann_machine), and [restricted Boltzmann machines (RBMs)](https://en.wikipedia.org/wiki/Restricted_Boltzmann_machine) output 0 or 1 randomly from each "neuron" unit, with the probability decided by sum of inputs. They are used to create *generative* models, not classifiers though. Another difference is that the randomness is applied both during training and during inference, whilst dropout is most often applied to augment training. Early on in the days of deep learning, RBMs were used to pre-train layers in a deep neural network. This was effective, but other simpler methods were discovered later and are nowadays preferred in most cases.
  * A variant of dropout call called [Monte Carlo dropout](https://arxiv.org/pdf/1506.02142.pdf) is used at inference time. This can be used to measure *uncertainty* in a model's individual predictions, which is otherwise hard to obtain.
  * Although not quite as freeform as your random connections on a per neuron basis. If you [applied dropout to a recurrent neural network](https://medium.com/@bingobee01/a-review-of-dropout-as-applied-to-rnns-72e79ecd5b7b), that might be quite close to your idea, because the existence of loops between neurons in each time step would be random. This could be applied in language modelling and classifiers for sequence data. The same motivations apply here as for dropout in simpler feed forward classifiers - it can in theory make a classifier more robust against noise in the inputs and more accurate.
  
  
  ---
  
  