Answers.Body: <p>You seem to be seeking an implementation of a Residual Neural Network (<a href="https://en.m.wikipedia.org/wiki/Residual_neural_network" rel="nofollow noreferrer">https://en.m.wikipedia.org/wiki/Residual_neural_network</a>), or ResNET for short. If you want some premade networks, the module tf.keras.applications.resnet from tensorflow (do check TF's documentation) might help you.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-17T14:34:11.020Z
Answers.LastActivityDate: 2021-03-17T14:34:11.020Z
Answers.UserDisplayName: Lucas Pelizzari
Answers.Body: <p>With the existing frameworks <code>PyTorch</code>, <code>Tensorflow</code> you can easily implement this functionality, by keeping some of the intermediate computations inside <code>forward</code> or <code>call</code> method and passing them as an input to the given layer. For example:</p>
  <pre><code>x[i] = layer[i-1](x[i - 1])
  ...
  x[j] = layer[j-1](x[j - 1] + x[i]) # resnet-like skip connection
  or
  x[j] = layer[j-1](concat(x[j - 1], x[i])) # densenet-like skip connection
  </code></pre>
  <p>However, if you are asking whether, there is some more educated way, or a constructor of this functionality, as far as I know, it is not implemented in these frameworks.</p>
  <p>In case you want to have a computational graph of some arbitrary DAG structure, you will need to create some structures inside you NN class in order to know, where to keep activations in order to pass them further as skip connection and where to sum/concatenate them with the output of some other layer.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-04T06:30:32.050Z
Answers.LastActivityDate: 2021-06-04T06:30:32.050Z
Answers.UserDisplayName: spiridon_the_sun_rotator
Answers.Body: <p>Mentioned frameworks don't restrict you with linear layers sequence, you could do any acyclic sequence. I.e. very popular resnet architecture based on skip-connections that jumps over the layers.</p>
  <p>I.e. simple example on pytorch</p>
  <pre><code>import torch
  from torch import nn
  import torch.nn.functional as F
  
  class Custom(nn.Module):
      def __init__(self):
          super().__init__()
          self.a = nn.Parameter(torch.tensor(1.))
          self.b = nn.Parameter(torch.tensor(2.))
          self.c = nn.Parameter(torch.tensor(3.))
  
  
      def forward(self, x):
          x1 = F.relu(self.a * x)
          # take note, we skip
          out = F.relu(self.b * x1 + self.c*x)
          return out
  
  model = Custom()
  print('before', model.a)
  # You could do pretty much the same training
  x = torch.tensor([2])
  criterion = nn.MSELoss()
  optimizer = torch.optim.SGD(model.parameters(), lr=1e-2)
  prediction = model(x)
  loss = criterion(prediction, torch.tensor([20.]))
  loss.backward()
  optimizer.step()
  print('after', model.a) 
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-17T14:42:22.700Z
Answers.LastActivityDate: 2021-03-17T14:42:22.700Z
Answers.UserDisplayName: Kirill Fedyanin
Body: <p>Hello I am currently doing research on the effect of altering a neural network's structure. Particularly I am investigating what affect would putting a random DAG (directed acyclic graph) in the hidden layer of a network instead of a usual fully connected bipartite graph.</p>
  <p>For instance my neural network would look something like this:
  <a href="https://i.sstatic.net/xa5wm.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/xa5wm.png" alt="enter image description here" /></a></p>
  <p>Basically I want the ability to create any structure in my hidden layer as long as it remains a DAG [add any edge between any node regardless of layers]. I have tried creating my own library to do so but it proved to be much more tedious than anticipated therefore I am looking for ways to do this on existing libraries such as Keras, pytorch, or tensorflow.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-29T10:56:27.280Z
Comments.Text: You may want to expand the diagram to make it clearer how arbitrary the connections are allowed to be. The answer about residual networks might not be enough for you if you want to work at the level of individual neurons
Comments.UserDisplayName: Neil Slater
Tags: <neural-networks><tensorflow><keras><pytorch><implementation>
Title: How can I model any structure for a neural network?
UserDisplayName: Rami Hoteit
ThreadText: # How can I model any structure for a neural network?
  
  (Asked by: username_0 on 2021-03-17)
  
  Hello I am currently doing research on the effect of altering a neural network's structure. Particularly I am investigating what affect would putting a random DAG (directed acyclic graph) in the hidden layer of a network instead of a usual fully connected bipartite graph.
  
  For instance my neural network would look something like this:
  [![enter image description here](https://i.sstatic.net/xa5wm.png)](https://i.sstatic.net/xa5wm.png)
  
  Basically I want the ability to create any structure in my hidden layer as long as it remains a DAG [add any edge between any node regardless of layers]. I have tried creating my own library to do so but it proved to be much more tedious than anticipated therefore I am looking for ways to do this on existing libraries such as Keras, pytorch, or tensorflow.
  
  **Tags:** neural-networks, tensorflow, keras, pytorch, implementation
  
  > **Comments:**
  > **username_1**: You may want to expand the diagram to make it clearer how arbitrary the connections are allowed to be. The answer about residual networks might not be enough for you if you want to work at the level of individual neurons  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_2 on 2021-03-17. Score: 1
  
  You seem to be seeking an implementation of a Residual Neural Network (<https://en.m.wikipedia.org/wiki/Residual_neural_network>), or ResNET for short. If you want some premade networks, the module tf.keras.applications.resnet from tensorflow (do check TF's documentation) might help you.
  
  
  ---
  
  ### Answer by username_3 on 2021-06-04. Score: 0
  
  With the existing frameworks `PyTorch`, `Tensorflow` you can easily implement this functionality, by keeping some of the intermediate computations inside `forward` or `call` method and passing them as an input to the given layer. For example:
  
  ```
  x[i] = layer[i-1](x[i - 1])
  ...
  x[j] = layer[j-1](x[j - 1] + x[i]) # resnet-like skip connection
  or
  x[j] = layer[j-1](concat(x[j - 1], x[i])) # densenet-like skip connection
  
  ```
  
  However, if you are asking whether, there is some more educated way, or a constructor of this functionality, as far as I know, it is not implemented in these frameworks.
  
  In case you want to have a computational graph of some arbitrary DAG structure, you will need to create some structures inside you NN class in order to know, where to keep activations in order to pass them further as skip connection and where to sum/concatenate them with the output of some other layer.
  
  
  ---
  
  ### Answer by username_4 on 2021-03-17. Score: 0
  
  Mentioned frameworks don't restrict you with linear layers sequence, you could do any acyclic sequence. I.e. very popular resnet architecture based on skip-connections that jumps over the layers.
  
  I.e. simple example on pytorch
  
  ```
  import torch
  from torch import nn
  import torch.nn.functional as F
  
  class Custom(nn.Module):
      def __init__(self):
          super().__init__()
          self.a = nn.Parameter(torch.tensor(1.))
          self.b = nn.Parameter(torch.tensor(2.))
          self.c = nn.Parameter(torch.tensor(3.))
  
      def forward(self, x):
          x1 = F.relu(self.a * x)
          # take note, we skip
          out = F.relu(self.b * x1 + self.c*x)
          return out
  
  model = Custom()
  print('before', model.a)
  # You could do pretty much the same training
  x = torch.tensor([2])
  criterion = nn.MSELoss()
  optimizer = torch.optim.SGD(model.parameters(), lr=1e-2)
  prediction = model(x)
  loss = criterion(prediction, torch.tensor([20.]))
  loss.backward()
  optimizer.step()
  print('after', model.a) 
  
  ```
  
  
  ---
  
  