Body: <p>I ended up working with a neural network <span class="math-container">$N(\cdot)$</span> characterized by the following structure:<span class="math-container">$$N(x)=V[\text{ReLU}(Ax+b)+Cx+d],$$</span>
  with input <span class="math-container">$x \in \mathbb{R}^n$</span> and parameters <span class="math-container">$V\in\mathbb{R}^{1\times m}$</span>, <span class="math-container">$A$</span>, <span class="math-container">$C \in \mathbb{R}^{m \times n}$</span> and <span class="math-container">$b$</span>, <span class="math-container">$d \in \mathbb{R}^m$</span>. Does anyone know what sort of &quot;standard&quot; architecture it represents?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-18T16:13:16.500Z
Comments.Text: Are there any constraints on the parameters? Otherwise, if $A$ and $b$ can vary freely, then $A-C$ and $b-d$ look like they don't do much, because neither $A$ nor $b$ appear anywhere else, which would allow $A-C$ and $b-d$ to be arbitrary with no loss of generalisation if you simply dropped the subtractions.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-18T16:56:15.710Z
Comments.Text: @NeilSlater You're right - just edited, accordingly.
Comments.UserDisplayName: GuidoLaremi
Tags: <neural-networks><machine-learning><deep-learning>
Title: A neural network with peculiar architecture?
UserDisplayName: GuidoLaremi
ThreadText: # A neural network with peculiar architecture?
  
  (Asked by: username_0 on 2023-07-18)
  
  I ended up working with a neural network $N(\cdot)$ characterized by the following structure:$$N(x)=V[\text{ReLU}(Ax+b)+Cx+d],$$
  with input $x \in \mathbb{R}^n$ and parameters $V\in\mathbb{R}^{1\times m}$, $A$, $C \in \mathbb{R}^{m \times n}$ and $b$, $d \in \mathbb{R}^m$. Does anyone know what sort of "standard" architecture it represents?
  
  **Tags:** neural-networks, machine-learning, deep-learning
  
  > **Comments:**
  > **username_1**: Are there any constraints on the parameters? Otherwise, if $A$ and $b$ can vary freely, then $A-C$ and $b-d$ look like they don't do much, because neither $A$ nor $b$ appear anywhere else, which would allow $A-C$ and $b-d$ to be arbitrary with no loss of generalisation if you simply dropped the subtractions.  
  > **username_0**: @username_1 You're right - just edited, accordingly.  
  
  ---
  
  ## 0 answers
  
  