Answers.Body: <p>You mentioned dynamics, so I assume you have some kind of vectorised state <span class="math-container">$S$</span> and the model should output <span class="math-container">$S'$</span> in the same space. A transformer will do this.</p>
  <p>A special case of UNet which outputs the same size tensor as its input would also work, if your input is image-like (amenable to convolution).</p>
  <p>An RNN would also work, but its recursive architecture wouldn't be the reason.</p>
  <p>A neural network is essentially just a function, and they have very flexible domains and codomains.</p>
  <p>If you want <span class="math-container">$M(M(x))$</span> to be well defined, you just need <span class="math-container">$M$</span> to map <span class="math-container">$X \rightarrow X$</span> which basically any neural network can do. The best architecture mostly depends on <span class="math-container">$X$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-19T15:04:05.957Z
Answers.Text: In my experiments (Lorenz system), a fully-connected NN completely fails. What stops the model from mapping inputs in the $X$ domain to the $Y$ domain and inputs in the $Y$ domain to the $X$ domain?
Answers.UserDisplayName: user572780
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-19T16:07:31.837Z
Answers.Text: @user572780 - depends on how you are using "domain", I think the answer here is focused on dimensionality only. And you cannot write a neural network that converts between different dimensionalities by itself automatically e.g. if it receives $\mathbb{R}^n$ it should output in  $\mathbb{R}^m$ and vice-versa  . . . although you could add supporting code for this and use the same neural network to attempt both tasks with redundancy in unused inputs or outputs depending which step was being processed.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-19T16:22:58.680Z
Answers.Text: @NeilSlater By "domain" I mean an open connected set—my question was not to do with dimensionality.
Answers.UserDisplayName: user572780
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-19T16:35:23.753Z
Answers.Text: @user572780 Then you may have a problem in a single model if you need specific mappings and X/Y domains overlap. But if you don't care how X/Y mapping works, or if the domains do not overlap, then a single NN model should be fine. Or, as a special case, if where X and Y domains overlap, then the mapping is always the exact same vector in X and Y.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-19T17:00:09.403Z
Answers.Text: @NeilSlater I think there was a misunderstanding. Essentially, I am applying a NN twice so that the intermediate results follow the dynamics of the problem. It seems that a fully-connected NN cannot do this.
Answers.UserDisplayName: user572780
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-19T17:12:22.453Z
Answers.Text: @user572780 Fully connected networks are known to be universal approximators. We just use different architectures to make computation feasible. How was your fully connected network implemented?
Answers.UserDisplayName: Venna Banana
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-19T17:36:23.590Z
Answers.Text: Also, suppose f has domain X but f's codomain Y is not a subset of X. Then there is some x in X such that f(x) is not in X, which means f(f(x)) is not defined. So even if your network maps X to Y and Y to X, outputs of f(x) still need to be in X. Hence mapping X to X. But that is not really the point. It was rather that any network with the same input and output space can work in theory, and a huge number of networks satisfy this.
  Besides, your issue is probably that your loss says nothing about M(x) matching intermediary state. You can't expect NN to do something you didn't train it to do.
Answers.UserDisplayName: Venna Banana
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-19T22:48:10.300Z
Answers.Text: @VennaBanana It is just a regular fully-connected neural network—not sure what you mean by how it was implemented. The loss indeed says nothing about M(x), but it does say something about M(M(x)) and it performs poorly in that scenario. I double-checked this by implementing a single network (L = |M(x) - y|) with repeated fully-connected layers: fc1 -> fc2 -> fc1 -> fc2.
Answers.UserDisplayName: user572780
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-20T07:48:29.087Z
Answers.Text: @user572780 Yes a fully connected nn can do that. It can be used in RL to predict state progression, is used in image generators that progressively remove noise (e.g. diffusion models) and there are physics based predictions such as turbulence models. All of these however will have same domain input and output, perhaps with additional conditioning input
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-20T10:09:58.170Z
Answers.Text: @NeilSlater Do they use fully connected NNs in RL/image generation?
Answers.UserDisplayName: user572780
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-20T10:24:19.953Z
Answers.Text: @user572780 In RL, yes, typically, although it will depend on the state vector type, and it is not a common/standard requirement to predict next state. In image generation, it is typically CNNs, but there's no meaningful difference between CNNs and FCNs in terms of how they behave in your $M(M(x))$ composition - what you care about is having an output structure and values that are also suitable as an input, so typically the same domain $M(x): \mathcal{X} \rightarrow \mathcal{X}$
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-20T10:36:07.017Z
Answers.Text: @user572780 As per this answer, the internal architecture choice once input/output are set is not about the composition problem that you are focusing on in the question. It depends on the nature of $x$ and the problem space you are trying to solve. Your need to be able to feed back the output as a new input is not unusual. But perhaps your problem setup is unusual in other ways?
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-20T11:13:25.250Z
Answers.Text: @NeilSlater Of course in my experiments, the input is the same shape as the output.
Answers.UserDisplayName: user572780
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-19T14:05:24.537Z
Answers.LastActivityDate: 2023-07-19T14:05:24.537Z
Answers.UserDisplayName: Venna Banana
Body: <p>I come from a math background, so I am not up-to-date with machine learning literature.</p>
  <p>For the purpose of learning dynamics, I would like to train a model to minimize the following loss:
  <span class="math-container">$$\mathcal{L} = |\mathcal{M}(\mathcal{M}(x)) - y|,$$</span>
  more generally, there can be <span class="math-container">$k \geq 2$</span> recursive applications of the model <span class="math-container">$\mathcal{M}$</span>.</p>
  <p>From my experimentation, regular fully-connected neural networks perform poorly—a single application of <span class="math-container">$\mathcal{M}$</span> performs better and intermediate results do not correspond to intermediate data.</p>
  <p>Are there other architectures that are better suited for this problem? Maybe graph-based models or the U-net from DDPMs?</p>
  <p>Edit: Just a note on RNNs (I could be completely wrong): Although RNNs are used for recursive applications, I don't think they are what I am looking for. RNNs accept a sequence of inputs, whereas my input is a single tensor. Also, going from one hidden state to the next corresponds to a matrix multiplication which seems not very expressive(?).</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-22T00:50:01.583Z
Comments.Text: In StyleGAN there is two steps mapping and synthesis. Since the mapping step has same amount of inputs as outputs, I once had an idea to run it twice using its own output as input. And let's just say the results can be interesting sometimes if you do something you do not have any idea about the outcome.
Comments.UserDisplayName: harism
Tags: <machine-learning><deep-learning><graph-neural-networks><architecture><u-net>
Title: Which models can be applied recursively?
UserDisplayName: user572780
ThreadText: # Which models can be applied recursively?
  
  (Asked by: username_0 on 2023-07-19)
  
  I come from a math background, so I am not up-to-date with machine learning literature.
  
  For the purpose of learning dynamics, I would like to train a model to minimize the following loss:
  $$\mathcal{L} = |\mathcal{M}(\mathcal{M}(x)) - y|,$$
  more generally, there can be $k \geq 2$ recursive applications of the model $\mathcal{M}$.
  
  From my experimentation, regular fully-connected neural networks perform poorly—a single application of $\mathcal{M}$ performs better and intermediate results do not correspond to intermediate data.
  
  Are there other architectures that are better suited for this problem? Maybe graph-based models or the U-net from DDPMs?
  
  Edit: Just a note on RNNs (I could be completely wrong): Although RNNs are used for recursive applications, I don't think they are what I am looking for. RNNs accept a sequence of inputs, whereas my input is a single tensor. Also, going from one hidden state to the next corresponds to a matrix multiplication which seems not very expressive(?).
  
  **Tags:** machine-learning, deep-learning, graph-neural-networks, architecture, u-net
  > **Comments:**
  > **username_1**: In StyleGAN there is two steps mapping and synthesis. Since the mapping step has same amount of inputs as outputs, I once had an idea to run it twice using its own output as input. And let's just say the results can be interesting sometimes if you do something you do not have any idea about the outcome.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-07-19. Score: 1
  
  You mentioned dynamics, so I assume you have some kind of vectorised state $S$ and the model should output $S'$ in the same space. A transformer will do this.
  
  A special case of UNet which outputs the same size tensor as its input would also work, if your input is image-like (amenable to convolution).
  
  An RNN would also work, but its recursive architecture wouldn't be the reason.
  
  A neural network is essentially just a function, and they have very flexible domains and codomains.
  
  If you want $M(M(x))$ to be well defined, you just need $M$ to map $X \rightarrow X$ which basically any neural network can do. The best architecture mostly depends on $X$.
  
  > **Comments:**
  > **username_0**: In my experiments (Lorenz system), a fully-connected NN completely fails. What stops the model from mapping inputs in the $X$ domain to the $Y$ domain and inputs in the $Y$ domain to the $X$ domain?  
  > **username_3**: @user572780 - depends on how you are using "domain", I think the answer here is focused on dimensionality only. And you cannot write a neural network that converts between different dimensionalities by itself automatically e.g. if it receives $\mathbb{R}^n$ it should output in  $\mathbb{R}^m$ and vice-versa  . . . although you could add supporting code for this and use the same neural network to attempt both tasks with redundancy in unused inputs or outputs depending which step was being processed.  
  > **username_0**: @NeilSlater By "domain" I mean an open connected set—my question was not to do with dimensionality.  
  > **username_3**: @user572780 Then you may have a problem in a single model if you need specific mappings and X/Y domains overlap. But if you don't care how X/Y mapping works, or if the domains do not overlap, then a single NN model should be fine. Or, as a special case, if where X and Y domains overlap, then the mapping is always the exact same vector in X and Y.  
  > **username_0**: @NeilSlater I think there was a misunderstanding. Essentially, I am applying a NN twice so that the intermediate results follow the dynamics of the problem. It seems that a fully-connected NN cannot do this.  
  > **username_2**: @user572780 Fully connected networks are known to be universal approximators. We just use different architectures to make computation feasible. How was your fully connected network implemented?  
  > **username_2**: Also, suppose f has domain X but f's codomain Y is not a subset of X. Then there is some x in X such that f(x) is not in X, which means f(f(x)) is not defined. So even if your network maps X to Y and Y to X, outputs of f(x) still need to be in X. Hence mapping X to X. But that is not really the point. It was rather that any network with the same input and output space can work in theory, and a huge number of networks satisfy this. Besides, your issue is probably that your loss says nothing about M(x) matching intermediary state. You can't expect NN to do something you didn't train it to do.  
  > **username_0**: @VennaBanana It is just a regular fully-connected neural network—not sure what you mean by how it was implemented. The loss indeed says nothing about M(x), but it does say something about M(M(x)) and it performs poorly in that scenario. I double-checked this by implementing a single network (L = |M(x) - y|) with repeated fully-connected layers: fc1 -> fc2 -> fc1 -> fc2.  
  > **username_3**: @user572780 Yes a fully connected nn can do that. It can be used in RL to predict state progression, is used in image generators that progressively remove noise (e.g. diffusion models) and there are physics based predictions such as turbulence models. All of these however will have same domain input and output, perhaps with additional conditioning input  
  > **username_0**: @NeilSlater Do they use fully connected NNs in RL/image generation?  
  > **username_3**: @user572780 In RL, yes, typically, although it will depend on the state vector type, and it is not a common/standard requirement to predict next state. In image generation, it is typically CNNs, but there's no meaningful difference between CNNs and FCNs in terms of how they behave in your $M(M(x))$ composition - what you care about is having an output structure and values that are also suitable as an input, so typically the same domain $M(x): \mathcal{X} \rightarrow \mathcal{X}$  
  > **username_3**: @user572780 As per this answer, the internal architecture choice once input/output are set is not about the composition problem that you are focusing on in the question. It depends on the nature of $x$ and the problem space you are trying to solve. Your need to be able to feed back the output as a new input is not unusual. But perhaps your problem setup is unusual in other ways?  
  > **username_0**: @NeilSlater Of course in my experiments, the input is the same shape as the output.  
  
  ---
  
  