Body: <p>Communication requires energy, and using energy requires communication. According to Shannon, the entropy value of a piece of information provides an absolute limit on the shortest possible average length of a message without losing information as it is transmitted. (<a href="https://towardsdatascience.com/entropy-the-pillar-of-both-thermodynamics-and-information-theory-138d6e4872fa" rel="nofollow noreferrer">https://towardsdatascience.com/entropy-the-pillar-of-both-thermodynamics-and-information-theory-138d6e4872fa</a>)</p>
  <p>I don't know whether Neural Network actually deals with information flow or not. This information flow is taken from the idea of entropy. Since I haven't found any paper or ideas based on the law of energy for neural networks. The law of energy states that energy can neither be created nor destroyed. If it is creating information (energy) (e.g. in the case of a generative model), then some information may be lost while updating weights. How is Neural Network ensuring this energy conservation?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-22T19:47:39.607Z
Comments.Text: Please, you should still focus on a more specific question. Here you're raising several non-trivial issues. We could start from the question: "Is there an information-theoretic view of neural networks", to which you can find the answer [here](https://ai.stackexchange.com/a/20574/2444). Your question about the "conservation of energy" is apparently based on the assumption that there's information in the NNs, but you don't even know what that means, so your post could lead to a big answer. So, again, you should probably start by reading that answer, then formulate your question again.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-03T01:36:20.420Z
Comments.Text: I personally find this to be an interesting question b/c cost of computation is extremely important (as is waste heat in processing.)
Comments.UserDisplayName: DukeZhou
Tags: <neural-networks><generative-model><information-theory><minimum-description-length><entropy>
Title: How does NN follows law of energy conservation?
UserDisplayName: Satyam Kumar
ThreadText: # How does NN follows law of energy conservation?
  
  (Asked by: username_0 on 2020-11-21)
  
  Communication requires energy, and using energy requires communication. According to Shannon, the entropy value of a piece of information provides an absolute limit on the shortest possible average length of a message without losing information as it is transmitted. (<https://towardsdatascience.com/entropy-the-pillar-of-both-thermodynamics-and-information-theory-138d6e4872fa>)
  
  I don't know whether Neural Network actually deals with information flow or not. This information flow is taken from the idea of entropy. Since I haven't found any paper or ideas based on the law of energy for neural networks. The law of energy states that energy can neither be created nor destroyed. If it is creating information (energy) (e.g. in the case of a generative model), then some information may be lost while updating weights. How is Neural Network ensuring this energy conservation?
  
  **Tags:** neural-networks, generative-model, information-theory, minimum-description-length, entropy
  > **Comments:**
  > **username_1**: Please, you should still focus on a more specific question. Here you're raising several non-trivial issues. We could start from the question: "Is there an information-theoretic view of neural networks", to which you can find the answer [here](https://ai.stackexchange.com/a/20574/2444). Your question about the "conservation of energy" is apparently based on the assumption that there's information in the NNs, but you don't even know what that means, so your post could lead to a big answer. So, again, you should probably start by reading that answer, then formulate your question again.  
  > **username_2**: I personally find this to be an interesting question b/c cost of computation is extremely important (as is waste heat in processing.)  
  
  ---
  
  ## 0 answers
  
  