Answers.Body: <p>I did some testing on nanoGPT (<a href="https://github.com/karpathy/nanoGPT" rel="nofollow noreferrer">https://github.com/karpathy/nanoGPT</a>) using triangular weights matrices instead of full one and the result were interesting : using triangular matrices seems to have a very limited impact on the learning (for gpt2).
  <a href="https://i.sstatic.net/x9oHv.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/x9oHv.png" alt="enter image description here" /></a></p>
  <p>(black and orange curve are the learning for triangular matrix and the pink one is the full linear one)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-30T19:31:35.033Z
Answers.Text: I would bet It perform the same as dropout 0.5
Answers.UserDisplayName: CoffeDeveloper
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-01T09:39:19.137Z
Answers.Text: My bet (but i'm not sure) that it will not be the case ... dropout randomly choose some activation to shut down. With triangular matrix you have a deterministic behaviour.
  Also the advantage of triangular is that you can hope to find better gpu kernel to do your triangular matrix multiplication (but I will not do that, I will not code in cuda :)).
Answers.UserDisplayName: Adrien Forbu
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-04T12:49:48.593Z
Answers.LastActivityDate: 2023-07-04T12:49:48.593Z
Answers.UserDisplayName: Adrien Forbu
Answers.Body: <p>Constraining weights in a neural network, either a hard constraint like your idea, or a soft constraint that adds to the cost function, is often a form of regularisation. By limiting the form of acceptable solutions, regularisation makes it harder, often impossible, to exactly match training data - this is a good thing because what you most often want to learn is a <em>generalisation</em> of the examples that works well with new data. Forcing a machine learning process into a space where it can approach but never reach a perfect solution can make good generalisation.</p>
  <p>That means you might use such an idea to combat overfitting. The utility of the idea will vary depending on the problem. There may be no general use for it, because it's never better than other well established approaches. But there could be some problems it is particularly suitable for. You would have to try it and see.</p>
  <p>The main problem I see with the approach in general is that it's all or nothing. You cannot easily have an arbitrary fraction of the diagonal mask.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-04T15:43:54.867Z
Answers.LastActivityDate: 2023-07-04T15:43:54.867Z
Answers.UserDisplayName: Neil Slater
Body: <p>I need to implement a rule and have defined a lower triangular boolean mask for the weights that I want to keep static for a zero value. In which condition triangular weight matrix will  be used?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-20T17:54:08.283Z
Comments.Text: Triangular weight matrix could be much weaker to approximate complicated functions, unless your computing resources are limited you may try it.
Comments.UserDisplayName: cinch
Tags: <neural-networks><deep-learning><deep-neural-networks>
Title: Does it make sense to build a two-layer neural network with a triangular weight matrix?
UserDisplayName: Bouarfa Mahi
ThreadText: # Does it make sense to build a two-layer neural network with a triangular weight matrix?
  
  (Asked by: username_0 on 2022-12-20)
  
  I need to implement a rule and have defined a lower triangular boolean mask for the weights that I want to keep static for a zero value. In which condition triangular weight matrix will be used?
  
  **Tags:** neural-networks, deep-learning, deep-neural-networks
  > **Comments:**
  > **username_1**: Triangular weight matrix could be much weaker to approximate complicated functions, unless your computing resources are limited you may try it.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2023-07-04. Score: 1
  
  I did some testing on nanoGPT (<https://github.com/karpathy/nanoGPT>) using triangular weights matrices instead of full one and the result were interesting : using triangular matrices seems to have a very limited impact on the learning (for gpt2).
  [![enter image description here](https://i.sstatic.net/x9oHv.png)](https://i.sstatic.net/x9oHv.png)
  
  (black and orange curve are the learning for triangular matrix and the pink one is the full linear one)
  
  > **Comments:**
  > **username_3**: I would bet It perform the same as dropout 0.5  
  > **username_2**: My bet (but i'm not sure) that it will not be the case ... dropout randomly choose some activation to shut down. With triangular matrix you have a deterministic behaviour. Also the advantage of triangular is that you can hope to find better gpu kernel to do your triangular matrix multiplication (but I will not do that, I will not code in cuda :)).  
  
  ---
  
  ### Answer by username_4 on 2023-07-04. Score: 0
  
  Constraining weights in a neural network, either a hard constraint like your idea, or a soft constraint that adds to the cost function, is often a form of regularisation. By limiting the form of acceptable solutions, regularisation makes it harder, often impossible, to exactly match training data - this is a good thing because what you most often want to learn is a *generalisation* of the examples that works well with new data. Forcing a machine learning process into a space where it can approach but never reach a perfect solution can make good generalisation.
  
  That means you might use such an idea to combat overfitting. The utility of the idea will vary depending on the problem. There may be no general use for it, because it's never better than other well established approaches. But there could be some problems it is particularly suitable for. You would have to try it and see.
  
  The main problem I see with the approach in general is that it's all or nothing. You cannot easily have an arbitrary fraction of the diagonal mask.
  
  
  ---
  
  