Answers.Body: <p>In general, many of the parameters you mentioned are called hyperparameters. All hyperparameters are user-adjusted (or user-programmed) in training phase. Some hyperparameters are:</p>
  <ul>
  <li>learning rate,</li>
  <li>batch size,</li>
  <li>epochs,</li>
  <li>optimizer,</li>
  <li>layers,</li>
  <li>activation functions etc.</li>
  </ul>
  <p><strong>To answer your (a) part of your question</strong>, there are obsiously many frameworks and libraries, for example in python; TensorFlow, pytorch and so on. You might never create a net from the very beginning; maybe only in order to understand the forward and backpropagation algorithms. When we call from scatch networks, we mean that these networks are trained from scratch, with learnable weights and chosen hyperparameters; with no transfer learning.</p>
  <p><strong>To answer your (b) part of your question</strong>, I can understand from it that you mean when a net is good enough. Dependently of your data, of course, a neural network is good enough, when it is trained adequately on them. That is, you should be aware of <a href="https://ai.stackexchange.com/questions/17287/is-running-more-epochs-really-a-direct-cause-of-overfitting/21091#21091">overfitting</a>, <a href="https://medium.com/greyatom/what-is-underfitting-and-overfitting-in-machine-learning-and-how-to-deal-with-it-6803a989c76" rel="nofollow noreferrer">underfitting</a>, and in general of the model you are trying to train with all its parameters and hyperparameters.</p>
  <p>Since, you are at the very beginning with Machine Learning, I propose you read some books, in order to get everything needed, in terms of Mathematical and Computer Science aspects.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-11T07:49:02.943Z
Answers.Text: Hi thanks for answer. 
  About part a:
  Do I get right that "FRAMEWORKS" like those you had mentioned (tensor-flow and pytorch) are NOT ready architectures but kind of a sandbox which allows me to build my own? including the adjusting all hyperparameters like those I've mentioned above? (weights range, weights distribution, option to decide in  layers to use or not use activation functions, and if use which kiind of them)... Do I get right?
  
  About part b: is not what  mean. I'll edit this part of a question.
Answers.UserDisplayName: Igor
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-11T08:49:22.320Z
Answers.Text: @Igor as ddaedalus mentioned above, "You might never create a net from the very beginning; maybe only in order to understand the forward and backpropagation algorithms". I am sure with you with the involution of frameworks in this machine learning era, nothing you can't do with them. You can create a new model, tune all parameters, all weights range, change other SOTA model, and they also open the source code of those frameworks, so if you have any problems, feel free to change it, or contribute it to the community.
Answers.UserDisplayName: CuCaRot
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-11T08:54:32.980Z
Answers.Text: There are some differences between those frameworks but it doesn't affect you right now, just choose which has syntax you feel comfortable with and try to read and understand as much as possible. Welcome to this world!
Answers.UserDisplayName: CuCaRot
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-10T17:07:27.553Z
Answers.LastActivityDate: 2020-10-10T17:07:27.553Z
Answers.UserDisplayName: ddaedalus
Body: <p>I am new to Neural Networks and my questions are still very basic.
  I know that most of neural networks allow and even ask user to chose hyper-parameters like:</p>
  <ul>
  <li>amount of hidden layers</li>
  <li>amount of neurons in each layer</li>
  <li>amount of inputs and outputs</li>
  <li>batches and epochs steps and some stuff related to back-propagation and gradient descent</li>
  </ul>
  <p>But as I keep reading and youtubing, I understand that there are another important &quot;mini-parameters&quot; such as:</p>
  <ul>
  <li><p>activation functions type</p>
  </li>
  <li><p>activation functions fine-tuning (for example shift and slope of sigmoid)
  <a href="https://i.sstatic.net/tMAQw.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/tMAQw.png" alt="types of sigmoid finetuning" /></a></p>
  </li>
  <li><p>whether there is an activation funciton in the output</p>
  </li>
  <li><p>range of weights (are they from zero to one or from -1 to 1  or -100 to +100 or any other range)</p>
  </li>
  <li><p>are the weights normally distributed or they just random</p>
  </li>
  </ul>
  <p>etc...</p>
  <p><strong>Actually the question is:</strong></p>
  <p>Part a:</p>
  <p>Do I understand right that most of  neural networks do not allow to change those &quot;mini-parameters&quot;, as long as you are using &quot;readymade&quot; solutions?
  In other words if I want to have an access to those &quot;mini-parameters&quot; I need to program the whole neural network by myself or there are  &quot;semi-finished products&quot;</p>
  <p>Part b:<strong>(edited)</strong>
  For someone who uses neural network as an everyday routine tool to solve problems(Like data scientist), How common and how often do those people deal with fine tuning things which I refer to as &quot;mini-parameters&quot;? Or those parameters are usually adjusted by a neural network developers who create the frameworks like pytorch, tensorflow etc?</p>
  <p>Thank you very much</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-11T01:49:26.067Z
Comments.Text: You can look at the API of whichever library you're using to see which hyperparameters you can set yourself, e.g. [scikit-learn lists a bunch](https://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.html). The default values are decent though.
Comments.UserDisplayName: NotThatGuy
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-11T01:54:18.480Z
Comments.Text: Often you'd need to adjust the weights yourself so it's in the expected range (which is called "normalisation"). The distribution of values may affect the result and there may hyperparameters to indirectly deal with an improper distribution (like providing weights for different classes), but I wouldn't expect there to be an parameter for specifying the distribution directly. I don't think it would be able to do too much with that information.
Comments.UserDisplayName: NotThatGuy
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-11T08:11:46.300Z
Comments.Text: NotThatGuy , thanks.
  Can you please make me some order in understanding. 
  Is it right to say: that things like pytorch, tensorflow, keras, scikit learn are alternatives one to another? Thay all allow to build me a desired network with 
  all hyperparameters ibe mentioned? 
   Are they all called "frameworks"?
Comments.UserDisplayName: Igor
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-12T15:48:45.290Z
Comments.Text: @Igor I'm unfortunately not familiar enough with the others to be able to answer that, but you can certainly create neural networks using scikit-learn (with whichever hyperparameters it supports). [Wikipedia](https://en.wikipedia.org/wiki/Scikit-learn) calls that a "library". I'll leave it to other people to [explain the difference between a framework and a library](https://stackoverflow.com/questions/148747/what-is-the-difference-between-a-framework-and-a-library) and judge what is a library and what is a framework.
Comments.UserDisplayName: NotThatGuy
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-13T07:52:26.597Z
Comments.Text: NotThatGuy , thanks again. You got my question right. BTW It is not that I am trying to learn all frameworks at once, I just understand things better when I have some kind of taxonomy (tree like structure)of relevant terms inside my head. Like which term is a private case of which. Which term is a generalization of some other terms, and which terms are equal in hierarchy (they may be not totally equal but more alternatives)
Comments.UserDisplayName: Igor
Tags: <activation-functions><hyper-parameters><weights><pretrained-models>
Title: Which hyperparameters in neural network are accesible to users adjustment
UserDisplayName: Igor
ThreadText: # Which hyperparameters in neural network are accesible to users adjustment
  
  (Asked by: username_0 on 2020-10-10)
  
  I am new to Neural Networks and my questions are still very basic.
  I know that most of neural networks allow and even ask user to chose hyper-parameters like:
  
  * amount of hidden layers
  * amount of neurons in each layer
  * amount of inputs and outputs
  * batches and epochs steps and some stuff related to back-propagation and gradient descent
  
  But as I keep reading and youtubing, I understand that there are another important "mini-parameters" such as:
  
  * activation functions type
  * activation functions fine-tuning (for example shift and slope of sigmoid)
    [![types of sigmoid finetuning](https://i.sstatic.net/tMAQw.png)](https://i.sstatic.net/tMAQw.png)
  * whether there is an activation funciton in the output
  * range of weights (are they from zero to one or from -1 to 1 or -100 to +100 or any other range)
  * are the weights normally distributed or they just random
  
  etc...
  
  **Actually the question is:**
  
  Part a:
  
  Do I understand right that most of neural networks do not allow to change those "mini-parameters", as long as you are using "readymade" solutions?
  In other words if I want to have an access to those "mini-parameters" I need to program the whole neural network by myself or there are "semi-finished products"
  
  Part b:**(edited)**
  For someone who uses neural network as an everyday routine tool to solve problems(Like data scientist), How common and how often do those people deal with fine tuning things which I refer to as "mini-parameters"? Or those parameters are usually adjusted by a neural network developers who create the frameworks like pytorch, tensorflow etc?
  
  Thank you very much
  
  **Tags:** activation-functions, hyper-parameters, weights, pretrained-models
  > **Comments:**
  > **username_1**: You can look at the API of whichever library you're using to see which hyperparameters you can set yourself, e.g. [scikit-learn lists a bunch](https://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.html). The default values are decent though.  
  > **username_1**: Often you'd need to adjust the weights yourself so it's in the expected range (which is called "normalisation"). The distribution of values may affect the result and there may hyperparameters to indirectly deal with an improper distribution (like providing weights for different classes), but I wouldn't expect there to be an parameter for specifying the distribution directly. I don't think it would be able to do too much with that information.  
  > **username_0**: NotThatGuy , thanks. Can you please make me some order in understanding.  Is it right to say: that things like pytorch, tensorflow, keras, scikit learn are alternatives one to another? Thay all allow to build me a desired network with  all hyperparameters ibe mentioned?   Are they all called "frameworks"?  
  > **username_1**: @Igor I'm unfortunately not familiar enough with the others to be able to answer that, but you can certainly create neural networks using scikit-learn (with whichever hyperparameters it supports). [Wikipedia](https://en.wikipedia.org/wiki/Scikit-learn) calls that a "library". I'll leave it to other people to [explain the difference between a framework and a library](https://stackoverflow.com/questions/148747/what-is-the-difference-between-a-framework-and-a-library) and judge what is a library and what is a framework.  
  > **username_0**: NotThatGuy , thanks again. You got my question right. BTW It is not that I am trying to learn all frameworks at once, I just understand things better when I have some kind of taxonomy (tree like structure)of relevant terms inside my head. Like which term is a private case of which. Which term is a generalization of some other terms, and which terms are equal in hierarchy (they may be not totally equal but more alternatives)  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-10-10. Score: 3 | Accepted answer
  
  In general, many of the parameters you mentioned are called hyperparameters. All hyperparameters are user-adjusted (or user-programmed) in training phase. Some hyperparameters are:
  
  * learning rate,
  * batch size,
  * epochs,
  * optimizer,
  * layers,
  * activation functions etc.
  
  **To answer your (a) part of your question**, there are obsiously many frameworks and libraries, for example in python; TensorFlow, pytorch and so on. You might never create a net from the very beginning; maybe only in order to understand the forward and backpropagation algorithms. When we call from scatch networks, we mean that these networks are trained from scratch, with learnable weights and chosen hyperparameters; with no transfer learning.
  
  **To answer your (b) part of your question**, I can understand from it that you mean when a net is good enough. Dependently of your data, of course, a neural network is good enough, when it is trained adequately on them. That is, you should be aware of [overfitting](https://ai.stackexchange.com/questions/17287/is-running-more-epochs-really-a-direct-cause-of-overfitting/21091#21091), [underfitting](https://medium.com/greyatom/what-is-underfitting-and-overfitting-in-machine-learning-and-how-to-deal-with-it-6803a989c76), and in general of the model you are trying to train with all its parameters and hyperparameters.
  
  Since, you are at the very beginning with Machine Learning, I propose you read some books, in order to get everything needed, in terms of Mathematical and Computer Science aspects.
  
  > **Comments:**
  > **username_0**: Hi thanks for answer.  About part a: Do I get right that "FRAMEWORKS" like those you had mentioned (tensor-flow and pytorch) are NOT ready architectures but kind of a sandbox which allows me to build my own? including the adjusting all hyperparameters like those I've mentioned above? (weights range, weights distribution, option to decide in  layers to use or not use activation functions, and if use which kiind of them)... Do I get right?  About part b: is not what  mean. I'll edit this part of a question.  
  > **username_3**: @Igor as ddaedalus mentioned above, "You might never create a net from the very beginning; maybe only in order to understand the forward and backpropagation algorithms". I am sure with you with the involution of frameworks in this machine learning era, nothing you can't do with them. You can create a new model, tune all parameters, all weights range, change other SOTA model, and they also open the source code of those frameworks, so if you have any problems, feel free to change it, or contribute it to the community.  
  > **username_3**: There are some differences between those frameworks but it doesn't affect you right now, just choose which has syntax you feel comfortable with and try to read and understand as much as possible. Welcome to this world!  
  
  ---
  
  