Answers.Body: <p>In some sense yes, in some sense no. The "error" word you are using is not defined in the question. The network can have two type of errors, quite different - <a href="https://en.wikipedia.org/wiki/Generalization_error" rel="nofollow noreferrer">generalization error</a> and loss (training error). The lower bound of loss on the dataset obviously depend on the network architecture, because loss is value produced by network as a whole, and some network will be able to produce exactly zero loss (which is likely be overfitting, wich is bad).</p>
  
  <p>The generalization error is how well network generalize form training dataset to potentially infinite (or very big) amount of data samples "in the wild", so generalization error can not be calculated precisely, only estimated. The lower bound on generalization error is studied in Statistical Learning. It always depend on the complexity of the classifier (in our case architecture of network). Statistical Learning theory produce estimations like <a href="https://en.wikipedia.org/wiki/Vapnik%E2%80%93Chervonenkis_dimension" rel="nofollow noreferrer">VC generalization bound</a> which connect generalization error with size of the dataset and complexity of the classifer (netwroks in our case) space. <em>However for specific dataset it probably don't make sense to talk about generalization bound outside of context of complexity of classifier space</em>, because if there is no restriction we can always construct bigger network which would have lesser generalization error (for example by pretraning bigger network on bigger dataset)
  To put it short you have to put restriction on complexity of network space to be able estimate minimum generalization error (and that probably wouldn't be useful anyway, because it wouldn't say anything about if gradient descent would reach that error)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-24T10:08:21.777Z
Answers.LastActivityDate: 2019-06-24T12:17:14.690Z
Answers.LastEditDate: 2019-06-24T12:17:14.690Z
Answers.UserDisplayName: mirror2image
Body: <p>In error-based learning using gradient descent, if I give you a training dataset, then can you find the minimum error after training? And the minimum error should be true for all architectures of a neural network? Consider you will use MSE for calculating the error. You can choose anything you want other than my specified condition. It's like no matter how you change your network you can never cross the limit.</p>
  
Tags: <neural-networks><math><proofs>
Title: Is there a limit of minimum error for a particular training dataset in artificial Neural Network?
UserDisplayName: bipul kalita
ThreadText: # Is there a limit of minimum error for a particular training dataset in artificial Neural Network?
  
  (Asked by: username_0 on 2018-10-28)
  
  In error-based learning using gradient descent, if I give you a training dataset, then can you find the minimum error after training? And the minimum error should be true for all architectures of a neural network? Consider you will use MSE for calculating the error. You can choose anything you want other than my specified condition. It's like no matter how you change your network you can never cross the limit.
  
  **Tags:** neural-networks, math, proofs
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-06-24. Score: 1
  
  In some sense yes, in some sense no. The "error" word you are using is not defined in the question. The network can have two type of errors, quite different - [generalization error](https://en.wikipedia.org/wiki/Generalization_error) and loss (training error). The lower bound of loss on the dataset obviously depend on the network architecture, because loss is value produced by network as a whole, and some network will be able to produce exactly zero loss (which is likely be overfitting, wich is bad).
  
  The generalization error is how well network generalize form training dataset to potentially infinite (or very big) amount of data samples "in the wild", so generalization error can not be calculated precisely, only estimated. The lower bound on generalization error is studied in Statistical Learning. It always depend on the complexity of the classifier (in our case architecture of network). Statistical Learning theory produce estimations like [VC generalization bound](https://en.wikipedia.org/wiki/Vapnik%E2%80%93Chervonenkis_dimension) which connect generalization error with size of the dataset and complexity of the classifer (netwroks in our case) space. *However for specific dataset it probably don't make sense to talk about generalization bound outside of context of complexity of classifier space*, because if there is no restriction we can always construct bigger network which would have lesser generalization error (for example by pretraning bigger network on bigger dataset)
  To put it short you have to put restriction on complexity of network space to be able estimate minimum generalization error (and that probably wouldn't be useful anyway, because it wouldn't say anything about if gradient descent would reach that error)
  
  
  ---
  
  