Body: <p>Which specific performance evaluation metrics are used in training, validation, and testing, and <strong>why</strong>? I am thinking error metrics (RMSE, MAE, MSE) are used in validation, and testing should use a wide variety of metrics? I don't think performance is evaluated in training, but not 100% sure.</p>
  <p>Specifically, I am actually after deciding when to use (i.e. in training, validation, or testing) correlation coefficient, RMSE, MAE and others for numeric data (e.g. Willmott's Index of Agreement, Nash-Sutcliffe coefficient, etc.)</p>
  <p>Sorry about this being broad - I have actually been asked to define it generally (i.e. not for a specific dataset). But datasets I have been using have all numeric continuous values with supervised learning situations.</p>
  <p>Generally, I am using performance evaluation for environmental data where I am using ANNs. I have continuous features and am predicting a continuous variable.</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-04-14T14:04:47.123Z
Comments.Text: Thanks for that. Sorry about that - I will try to explain it more. I am actually after deciding when to use specifically correlation coefficient, RMSE, MAE and others for numeric data (e.g. Willmott's Index of Agreement, Nash-Sutcliffe coefficient, etc.)
Comments.UserDisplayName: user9645302
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-04-14T14:11:38.543Z
Comments.Text: Since I actually need to explore all ways performance evaluation is used in training, validation and testing, I would be happy to know how it used to train the network. Thank you
Comments.UserDisplayName: user9645302
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-04-15T11:56:42.330Z
Comments.Text: @DuttaA Thank you for your reply and sorry that this is not clear. Actually in my course I am doing it is all about using ANN to model data.  The question I have been asked to answer is about discussing which stages out of training, validation and testing should the performance metrics correlation coefficient, RMSE, MAE, Willmott's Index, Nash-Sutcliffe coefficient, Legate and McCabe's Index and Percentage Peak Deviation be used in? We haven't been given any more information or data. I think what I would like to know is probably when do we use error metrics and when to use non-error metrics?
Comments.UserDisplayName: user9645302
Tags: <machine-learning><training><metric><testing><validation>
Title: Which evaluation metrics should be used in training, validation and testing of a model?
UserDisplayName: user9645302
ThreadText: # Which evaluation metrics should be used in training, validation and testing of a model?
  
  (Asked by: username_0 on 2018-04-14)
  
  Which specific performance evaluation metrics are used in training, validation, and testing, and **why**? I am thinking error metrics (RMSE, MAE, MSE) are used in validation, and testing should use a wide variety of metrics? I don't think performance is evaluated in training, but not 100% sure.
  
  Specifically, I am actually after deciding when to use (i.e. in training, validation, or testing) correlation coefficient, RMSE, MAE and others for numeric data (e.g. Willmott's Index of Agreement, Nash-Sutcliffe coefficient, etc.)
  
  Sorry about this being broad - I have actually been asked to define it generally (i.e. not for a specific dataset). But datasets I have been using have all numeric continuous values with supervised learning situations.
  
  Generally, I am using performance evaluation for environmental data where I am using ANNs. I have continuous features and am predicting a continuous variable.
  
  **Tags:** machine-learning, training, metric, testing, validation
  
  > **Comments:**
  > **username_0**: Thanks for that. Sorry about that - I will try to explain it more. I am actually after deciding when to use specifically correlation coefficient, RMSE, MAE and others for numeric data (e.g. Willmott's Index of Agreement, Nash-Sutcliffe coefficient, etc.)  
  > **username_0**: Since I actually need to explore all ways performance evaluation is used in training, validation and testing, I would be happy to know how it used to train the network. Thank you  
  > **username_0**: @DuttaA Thank you for your reply and sorry that this is not clear. Actually in my course I am doing it is all about using ANN to model data.  The question I have been asked to answer is about discussing which stages out of training, validation and testing should the performance metrics correlation coefficient, RMSE, MAE, Willmott's Index, Nash-Sutcliffe coefficient, Legate and McCabe's Index and Percentage Peak Deviation be used in? We haven't been given any more information or data. I think what I would like to know is probably when do we use error metrics and when to use non-error metrics?  
  
  ---
  
  ## 0 answers
  
  