Answers.Body: <p>In general a <em>ML</em> model <code>g</code> as you specified needs training dataset and hyperparameters for itself, therefore you're still facing the same type of problems in the meta-level and it's computational expensive or infeasible to prepare a representative training and test datasets for <code>g</code> with satisfactory generalization error.</p>
  <p>Having said that, there's some related research such as <a href="https://en.wikipedia.org/wiki/Neural_architecture_search" rel="nofollow noreferrer">neural architecture search (NAS)</a> to achieve similar based on differentiable objectives to gradually optimize architectures and hyperparameters.</p>
  <blockquote>
  <p>Neural architecture search (NAS) is a technique for automating the design of artificial neural networks (ANN), a widely used model in the field of machine learning... NAS is closely related to hyperparameter optimization and meta-learning and is a subfield of automated machine learning (AutoML)</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-29T19:52:21.700Z
Answers.LastActivityDate: 2024-11-29T19:52:21.700Z
Answers.UserDisplayName: cinch
Body: <p>(I assume that this is not possible because I've never seen anyone talk about this.)</p>
  <p>Let's take a classic MLP (named <code>f</code>) that, for example classify some images (from a dataset <code>D</code>). This MLP have some hyperparameters <code>H</code> like the learning rate.</p>
  <p>Is it possible to have another ML model <code>g</code> that, based on the dataset <code>D</code>and the <code>f</code> model architecture, could tell the optimal hyperparameters <code>H*</code> (for example to use the optimal learning rate without any gridsearch) ?</p>
  <p>Summarized as :  <code>g(D,f) = H*</code></p>
  <p>Of course this question can be extended as : why don't we make a model that can learn to predict the best weight of the model directly with as input the dataset <code>D</code> and the model architecture <code>f</code> ?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-12-02T03:17:19.770Z
Comments.Text: this is related to meta-learning. you might find this paper interesting: https://arxiv.org/abs/1606.04474
Comments.UserDisplayName: Aditya Tomar
Tags: <neural-networks><machine-learning><hyperparameter-optimization><hyper-parameters>
Title: Can a ML model learn the hyperparameters landscape?
UserDisplayName: Arthur Delannoy
ThreadText: # Can a ML model learn the hyperparameters landscape?
  
  (Asked by: username_0 on 2024-11-29)
  
  (I assume that this is not possible because I've never seen anyone talk about this.)
  
  Let's take a classic MLP (named `f`) that, for example classify some images (from a dataset `D`). This MLP have some hyperparameters `H` like the learning rate.
  
  Is it possible to have another ML model `g` that, based on the dataset `D`and the `f` model architecture, could tell the optimal hyperparameters `H*` (for example to use the optimal learning rate without any gridsearch) ?
  
  Summarized as : `g(D,f) = H*`
  
  Of course this question can be extended as : why don't we make a model that can learn to predict the best weight of the model directly with as input the dataset `D` and the model architecture `f` ?
  
  **Tags:** neural-networks, machine-learning, hyperparameter-optimization, hyper-parameters
  
  > **Comments:**
  > **username_1**: this is related to meta-learning. you might find this paper interesting: https://arxiv.org/abs/1606.04474  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-11-29. Score: 3
  
  In general a *ML* model `g` as you specified needs training dataset and hyperparameters for itself, therefore you're still facing the same type of problems in the meta-level and it's computational expensive or infeasible to prepare a representative training and test datasets for `g` with satisfactory generalization error.
  
  Having said that, there's some related research such as [neural architecture search (NAS)](https://en.wikipedia.org/wiki/Neural_architecture_search) to achieve similar based on differentiable objectives to gradually optimize architectures and hyperparameters.
  
  > Neural architecture search (NAS) is a technique for automating the design of artificial neural networks (ANN), a widely used model in the field of machine learning... NAS is closely related to hyperparameter optimization and meta-learning and is a subfield of automated machine learning (AutoML)
  
  
  ---
  
  