<p>In general a <em>ML</em> model <code>g</code> as you specified needs training dataset and hyperparameters for itself, therefore you're still facing the same type of problems in the meta-level and it's computational expensive or infeasible to prepare a representative training and test datasets for <code>g</code> with satisfactory generalization error.</p>
<p>Having said that, there's some related research such as <a href="https://en.wikipedia.org/wiki/Neural_architecture_search" rel="nofollow noreferrer">neural architecture search (NAS)</a> to achieve similar based on differentiable objectives to gradually optimize architectures and hyperparameters.</p>
<blockquote>
<p>Neural architecture search (NAS) is a technique for automating the design of artificial neural networks (ANN), a widely used model in the field of machine learning... NAS is closely related to hyperparameter optimization and meta-learning and is a subfield of automated machine learning (AutoML)</p>
</blockquote>

CC BY-SA 4.0
2024-11-29T19:52:21.700Z
2024-11-29T19:52:21.700Z
cinch
<p>(I assume that this is not possible because I've never seen anyone talk about this.)</p>
<p>Let's take a classic MLP (named <code>f</code>) that, for example classify some images (from a dataset <code>D</code>). This MLP have some hyperparameters <code>H</code> like the learning rate.</p>
<p>Is it possible to have another ML model <code>g</code> that, based on the dataset <code>D</code>and the <code>f</code> model architecture, could tell the optimal hyperparameters <code>H*</code> (for example to use the optimal learning rate without any gridsearch) ?</p>
<p>Summarized as :  <code>g(D,f) = H*</code></p>
<p>Of course this question can be extended as : why don't we make a model that can learn to predict the best weight of the model directly with as input the dataset <code>D</code> and the model architecture <code>f</code> ?</p>

CC BY-SA 4.0
2024-12-02T03:17:19.770Z
this is related to meta-learning. you might find this paper interesting: https://arxiv.org/abs/1606.04474
Aditya Tomar
<neural-networks><machine-learning><hyperparameter-optimization><hyper-parameters>
Can a ML model learn the hyperparameters landscape?
Arthur Delannoy
# Can a ML model learn the hyperparameters landscape?

(Asked by: username_0 on 2024-11-29)

(I assume that this is not possible because I've never seen anyone talk about this.)

Let's take a classic MLP (named `f`) that, for example classify some images (from a dataset `D`). This MLP have some hyperparameters `H` like the learning rate.

Is it possible to have another ML model `g` that, based on the dataset `D`and the `f` model architecture, could tell the optimal hyperparameters `H*` (for example to use the optimal learning rate without any gridsearch) ?

Summarized as : `g(D,f) = H*`

Of course this question can be extended as : why don't we make a model that can learn to predict the best weight of the model directly with as input the dataset `D` and the model architecture `f` ?

**Tags:** neural-networks, machine-learning, hyperparameter-optimization, hyper-parameters

> **Comments:**
> **username_1**: this is related to meta-learning. you might find this paper interesting: https://arxiv.org/abs/1606.04474  

---

## 1 answers

### Answer by username_2 on 2024-11-29. Score: 3

In general a *ML* model `g` as you specified needs training dataset and hyperparameters for itself, therefore you're still facing the same type of problems in the meta-level and it's computational expensive or infeasible to prepare a representative training and test datasets for `g` with satisfactory generalization error.

Having said that, there's some related research such as [neural architecture search (NAS)](https://en.wikipedia.org/wiki/Neural_architecture_search) to achieve similar based on differentiable objectives to gradually optimize architectures and hyperparameters.

> Neural architecture search (NAS) is a technique for automating the design of artificial neural networks (ANN), a widely used model in the field of machine learning... NAS is closely related to hyperparameter optimization and meta-learning and is a subfield of automated machine learning (AutoML)


---

