<p><strong>Main question</strong></p>

<p>Is there some way we can leverage general knowledge of how certain hyperparameters affect performance, to very rapidly get some sort of estimate for how good a given architecture <em>could</em> be?</p>

<p><strong>Elaboration</strong></p>

<p>I'm working on a handwritten character recognition problem using CNNs. I want to try out a few different architectures (mostly at random) to iterate towards something which might work. The problem is that one run takes a really long time.</p>

<p>So what's a way to quickly verify if a given architecture is promising? And let me elaborate on what I've tried:</p>

<ul>
<li><strong>Just try it once</strong>. Yeah but maybe I chose some bad hyperparameter combination and actually that architecture was going to be the ground breaker.</li>
<li><strong>Do Bayesian optimisation</strong>. That's still really slow. From examples and trials, I've seen that it takes quite some time for convergence. And besides, I'm not trying to optimise yet, I just want to check if there's any potential.</li>
</ul>

CC BY-SA 4.0
2020-01-15T00:13:10.290Z
I am afraid that there are (still) no quick ways to evaluate the architecture of a NN. If there were, they probably would already be very famous and you would have found them with a quick search on the web, but I may be unaware of them (and I've not searched for them on the web after this question), even though I am familiar with certain hyper-parameter optimization techniques (like grid search, Bayesian optimization, RL, etc).
nbro
CC BY-SA 4.0
2020-01-15T07:47:37.890Z
@nbro while looking over the Bayesian charts on papers, articles and my own work I notice that something near enough to the best value is often found in the first few iterations.the rest seems like just a bunch of noise while getting marginal improvements on the best trial (actually if you know of a counter example I'd love to see it - I'm very new to the field so I won't be surprised if you have one).
Alexander Soare
CC BY-SA 4.0
2020-01-15T07:53:12.217Z
The other thing that inspires the thought is that if you randomly sample from a continuous function, if it's not a crazily dynamic one, you're likely to get an idea of it's full range after just a few samples. So let's say I have a one parameter function and I know that within my domain of interest it only has n significant turning points, then if I'm lucky I might get an idea of what it could look like in just n samples. And with some strategy I might not have to go much further than that even when I'm unlucky.
Alexander Soare
<neural-networks><convolutional-neural-networks><hyperparameter-optimization>
What are some ways to quickly evaluate the potential of a given NN architecture?
Alexander Soare
# What are some ways to quickly evaluate the potential of a given NN architecture?

(Asked by: username_0 on 2020-01-14)

**Main question**

Is there some way we can leverage general knowledge of how certain hyperparameters affect performance, to very rapidly get some sort of estimate for how good a given architecture *could* be?

**Elaboration**

I'm working on a handwritten character recognition problem using CNNs. I want to try out a few different architectures (mostly at random) to iterate towards something which might work. The problem is that one run takes a really long time.

So what's a way to quickly verify if a given architecture is promising? And let me elaborate on what I've tried:

* **Just try it once**. Yeah but maybe I chose some bad hyperparameter combination and actually that architecture was going to be the ground breaker.
* **Do Bayesian optimisation**. That's still really slow. From examples and trials, I've seen that it takes quite some time for convergence. And besides, I'm not trying to optimise yet, I just want to check if there's any potential.

**Tags:** neural-networks, convolutional-neural-networks, hyperparameter-optimization

> **Comments:**
> **username_1**: I am afraid that there are (still) no quick ways to evaluate the architecture of a NN. If there were, they probably would already be very famous and you would have found them with a quick search on the web, but I may be unaware of them (and I've not searched for them on the web after this question), even though I am familiar with certain hyper-parameter optimization techniques (like grid search, Bayesian optimization, RL, etc).  
> **username_0**: @username_1 while looking over the Bayesian charts on papers, articles and my own work I notice that something near enough to the best value is often found in the first few iterations.the rest seems like just a bunch of noise while getting marginal improvements on the best trial (actually if you know of a counter example I'd love to see it - I'm very new to the field so I won't be surprised if you have one).  
> **username_0**: The other thing that inspires the thought is that if you randomly sample from a continuous function, if it's not a crazily dynamic one, you're likely to get an idea of it's full range after just a few samples. So let's say I have a one parameter function and I know that within my domain of interest it only has n significant turning points, then if I'm lucky I might get an idea of what it could look like in just n samples. And with some strategy I might not have to go much further than that even when I'm unlucky.  

---

## 0 answers

