Body: <p><strong>Main question</strong></p>
  
  <p>Is there some way we can leverage general knowledge of how certain hyperparameters affect performance, to very rapidly get some sort of estimate for how good a given architecture <em>could</em> be?</p>
  
  <p><strong>Elaboration</strong></p>
  
  <p>I'm working on a handwritten character recognition problem using CNNs. I want to try out a few different architectures (mostly at random) to iterate towards something which might work. The problem is that one run takes a really long time.</p>
  
  <p>So what's a way to quickly verify if a given architecture is promising? And let me elaborate on what I've tried:</p>
  
  <ul>
  <li><strong>Just try it once</strong>. Yeah but maybe I chose some bad hyperparameter combination and actually that architecture was going to be the ground breaker.</li>
  <li><strong>Do Bayesian optimisation</strong>. That's still really slow. From examples and trials, I've seen that it takes quite some time for convergence. And besides, I'm not trying to optimise yet, I just want to check if there's any potential.</li>
  </ul>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-15T00:13:10.290Z
Comments.Text: I am afraid that there are (still) no quick ways to evaluate the architecture of a NN. If there were, they probably would already be very famous and you would have found them with a quick search on the web, but I may be unaware of them (and I've not searched for them on the web after this question), even though I am familiar with certain hyper-parameter optimization techniques (like grid search, Bayesian optimization, RL, etc).
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-15T07:47:37.890Z
Comments.Text: @nbro while looking over the Bayesian charts on papers, articles and my own work I notice that something near enough to the best value is often found in the first few iterations.the rest seems like just a bunch of noise while getting marginal improvements on the best trial (actually if you know of a counter example I'd love to see it - I'm very new to the field so I won't be surprised if you have one).
Comments.UserDisplayName: Alexander Soare
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-15T07:53:12.217Z
Comments.Text: The other thing that inspires the thought is that if you randomly sample from a continuous function, if it's not a crazily dynamic one, you're likely to get an idea of it's full range after just a few samples. So let's say I have a one parameter function and I know that within my domain of interest it only has n significant turning points, then if I'm lucky I might get an idea of what it could look like in just n samples. And with some strategy I might not have to go much further than that even when I'm unlucky.
Comments.UserDisplayName: Alexander Soare
Tags: <neural-networks><convolutional-neural-networks><hyperparameter-optimization>
Title: What are some ways to quickly evaluate the potential of a given NN architecture?
UserDisplayName: Alexander Soare
ThreadText: # What are some ways to quickly evaluate the potential of a given NN architecture?
  
  (Asked by: username_0 on 2020-01-14)
  
  **Main question**
  
  Is there some way we can leverage general knowledge of how certain hyperparameters affect performance, to very rapidly get some sort of estimate for how good a given architecture *could* be?
  
  **Elaboration**
  
  I'm working on a handwritten character recognition problem using CNNs. I want to try out a few different architectures (mostly at random) to iterate towards something which might work. The problem is that one run takes a really long time.
  
  So what's a way to quickly verify if a given architecture is promising? And let me elaborate on what I've tried:
  
  * **Just try it once**. Yeah but maybe I chose some bad hyperparameter combination and actually that architecture was going to be the ground breaker.
  * **Do Bayesian optimisation**. That's still really slow. From examples and trials, I've seen that it takes quite some time for convergence. And besides, I'm not trying to optimise yet, I just want to check if there's any potential.
  
  **Tags:** neural-networks, convolutional-neural-networks, hyperparameter-optimization
  
  > **Comments:**
  > **username_1**: I am afraid that there are (still) no quick ways to evaluate the architecture of a NN. If there were, they probably would already be very famous and you would have found them with a quick search on the web, but I may be unaware of them (and I've not searched for them on the web after this question), even though I am familiar with certain hyper-parameter optimization techniques (like grid search, Bayesian optimization, RL, etc).  
  > **username_0**: @username_1 while looking over the Bayesian charts on papers, articles and my own work I notice that something near enough to the best value is often found in the first few iterations.the rest seems like just a bunch of noise while getting marginal improvements on the best trial (actually if you know of a counter example I'd love to see it - I'm very new to the field so I won't be surprised if you have one).  
  > **username_0**: The other thing that inspires the thought is that if you randomly sample from a continuous function, if it's not a crazily dynamic one, you're likely to get an idea of it's full range after just a few samples. So let's say I have a one parameter function and I know that within my domain of interest it only has n significant turning points, then if I'm lucky I might get an idea of what it could look like in just n samples. And with some strategy I might not have to go much further than that even when I'm unlucky.  
  
  ---
  
  ## 0 answers
  
  