Answers.Body: <p>I'm not sure it's possible. Untrained CNN means it has random kernel values. Let's say you have a kernel with size 3x3 like below:</p>
  
  <pre><code>0 0 0
  0 0 0
  0 0 1
  </code></pre>
  
  <p>I don't think it is possible for that kernel to provide good information about the image. on the contrary, the kernel eliminates a lot of information. We cannot rely on random values for feature extraction.</p>
  
  <p>But, if you use CNN with "assigned" kernel, then you don't need to train the convolutional layer. For example, you can start a CNN with a kernel that designed to extract vertical line:</p>
  
  <pre><code>-1 2 -1
  -1 2 -1
  -1 2 -1
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-06T13:09:02.393Z
Answers.Text: there exists a decent amount of evidence showing you can achieve amazing feature representations using a randomly intiialized CNN as a feature extractor. Think of dart throwing, youll probably get alot of useless ones, but some really good ones will be there.
Answers.UserDisplayName: mshlis
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-06T13:49:48.097Z
Answers.Text: @mshlis I'm sorry, but what do you mean with "randomly initialized", is it "trained" or "untrained" after that?.
Answers.UserDisplayName: malioboro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-06T13:53:11.473Z
Answers.Text: Actually I heard about this random and untrained layer since [Yann Lecun's talk about ELM](https://www.reddit.com/r/MachineLearning/comments/34u0go/yann_lecun_whats_so_great_about_extreme_learning/). I just read papers that are referenced by @David, the first paper is specific in the image restoration cases, now I'm trying to understand the second paper, I hope I can find a new knowledge about this
Answers.UserDisplayName: malioboro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-06T14:00:09.740Z
Answers.Text: checkout ubers supermask paper (just finding a good mask for a random initialization can achieve >80% accuracy on many tasks)
Answers.UserDisplayName: mshlis
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-06T14:07:18.533Z
Answers.Text: @mshlis wow, thank you for the reference, I'll check it soon
Answers.UserDisplayName: malioboro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-06T05:41:21.837Z
Answers.LastActivityDate: 2019-08-06T05:41:21.837Z
Answers.UserDisplayName: malioboro
Answers.Body: <p>Yes, it has been demonstrated that the main factor for CNNs to work is its architecture, which exploits locality during the feature extraction. A CNN with random weights will do a random partition of the feature space, but still with that spatial prior that works so well, so those random features are OK for classification (and sometimes even better than trained ones, as they don't introduce additional bias).</p>
  
  <p>You can read more in these papers:</p>
  
  <ul>
  <li><p><a href="https://arxiv.org/abs/1711.10925" rel="noreferrer">D. Ulyanov et al. Deep Image Prior</a></p></li>
  <li><p><a href="https://arxiv.org/abs/1802.00844" rel="noreferrer">A. Rosenfeld and J. K. Tsotsos. Intriguing Properties of Randomly Weighted Networks: Generalizing While Learning Next to Nothing</a>.</p></li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-08T01:54:24.170Z
Answers.Text: Yeah this is exactly what I was looking for. Thanks!
Answers.UserDisplayName: Alex
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-06T10:57:34.543Z
Answers.LastActivityDate: 2019-08-06T11:14:10.187Z
Answers.LastEditDate: 2019-08-06T11:14:10.187Z
Answers.UserDisplayName: David
Body: <p>I've heard somewhere that due to their nature of capturing spatial relations, even untrained CNNs can be used as feature extractors? Is this true? Does anyone have any sources regarding this I can look at?</p>
  
Tags: <deep-learning><convolutional-neural-networks><computer-vision><feature-extraction>
Title: Is it true that untrained CNNs can be used as feature extractors?
UserDisplayName: Alex
ThreadText: # Is it true that untrained CNNs can be used as feature extractors?
  
  (Asked by: username_0 on 2019-08-06)
  
  I've heard somewhere that due to their nature of capturing spatial relations, even untrained CNNs can be used as feature extractors? Is this true? Does anyone have any sources regarding this I can look at?
  
  **Tags:** deep-learning, convolutional-neural-networks, computer-vision, feature-extraction
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2019-08-06. Score: 6 | Accepted answer
  
  Yes, it has been demonstrated that the main factor for CNNs to work is its architecture, which exploits locality during the feature extraction. A CNN with random weights will do a random partition of the feature space, but still with that spatial prior that works so well, so those random features are OK for classification (and sometimes even better than trained ones, as they don't introduce additional bias).
  
  You can read more in these papers:
  
  * [D. Ulyanov et al. Deep Image Prior](https://arxiv.org/abs/1711.10925)
  * [A. Rosenfeld and J. K. Tsotsos. Intriguing Properties of Randomly Weighted Networks: Generalizing While Learning Next to Nothing](https://arxiv.org/abs/1802.00844).
  
  > **Comments:**
  > **username_0**: Yeah this is exactly what I was looking for. Thanks!  
  
  ---
  
  ### Answer by username_1 on 2019-08-06. Score: -1
  
  I'm not sure it's possible. Untrained CNN means it has random kernel values. Let's say you have a kernel with size 3x3 like below:
  
  ```
  0 0 0
  0 0 0
  0 0 1
  
  ```
  
  I don't think it is possible for that kernel to provide good information about the image. on the contrary, the kernel eliminates a lot of information. We cannot rely on random values for feature extraction.
  
  But, if you use CNN with "assigned" kernel, then you don't need to train the convolutional layer. For example, you can start a CNN with a kernel that designed to extract vertical line:
  
  ```
  -1 2 -1
  -1 2 -1
  -1 2 -1
  
  ```
  
  > **Comments:**
  > **username_2**: there exists a decent amount of evidence showing you can achieve amazing feature representations using a randomly intiialized CNN as a feature extractor. Think of dart throwing, youll probably get alot of useless ones, but some really good ones will be there.  
  > **username_1**: @username_2 I'm sorry, but what do you mean with "randomly initialized", is it "trained" or "untrained" after that?.  
  > **username_1**: Actually I heard about this random and untrained layer since [Yann Lecun's talk about ELM](https://www.reddit.com/r/MachineLearning/comments/34u0go/yann_lecun_whats_so_great_about_extreme_learning/). I just read papers that are referenced by @username_3, the first paper is specific in the image restoration cases, now I'm trying to understand the second paper, I hope I can find a new knowledge about this  
  > **username_2**: checkout ubers supermask paper (just finding a good mask for a random initialization can achieve >80% accuracy on many tasks)  
  > **username_1**: @username_2 wow, thank you for the reference, I'll check it soon  
  
  ---
  
  