Answers.Body: <p>It is hard to tell what exactly is better because these are hyperparameters. However, the sigmoid activation function is closer to biological neurons.</p>
  <p>In the paper below, Bengio demonstrates why ReLU activation functions are better for hidden layers. In summary, they increase the sparsity of calculations (matrix in each layer shod multiply to its relative weights), and because of that, it made it possible for data to classify faster and easier.</p>
  <p><a href="http://proceedings.mlr.press/v15/glorot11a/glorot11a.pdf" rel="nofollow noreferrer">http://proceedings.mlr.press/v15/glorot11a/glorot11a.pdf</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-01T16:13:02.273Z
Answers.LastActivityDate: 2020-10-01T16:13:02.273Z
Answers.UserDisplayName: kiarash_kiani
Body: <p>I am designing a deep autoencoder for graph embedding (exactly node embedding) following this paper <a href="https://www.kdd.org/kdd2016/papers/files/rfp0191-wangAemb.pdf" rel="nofollow noreferrer">SDNE</a>. In the original paper, they used the <em>sigmoid</em> activation for all hidden layers in the autoencoder model, even for the embedding layer.</p>
  <p>However, I think the embedding layer should use the <em>tanh</em> activation and the reconstruction layer should be used <em>ReLU</em> activation. Because, embedding is in the range <span class="math-container">$[-1, 1]$</span> and reconstruction layer is in the range <span class="math-container">$[0, x]$</span>, which generates better results due to a larger range for representation and directed graph. Instead of in the range <span class="math-container">$[0,1]$</span> from <em>sigmoid</em> will lead to a lack of embedding information.</p>
  <p>So, what is the best activation function for deep autoencoders to capture good information about the structure of graph?</p>
  
Tags: <machine-learning><activation-functions><autoencoders><hyperparameter-optimization><hyper-parameters>
Title: What is the best activation function for the embedding layer in a deep auto-encoder?
UserDisplayName: Truong Hoang
ThreadText: # What is the best activation function for the embedding layer in a deep auto-encoder?
  
  (Asked by: username_0 on 2020-08-20)
  
  I am designing a deep autoencoder for graph embedding (exactly node embedding) following this paper [SDNE](https://www.kdd.org/kdd2016/papers/files/rfp0191-wangAemb.pdf). In the original paper, they used the *sigmoid* activation for all hidden layers in the autoencoder model, even for the embedding layer.
  
  However, I think the embedding layer should use the *tanh* activation and the reconstruction layer should be used *ReLU* activation. Because, embedding is in the range $[-1, 1]$ and reconstruction layer is in the range $[0, x]$, which generates better results due to a larger range for representation and directed graph. Instead of in the range $[0,1]$ from *sigmoid* will lead to a lack of embedding information.
  
  So, what is the best activation function for deep autoencoders to capture good information about the structure of graph?
  
  **Tags:** machine-learning, activation-functions, autoencoders, hyperparameter-optimization, hyper-parameters
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-10-01. Score: 1
  
  It is hard to tell what exactly is better because these are hyperparameters. However, the sigmoid activation function is closer to biological neurons.
  
  In the paper below, Bengio demonstrates why ReLU activation functions are better for hidden layers. In summary, they increase the sparsity of calculations (matrix in each layer shod multiply to its relative weights), and because of that, it made it possible for data to classify faster and easier.
  
  <http://proceedings.mlr.press/v15/glorot11a/glorot11a.pdf>
  
  
  ---
  
  