Answers.Body: <p>Instead of using the <code>Embedding()</code> layer directly, you can create a new <code>bertEmbedding()</code> layer and use it instead.</p>
  
  
  
  <pre><code># Sample code
  # Model architecture
  
  # Custom BERT layer
  bert_output = BertLayer(n_fine_tune_layers=10)(bert_inputs)
  
  # Build the rest of the classifier 
  dense = tf.keras.layers.Dense(256, activation='relu')(bert_output)
  pred = tf.keras.layers.Dense(1, activation='sigmoid')(dense)
  
  model = tf.keras.models.Model(inputs=bert_inputs, outputs=pred)
  model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
  model.fit(...)
  </code></pre>
  
  <p>This <a href="https://towardsdatascience.com/bert-in-keras-with-tensorflow-hub-76bcbc9417b" rel="nofollow noreferrer">article</a> will walk you through the entire process of creating the custom BERT layer along with example code. Give it a read.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-17T15:02:39.240Z
Answers.LastActivityDate: 2019-06-17T15:02:39.240Z
Answers.UserDisplayName: skillsmuggler
Body: <p>I am planning to use BERT embeddings in the LSTM embedding layer instead of the usual Word2vec/Glove Embeddings. What are the possible ways to do that? </p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-06-17T15:27:26.430Z
Comments.Text: do you want the entire bert contextual embedding or just the subword embeddings?
Comments.UserDisplayName: mshlis
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-06-18T05:07:21.090Z
Comments.Text: I would need the contextual embeddings.
Comments.UserDisplayName: Srikant Jayaraman
Tags: <deep-learning><keras><word-embedding><long-short-term-memory><bert>
Title: Adding BERT embeddings in LSTM embedding layer
UserDisplayName: Srikant Jayaraman
ThreadText: # Adding BERT embeddings in LSTM embedding layer
  
  (Asked by: username_0 on 2019-06-17)
  
  I am planning to use BERT embeddings in the LSTM embedding layer instead of the usual Word2vec/Glove Embeddings. What are the possible ways to do that?
  
  **Tags:** deep-learning, keras, word-embedding, long-short-term-memory, bert
  > **Comments:**
  > **username_1**: do you want the entire bert contextual embedding or just the subword embeddings?  
  > **username_0**: I would need the contextual embeddings.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-06-17. Score: 3
  
  Instead of using the Embedding() layer directly, you can create a new bertEmbedding() layer and use it instead.
  # Sample code
  # Model architecture
  
  # Custom BERT layer
  bert_output = BertLayer(n_fine_tune_layers=10)(bert_inputs)
  
  # Build the rest of the classifier 
  dense = tf.keras.layers.Dense(256, activation='relu')(bert_output)
  pred = tf.keras.layers.Dense(1, activation='sigmoid')(dense)
  
  model = tf.keras.models.Model(inputs=bert_inputs, outputs=pred)
  model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
  model.fit(...)
  
  This article will walk you through the entire process of creating the custom BERT layer along with example code. Give it a read.
  
  
  ---
  
  