<p>Instead of using the <code>Embedding()</code> layer directly, you can create a new <code>bertEmbedding()</code> layer and use it instead.</p>



<pre><code># Sample code
# Model architecture

# Custom BERT layer
bert_output = BertLayer(n_fine_tune_layers=10)(bert_inputs)

# Build the rest of the classifier 
dense = tf.keras.layers.Dense(256, activation='relu')(bert_output)
pred = tf.keras.layers.Dense(1, activation='sigmoid')(dense)

model = tf.keras.models.Model(inputs=bert_inputs, outputs=pred)
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
model.fit(...)
</code></pre>

<p>This <a href="https://towardsdatascience.com/bert-in-keras-with-tensorflow-hub-76bcbc9417b" rel="nofollow noreferrer">article</a> will walk you through the entire process of creating the custom BERT layer along with example code. Give it a read.</p>

CC BY-SA 4.0
2019-06-17T15:02:39.240Z
2019-06-17T15:02:39.240Z
skillsmuggler
<p>I am planning to use BERT embeddings in the LSTM embedding layer instead of the usual Word2vec/Glove Embeddings. What are the possible ways to do that? </p>

CC BY-SA 4.0
2019-06-17T15:27:26.430Z
do you want the entire bert contextual embedding or just the subword embeddings?
mshlis
CC BY-SA 4.0
2019-06-18T05:07:21.090Z
I would need the contextual embeddings.
Srikant Jayaraman
<deep-learning><keras><word-embedding><long-short-term-memory><bert>
Adding BERT embeddings in LSTM embedding layer
Srikant Jayaraman
# Adding BERT embeddings in LSTM embedding layer

(Asked by: username_0 on 2019-06-17)

I am planning to use BERT embeddings in the LSTM embedding layer instead of the usual Word2vec/Glove Embeddings. What are the possible ways to do that?

**Tags:** deep-learning, keras, word-embedding, long-short-term-memory, bert

> **Comments:**
> **username_1**: do you want the entire bert contextual embedding or just the subword embeddings?  
> **username_0**: I would need the contextual embeddings.  

---

## 1 answers

### Answer by username_2 on 2019-06-17. Score: 3

Instead of using the Embedding() layer directly, you can create a new bertEmbedding() layer and use it instead.
# Sample code
# Model architecture

# Custom BERT layer
bert_output = BertLayer(n_fine_tune_layers=10)(bert_inputs)

# Build the rest of the classifier 
dense = tf.keras.layers.Dense(256, activation='relu')(bert_output)
pred = tf.keras.layers.Dense(1, activation='sigmoid')(dense)

model = tf.keras.models.Model(inputs=bert_inputs, outputs=pred)
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
model.fit(...)

This article will walk you through the entire process of creating the custom BERT layer along with example code. Give it a read.


---

