Add an LSTM and CNN layer on top of BERT embeddings for sentiment analysis task
I am trying to add an LSTM and a convolutional layer on top of my BERT embeddings using the Transformers package in Tensorflow for a sentiment analysis task. Does someone know how I can go about that?
Hi @johnahug, I would recommend checking out how the TFBertFor* models work and trying a similar method to add the desired layers via Keras, for example: https://github.com/huggingface/transformers/blob/155c782a2ccd103cf63ad48a2becd7c76a7d2115/transformers/modeling_tf_bert.py#L834

I might be able to help out more at a later time, but I'm not sure exactly when just yet. So it might be worthwhile to go ahead and try it out :)<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>Hi @watkinsm any update on this?<|||||>hey @johnahug  have you got what you were looking for? I am also trying to add CNN to my bert embedding but have no idea how to