Any insight to this mystery issue? Using the Keras functional API results in whole deleted weights/layers for transformer layers. 
# ❓ Questions & Help

This is probably a Keras, tensorflow bug, but just wanted to check-in here in case I overlooked anything. 

I've discovered a bug, where if transformer layers are copied from a transformer model, and used as individual layers, and then a Keras model is made using the functional API, this seems to result in missing weights/layers from the list of trainable layers. However, the issue goes away if I use model subclassing to make the Keras model. 

I raised the issue here

https://github.com/tensorflow/tensorflow/issues/40638#event-3468314954

but you can directly checkout the issue in this colab notebook 

https://colab.research.google.com/gist/Santosh-Gupta/273361f873e4daf572fddea691b1f325/missingtrainablevars.ipynb

Which copies the layers from one of your transformer models. I also made one where I implemented the transformer layers from (near) scratch, and got the same result

https://colab.research.google.com/gist/ravikyram/0191e12b7c6d9afeb80ccc009870b255/untitled52.ipynb

This likely seems like a bug in Keras, but may take a while to pinpoint what exactly is causing this, since a transformer layer has many parts. But just wanted to pop this in here, in case there was some insight to the issue, or a way to pinpoint the cause. 
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
