Move TF building to an actual build() method
This has been a longstanding dream of mine: To move all TF model building into a proper `build()` method, using symbolic tensors instead of actual dummies. This would allow us to, among other things, stop our very hacky overriding of `save_spec`, as well as allowing us to build our TF models with zero device flops (although the speedup may be system-dependent, as we do have some compile time with this approach). It would make our models much closer to the Keras standard, which would stop Chollet casting curses upon me from afar.

In the past, we've run into serious problems with tensor names moving around when we tried this - I think I've figured out why, though, and I have a couple of ideas to resolve that without lots of hacky edge-case code.

This is an extremely draft PR that will break everything until I finish testing it properly!

**Update:** Using symbolic tensors is much slower - it works in most cases, but increases the time it takes for our tests to run by a factor of ~4, which is probably not acceptable. Instead, I'm going to rework this PR to move to a standard build() method using actual dummies. With some optimizations, I believe we can make this work, while still preserving most of the benefits of this PR, including not repeating the build unnecessarily and adding the ability to override `build()` to speed up our slowest models
_The documentation is not available anymore as the PR was closed or merged._<|||||>This should be ready to review now! Some tests failing, but that looks like Hub connection issues<|||||>Actually, I should explain my reasoning for some of the changes here - you're probably right that I can improve the API, though!

Firstly, the removal of `tf.cond` is actually not a necessary part of this PR anymore, but it is good practice (Longformer and LED are the only two models in all of Transformers that use it in their modelling code). The reason is because of the Keras call stack. In the `__call__` method for any TF module, Keras appends that layer to the call stack, and enters that layer's namespace. This means that if you have `self.bert` and that calls `self.encoder` and that calls `self.attn`, Keras will be in the `bert/encoder/attn` namespace. 

Incredibly, though, `tf.cond` counts as a layer with its own namespace, but **only when the tf.cond is not being eagerly evaluated**. In my initial PR, I was trying to replace our dummies with symbolic TF tensors, which meant the `tf.cond` was not evaluated at compile time, but instead had to be compiled as a conditional in the model graph. The result is that all layer weights inside the conditional got encapsulated in a `/cond.1/` namespace. This broke compatibility with existing checkpoints. 

Removing `tf.cond` helped, but to be safe I added a manual build to those layers to directly control the weight naming regardless of what the call stack thought it should be. As a result, I could probably revert the `tf.cond` calls, but I think it's preferable if we don't, and just try to keep it out of modelling code and just use `if` statements instead (which TF can compile into graph conditionals if it can't resolve the branch to be chosen at compile time). `tf.cond` is fine in generation code where no weight names are created.

Secondly, the distinction between `build()` and `build_with_dummies()` is a bit of an ugly hack - I think I could probably remove `build_with_dummies()` entirely, but there was a piece of the TF-PT crossloading code that only worked if it could build the model with specific inputs of its choice. I added `build_with_dummies()` to support that, with a separate `built_with_dummies` flag to make sure that any repeated calls wouldn't waste more time. However, it would probably make more sense to just manually pass the inputs through the model in those particular crossloading functions and delete the method and the flag. WDYT?<|||||>> tf.cond counts as a layer with its own namespace, but only when the tf.cond is not being eagerly evaluated.

😑 

In this case, let's rid ourselves of this pseudolayer! I'm pro the if/else changes :) 

>  it would probably make more sense to just manually pass the inputs through the model in those particular crossloading functions and delete the method and the flag. WDYT?

Yep, that's what I would go for. Would it be possible to still have some of the logic to exit early if already built? Or would this be to tricky to handle to be worth it? <|||||>I think we could, but it's probably not necessary - the only cases where we build the model with specific inputs are in weird PT-TF crossloading functions, which should always be called during or near model init anyway, so I think it's fine if there's a risk of a little bit of duplicated work there to save on overall code complexity.<|||||>@amyeroberts Done! `build_with_dummies` is no more<|||||>Also, this PR looks ready but I'm going to let it sit for a couple of days to make sure the CI is working again after my last library-breaking PR, then merge it.<|||||>Change of plans: The CI is working except for OOM errors during building for some of the pipelines, and since this cleans up building a bit we're going to merge this one too and see if it helps. If it doesn't, I'll open a new PR to see if I can lower the memory usage in the affected models.