hi Dan,
I’m sorry about the memory issue in your case. It is very likely due to the high cost of TF Ops – we create one per feature, and that breaks when you have tens of thousands of features. In our list of things to fix (it should be fixable), but it will have to wait Mathieu to be back.
In the short term, there is the Yggdrasil command line trainer that won’t suffer from this – but with the disadvantage that you will not be in TF – Another TODO for us is to create a converter from Yggdrasil model to TF SavedModel, so it can easily be imported back to TF. Internally we have that, but it needs some updates before OSSing. Hopefully we can get this done this quarter.
Apologies for not being able to fix this faster …
Note: DNNs usually do all in fewer (but “fatter”) TF Ops, that do giant matrix multiplications (so they take O(num_layers) TF Ops). If a DNN had 10s of thousands of layers, it would likely suffer from the same issue.