I want to create an embedding for problem in the physical sciences, the dataset i’ve curated has a lot of missing entries and collecting data is not viable as the relevant experiments are prohibitively expensive. I want to experiment on as to whether I can train an auto-encoder with a custom loss function that excludes the entries corresponding to missing data in the output layer. If I can get near zero reconstruction loss on this custom loss function then I can reasonably take the results of the output layer as an approximation to the dense featurisation. There is also the question of whether the output layer or the latent representation would be a better embedding to use for subsequent modelling.
After thinking a bit more I realise I can achieve pretty much the same effect just by using zeros as my padding token for my missing entries. Doing this there may be an issue with the fact that some of the features are boolean with a binary encoding but I guess I will just have to run the experiment and see what the result is.
I am not sure if zeroing out the activation after the first layer is sensible because we may have co-dependence in the subsequent layers on the bias for a missing entry, but these are all things I can experiment with. If I have no bias in the first layer and use an activation that is non-zero for zero like softplus then reset the activations this may be the way to go as then the masking would provide the desired regularisation effect without impacting upon the one-hot features.
I am not going off any particular reference paper just the belief that denoising autoencoders with masked noise should be able to be re-purposed for data imputation. From a quick literature search I know this is not an original idea but nor did I find anything that I thought was an improvement on my basic idea.
I don’t think this question will be particularly useful for other users and so feel free to delete it.