Hi all,
I was wondering if anyone has tried approximating nn.Embedding (say for embedding words) with a set of fully connected layers or something like that?
My situation (in somewhat more details) is as follows:
I have a pre-trained embedding but at inference time, I can only provide floating point tensors as input (not one-hot vectors or integers) to the Embedding, hence the need to approximate it with a neural network.
Also if there can be an approximation, what is a good loss function for training such a proxy network (L2)?
Really appreciate your time/response.