Body: <p>I'm working on research in this sector where my supervisor wants to do cannonicalization of name data using VAEs, but I don't think it's possible to do, but I don't know explicitly how to show it mathematically. I just know empirically that VAEs don't do good on discrete distributions of latents and observed variables(Because in order to do names you need your latent to be the character at each index and it can be any ASCII char, which can only be represented as a distribution). So the setup I'm using is a VAE with 3 autoencoders, for latents, one for first, middle and last name and all of them sample each character of their respective names from the gumbel-softmax distribution(A form a categorical that is differentiable where the parameters is a categorical dist). From what I've seen in the original paper on the simple problem of MNIST digit image generation, the inference and generative network both did worse as latent dimension increased and as you can imagine the latent dimension of my problem is quite large. That's the only real argument for why this can't work, that I have. The other would have been it's on a discrete distribution, but I solved that by using a gumbel softmax dist instead. </p>
  
  <p>This current setup isn't working at all, the name generations are total gibberish and it plateaus really early. Are there any mathematical intuitions or reasons that VAEs won't work on a problem like this?</p>
  
  <p>As a note I've also tried semi-supervised VAEs and it didn't do much better. I even tried it for seq2seq of just first names given a first and it super failed as well and I'm talking like not even close to generation of names or the original input.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-02-13T01:37:37.930Z
Comments.Text: Im confused. For GANs you'd need Gumbell Sfotmax or REINFORCE or something similar, but for VAEs this can be done directly without approximation. I see your approach but have you tried just teacher forcing?
Comments.UserDisplayName: mshlis
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-02-13T06:49:08.847Z
Comments.Text: VAEs can't do this because they can't backprop on discrete distributions, this was stated in Bengio's paper. I don't know what teacher forcing is.
Comments.UserDisplayName: user8714896
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-02-13T20:12:59.507Z
Comments.Text: Interesting. I don't know how much that helps VAEs cause I don't think that helps it approximate the inference or generation. If anything a semi supervised VAE makes more sense
Comments.UserDisplayName: user8714896
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-02-16T23:04:00.640Z
Comments.Text: Have you tried this using a normal AE? How does exactly the same input and structure perform if it is just a standard AE? This might help troubleshoot potential logic errors that could be causing this behaviour. Sorry I can't help with a mathematical intuition, but it does seem odd to use something meant for continuous distributions for something discrete.
Comments.UserDisplayName: Recessive
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-02-16T23:10:25.613Z
Comments.Text: @Recessive true but there are discrete VAEs that handle this by using the discrete outputs as a parameter to a continuous dist like Gumbel softmax. A normal AE does pretty good
Comments.UserDisplayName: user8714896
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-02-16T23:20:06.967Z
Comments.Text: @user8714896 It's a long shot, but you *might* have some luck training an AE first, then applying gumbel softmax to the latent space of the AE to make it variational. Sort of what you're already doing (by the sounds of it) but in reverse.
Comments.UserDisplayName: Recessive
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-02-16T23:24:47.737Z
Comments.Text: @Recessive that sounds like a semi-supervised VAE. Well sort of, you're suggesting training it as a DAE then making it variational after, which is pretty interesting
Comments.UserDisplayName: user8714896
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-02-16T23:27:03.710Z
Comments.Text: @Recessive actually the more I think about it the more I like it
Comments.UserDisplayName: user8714896
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-02-16T23:28:21.310Z
Comments.Text: @user8714896 Let me know how it goes if you try it!
Comments.UserDisplayName: Recessive
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-20T21:43:37.957Z
Comments.Text: What's their terminal goal with a Name VAE? Is it just an academic challenge? If it has a real goal behind (like generating new names, or saving data with less storage) you could just suggest another plausible approach instead of trying to prove them wrong.
Comments.UserDisplayName: Andre Goulart
Tags: <variational-autoencoder>
Title: Why can't VAE do sequence to sequence name generation?
UserDisplayName: user8714896
ThreadText: # Why can't VAE do sequence to sequence name generation?
  
  (Asked by: username_0 on 2020-02-09)
  
  I'm working on research in this sector where my supervisor wants to do cannonicalization of name data using VAEs, but I don't think it's possible to do, but I don't know explicitly how to show it mathematically. I just know empirically that VAEs don't do good on discrete distributions of latents and observed variables(Because in order to do names you need your latent to be the character at each index and it can be any ASCII char, which can only be represented as a distribution). So the setup I'm using is a VAE with 3 autoencoders, for latents, one for first, middle and last name and all of them sample each character of their respective names from the gumbel-softmax distribution(A form a categorical that is differentiable where the parameters is a categorical dist). From what I've seen in the original paper on the simple problem of MNIST digit image generation, the inference and generative network both did worse as latent dimension increased and as you can imagine the latent dimension of my problem is quite large. That's the only real argument for why this can't work, that I have. The other would have been it's on a discrete distribution, but I solved that by using a gumbel softmax dist instead.
  
  This current setup isn't working at all, the name generations are total gibberish and it plateaus really early. Are there any mathematical intuitions or reasons that VAEs won't work on a problem like this?
  
  As a note I've also tried semi-supervised VAEs and it didn't do much better. I even tried it for seq2seq of just first names given a first and it super failed as well and I'm talking like not even close to generation of names or the original input.
  
  **Tags:** variational-autoencoder
  > **Comments:**
  > **username_1**: Im confused. For GANs you'd need Gumbell Sfotmax or REINFORCE or something similar, but for VAEs this can be done directly without approximation. I see your approach but have you tried just teacher forcing?  
  > **username_0**: VAEs can't do this because they can't backprop on discrete distributions, this was stated in Bengio's paper. I don't know what teacher forcing is.  
  > **username_0**: Interesting. I don't know how much that helps VAEs cause I don't think that helps it approximate the inference or generation. If anything a semi supervised VAE makes more sense  
  > **username_2**: Have you tried this using a normal AE? How does exactly the same input and structure perform if it is just a standard AE? This might help troubleshoot potential logic errors that could be causing this behaviour. Sorry I can't help with a mathematical intuition, but it does seem odd to use something meant for continuous distributions for something discrete.  
  > **username_0**: @Recessive true but there are discrete VAEs that handle this by using the discrete outputs as a parameter to a continuous dist like Gumbel softmax. A normal AE does pretty good  
  > **username_2**: @user8714896 It's a long shot, but you *might* have some luck training an AE first, then applying gumbel softmax to the latent space of the AE to make it variational. Sort of what you're already doing (by the sounds of it) but in reverse.  
  > **username_0**: @Recessive that sounds like a semi-supervised VAE. Well sort of, you're suggesting training it as a DAE then making it variational after, which is pretty interesting  
  > **username_0**: @Recessive actually the more I think about it the more I like it  
  > **username_2**: @user8714896 Let me know how it goes if you try it!  
  > **username_3**: What's their terminal goal with a Name VAE? Is it just an academic challenge? If it has a real goal behind (like generating new names, or saving data with less storage) you could just suggest another plausible approach instead of trying to prove them wrong.  
  
  ---
  
  ## 0 answers
  
  