Text embeddings reveal almost as much as text
First author here! I thought it was interesting that so many people thought this was obvious after-the-fact. What we're trying to convey is that although inversion turns out to be possible, it's not trivial.<p>Here are the three levels of inversion model described in the paper:<p>(1) Training a decoder to invert embeddings gives you some similar text, but only with some overlapping words.<p>(2) Training an encoder-decoder with the embedding projected up to a 'sequence' for the encoder works better, and gets more words back, but almost never gets the right text back exactly, especially when sequences are long.<p>(3) Training this fancy architecture as a 'correction' model that recursively corrects and re-embeds text works really well. It turns out that search is necessary for this kind of thing, and can get many texts back exactly.<p>A fourth option would be to simply scale the inversion model up by 10x or 100x, which would give us predictably better performance. We didn't try this because it's just not that interesting.