Answers.Body: <p>There is a recent paper <a href="https://arxiv.org/pdf/2001.11692" rel="nofollow noreferrer">&quot;Convolutional embedding for edit distance&quot;</a> (Dai et al., 2020) that shows that a convolutional net using one-hot character-based input encoding computes vector embeddings whose Euclidean distances correspond to edit distance. So it is not surprising that, for problems where edit distance makes sense as a similarity metric among texts, this approach will perform quite well out of the box. This is a very useful inductive prior, especially if there is limited training data.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-15T22:04:54.747Z
Answers.LastActivityDate: 2024-11-29T07:51:31.173Z
Answers.LastEditDate: 2024-11-29T07:51:31.173Z
Answers.UserDisplayName: calmcc
Body: <p>I experimented with a CNN operating on texts encoded as sequences of character vectors, where characters are encoded as one-hot vectors in one embedding and as random unit length pairwise orthogonal vectors (orthogonal matrix) in another. While geometrically these encode the same vector space, the one-hot embedding outperformed the random orthogonal one consistently. I suppose this has to do with the clarity of the signal: A zero vector with a single 1-valued cell is an easier to learn signal than just <em>some</em> vector with lots of different values in each cell.</p>
  
  <p>I wondered if you know of any papers on this kind of effect. I did not find any but would like to back up this finding and check if my reasoning for why this is the case makes sense/ find a better or more in-depth explanation.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-08T10:25:43.863Z
Comments.Text: Why were you using a CNN in this case? It would not seem to offer any advantage *unless* you had repeated vectors (i.e. classifying a *sequence* of vectors in one go) where patterns would be meaningful.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-08T16:38:25.880Z
Comments.Text: @NeilSlater I use the CNN for text classification. Texts are encoded as sequences of vectors (stacked to form a 2d matrix, like an image). Each vector corresponds to one character. Does that answer your question?
Comments.UserDisplayName: lo tolmencre
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-08T16:49:33.720Z
Comments.Text: I guess you thought I was talking about optical character recognition or something?
Comments.UserDisplayName: lo tolmencre
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-08T17:03:57.500Z
Comments.Text: Yes that adds useful detail. It may be worth adding it to the question in case it helps someone, because it explains why you are using a CNN (you have *multiple* encoded items in the input layer). The current question leaves that open and makes it look like you might be trying to apply a CNN against a single one-hot-encoded vector, which would be odd
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-08T17:23:57.673Z
Comments.Text: @NeilSlater Ok. I take it then you do not know of any studies on this type of effect of ne-hot encoded data performing better than "orthogonally encoded" data, yes? This is not necessarily specific to CNNs, I'd assume.
Comments.UserDisplayName: lo tolmencre
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-08T17:33:50.277Z
Comments.Text: No I don't know of any studies and don't have any insight to the problem. Just checking that the details given in the question make sense (because one option without those details is that you have made a mistake and applying inappropriate arcitecture to your problem - you don't want someone answering with that aussumption, it wastes your and their time)
Comments.UserDisplayName: Neil Slater
Tags: <neural-networks><convolutional-neural-networks><reference-request><word-embedding><papers>
Title: Are there any papers explaining why one-hot encoding outperforms random orthogonal encoding in CNN?
UserDisplayName: lo tolmencre
ThreadText: # Are there any papers explaining why one-hot encoding outperforms random orthogonal encoding in CNN?
  
  (Asked by: username_0 on 2019-09-07)
  
  I experimented with a CNN operating on texts encoded as sequences of character vectors, where characters are encoded as one-hot vectors in one embedding and as random unit length pairwise orthogonal vectors (orthogonal matrix) in another. While geometrically these encode the same vector space, the one-hot embedding outperformed the random orthogonal one consistently. I suppose this has to do with the clarity of the signal: A zero vector with a single 1-valued cell is an easier to learn signal than just *some* vector with lots of different values in each cell.
  
  I wondered if you know of any papers on this kind of effect. I did not find any but would like to back up this finding and check if my reasoning for why this is the case makes sense/ find a better or more in-depth explanation.
  
  **Tags:** neural-networks, convolutional-neural-networks, reference-request, word-embedding, papers
  > **Comments:**
  > **username_1**: Why were you using a CNN in this case? It would not seem to offer any advantage *unless* you had repeated vectors (i.e. classifying a *sequence* of vectors in one go) where patterns would be meaningful.  
  > **username_0**: @NeilSlater I use the CNN for text classification. Texts are encoded as sequences of vectors (stacked to form a 2d matrix, like an image). Each vector corresponds to one character. Does that answer your question?  
  > **username_0**: I guess you thought I was talking about optical character recognition or something?  
  > **username_1**: Yes that adds useful detail. It may be worth adding it to the question in case it helps someone, because it explains why you are using a CNN (you have *multiple* encoded items in the input layer). The current question leaves that open and makes it look like you might be trying to apply a CNN against a single one-hot-encoded vector, which would be odd  
  > **username_0**: @NeilSlater Ok. I take it then you do not know of any studies on this type of effect of ne-hot encoded data performing better than "orthogonally encoded" data, yes? This is not necessarily specific to CNNs, I'd assume.  
  > **username_1**: No I don't know of any studies and don't have any insight to the problem. Just checking that the details given in the question make sense (because one option without those details is that you have made a mistake and applying inappropriate arcitecture to your problem - you don't want someone answering with that aussumption, it wastes your and their time)  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-10-15. Score: 0
  
  There is a recent paper ["Convolutional embedding for edit distance"](https://arxiv.org/pdf/2001.11692) (Dai et al., 2020) that shows that a convolutional net using one-hot character-based input encoding computes vector embeddings whose Euclidean distances correspond to edit distance. So it is not surprising that, for problems where edit distance makes sense as a similarity metric among texts, this approach will perform quite well out of the box. This is a very useful inductive prior, especially if there is limited training data.
  
  
  ---
  
  