Answers.Body: <p>If you use fully connected neural network, it will be hard to do what you want. There's no inductive bias to easily generalize from different positions of the strings which you need to extract. </p>
  
  <p>I suggest you use Seq2Seq models here: <a href="http://cs224d.stanford.edu/papers/seq2seq.pdf" rel="nofollow noreferrer">http://cs224d.stanford.edu/papers/seq2seq.pdf</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-21T02:54:54.387Z
Answers.LastActivityDate: 2018-12-21T03:02:45.353Z
Answers.LastEditDate: 2018-12-21T03:02:45.353Z
Answers.UserDisplayName: Konstantin Solomatov
Body: <p>I have a fully connected network that takes in a variable-length input padded with 0.</p>
  <p>However, the network doesn't seem to be learning and I am guessing that the high number of zeros in the input might have something to do with that.</p>
  <p>Are there solutions for dealing with padded input in fully connected layers or should I consider a different architecture?</p>
  <p><strong>UPDATE (to provide more details)</strong>:</p>
  <p>The goal of the network if to clean full file paths:
  i.g.:</p>
  <ul>
  <li><code>/My document/some folder/a file name.txt &gt; a file name</code></li>
  <li><code>/Hard drive/book/deeplearning/1.txt &gt; deeplearning</code></li>
  </ul>
  <p>The constraint is that the training data labels have been generated using a regex on the file name itself so it's not very accurate.</p>
  <p>I am hoping that by treating every word equally (without sequential information) the network would be able to generalize as to which type of word is usually kept and which is usually discarded.</p>
  <p>Then network takes in a sequence of word embedding trained on paths data and outputs logits that correspond to probabilities of each word to be kept or not.</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-03-07T08:20:18.617Z
Comments.Text: There are a few choices of re-representation or different architectures, but the best choice will depend on details of the data. Please describe more about your input data, to help readers understand what the options might be. For instance, is the data from a sequence (text or signal processing), does it contain a variable number of equivalent items? A few simplified examples of the the data may help.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-03-07T08:44:29.490Z
Comments.Text: @NeilSlater I've updated the question with additional info
Comments.UserDisplayName: silkAdmin
Tags: <deep-learning><multilayer-perceptrons>
Title: How to deal with padded inputs in a fully connected feed forward network?
UserDisplayName: silkAdmin
ThreadText: # How to deal with padded inputs in a fully connected feed forward network?
  
  (Asked by: username_0 on 2018-03-07)
  
  I have a fully connected network that takes in a variable-length input padded with 0.
  
  However, the network doesn't seem to be learning and I am guessing that the high number of zeros in the input might have something to do with that.
  
  Are there solutions for dealing with padded input in fully connected layers or should I consider a different architecture?
  
  **UPDATE (to provide more details)**:
  
  The goal of the network if to clean full file paths:
  i.g.:
  
  * `/My document/some folder/a file name.txt > a file name`
  * `/Hard drive/book/deeplearning/1.txt > deeplearning`
  
  The constraint is that the training data labels have been generated using a regex on the file name itself so it's not very accurate.
  
  I am hoping that by treating every word equally (without sequential information) the network would be able to generalize as to which type of word is usually kept and which is usually discarded.
  
  Then network takes in a sequence of word embedding trained on paths data and outputs logits that correspond to probabilities of each word to be kept or not.
  
  **Tags:** deep-learning, multilayer-perceptrons
  > **Comments:**
  > **username_1**: There are a few choices of re-representation or different architectures, but the best choice will depend on details of the data. Please describe more about your input data, to help readers understand what the options might be. For instance, is the data from a sequence (text or signal processing), does it contain a variable number of equivalent items? A few simplified examples of the the data may help.  
  > **username_0**: @NeilSlater I've updated the question with additional info  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2018-12-21. Score: 1
  
  If you use fully connected neural network, it will be hard to do what you want. There's no inductive bias to easily generalize from different positions of the strings which you need to extract.
  
  I suggest you use Seq2Seq models here: <http://cs224d.stanford.edu/papers/seq2seq.pdf>
  
  
  ---
  
  