Answers.Body: <p>Your network is essentially memorizing data but not extracting features. You need to apply CNN. </p>
  
  <p>That said the CNN architecture will need to be kind of unusual, each bit will need to be turned into representing a positional element. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-20T20:22:30.577Z
Answers.Text: Can you please clarify what you mean by "network is essentially memorizing data" -> we are currently getting a very low training error but test performance is bad. This result basically seems to imply what you said above. However, can you please explain why this is happening with the "dense" layers? Any pointers to outside resources on this phenomenon will also be greatly appreciated.
Answers.UserDisplayName: Phani
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-20T20:26:15.477Z
Answers.Text: just look up CNN or time series analysis.
Answers.UserDisplayName: FourierFlux
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-20T20:18:17.217Z
Answers.LastActivityDate: 2020-04-20T20:18:17.217Z
Answers.UserDisplayName: FourierFlux
Answers.Body: <p>I was writing a comment, but I'll write an answer instead. With a dense network your model has too many parameters an will overfit to the data. Even if it learns that &quot;0 and 100 tend to appear together&quot; at the input's positions 34 and 35, it wouldn't automatically learn that this pattern is common also an all other positions. CNNs don't have this problem.</p>
  <p>I also suggest that you change the network's input, if it is now just a series of integers 0 - 255. I would either one-hot encode them, or pass the values through an embedding layer. Typically CNNs are used for image or sound signals where the values have a natural order, but data file bytes don't have such structure.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-11T12:49:58.480Z
Answers.LastActivityDate: 2022-05-11T12:49:58.480Z
Answers.UserDisplayName: NikoNyrh
Body: <p>I am a newbie in deep learning and wanted to know if the problem I have at hand is a suitable fit for deep learning algorithms. I have thousands of fragments each of about 1000 bytes size (i.e. numbers in the range of 0 to 255). There are two classes in the fragments:</p>
  
  <ol>
  <li>Some fragments have a high frequency of two particular byte values appearing next to one another: "0 and 100". This kind of pattern roughly appears once every 100 to 200 bytes.</li>
  <li>In the other class, the byte values are more randomly distributed. </li>
  </ol>
  
  <p>We have the ability to produce as many numbers of instances of each class as needed for training purposes.  However, I would like to differentiate with a machine learning algorithm without explicitly identifying the "0 and 100" pattern in the 1st class myself. Can deep learning help us solve this? If so, what kind of layers might be useful?</p>
  
  <p>As a preliminary experiment, we tried to train a deep learning network made up of 2 hidden layers of TensorFlow's "Dense" layers (of size 512 and 256 nodes in each of the hidden layers). However, unfortunately, our accuracy was indicative of simply a random guess (i.e. 50% accuracy). We were wondering why the results were so bad. Do you think a Convolutional Neural Network will better solve this problem? </p>
  
Tags: <neural-networks><machine-learning><deep-learning><tensorflow>
Title: Finding patterns in binary files using deep learning
UserDisplayName: Phani
ThreadText: # Finding patterns in binary files using deep learning
  
  (Asked by: username_0 on 2020-04-20)
  
  I am a newbie in deep learning and wanted to know if the problem I have at hand is a suitable fit for deep learning algorithms. I have thousands of fragments each of about 1000 bytes size (i.e. numbers in the range of 0 to 255). There are two classes in the fragments:
  
  1. Some fragments have a high frequency of two particular byte values appearing next to one another: "0 and 100". This kind of pattern roughly appears once every 100 to 200 bytes.
  2. In the other class, the byte values are more randomly distributed.
  
  We have the ability to produce as many numbers of instances of each class as needed for training purposes. However, I would like to differentiate with a machine learning algorithm without explicitly identifying the "0 and 100" pattern in the 1st class myself. Can deep learning help us solve this? If so, what kind of layers might be useful?
  
  As a preliminary experiment, we tried to train a deep learning network made up of 2 hidden layers of TensorFlow's "Dense" layers (of size 512 and 256 nodes in each of the hidden layers). However, unfortunately, our accuracy was indicative of simply a random guess (i.e. 50% accuracy). We were wondering why the results were so bad. Do you think a Convolutional Neural Network will better solve this problem?
  
  **Tags:** neural-networks, machine-learning, deep-learning, tensorflow
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2020-04-20. Score: 1
  
  Your network is essentially memorizing data but not extracting features. You need to apply CNN.
  
  That said the CNN architecture will need to be kind of unusual, each bit will need to be turned into representing a positional element.
  
  > **Comments:**
  > **username_0**: Can you please clarify what you mean by "network is essentially memorizing data" -> we are currently getting a very low training error but test performance is bad. This result basically seems to imply what you said above. However, can you please explain why this is happening with the "dense" layers? Any pointers to outside resources on this phenomenon will also be greatly appreciated.  
  > **username_1**: just look up CNN or time series analysis.  
  
  ---
  
  ### Answer by username_2 on 2022-05-11. Score: 1
  
  I was writing a comment, but I'll write an answer instead. With a dense network your model has too many parameters an will overfit to the data. Even if it learns that "0 and 100 tend to appear together" at the input's positions 34 and 35, it wouldn't automatically learn that this pattern is common also an all other positions. CNNs don't have this problem.
  
  I also suggest that you change the network's input, if it is now just a series of integers 0 - 255. I would either one-hot encode them, or pass the values through an embedding layer. Typically CNNs are used for image or sound signals where the values have a natural order, but data file bytes don't have such structure.
  
  
  ---
  
  