Answers.Body: <p>If you have a fixed predictor, then yes. If the predictor is not fixed but deterministic, the feasibility depends on the effort needed to update the predictor, and ensuring that messages include a time stamp to ensure the correct version of the predictor is used for compression and inflation.</p>
  <p>You get a really nice property if the prediction order is in order of probability that the word occurs, and if you use fewer bits for lower numbers. You would get something that is pretty close to <a href="https://en.wikipedia.org/wiki/Shannon_coding" rel="nofollow noreferrer">Shannon coding</a>, which is not optimal but is still valid.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-26T13:43:32.750Z
Answers.Text: Thanks for the reply. I guess the main challenge is to make it fast yet, effective. i guess having a limit of 100 predictions can work. I wonder if there is any research , research paper or program/demos etc done for this concept.
Answers.UserDisplayName: sktguha
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-26T13:45:08.193Z
Answers.Text: In more detail the approach is
  
  Deploy the same model on both sides
  Predict the next word after the starting word
  Keep the prediction limit say 100
  For any word which have more than 2 characters we do the prediction
  If the current word is predicted within the top 100 predictions of the model , we can essentially replace it with a numeric char between 0-99 (inclusive)
Answers.UserDisplayName: sktguha
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-26T12:37:31.830Z
Answers.LastActivityDate: 2020-08-26T12:37:31.830Z
Answers.UserDisplayName: Robby Goetschalckx
Body: <p>Is there any effort made to compress text (and maybe other media) using prediction of next word and thus sending only the order number of the word/token which will be predicted on the client side
  i.e<br>
  <strong>Server text</strong>: This is an example of a long text example, custom word flerfom inserted to confuse, that may appear on somewhere<br>
  <strong>Compressed Text transmitted</strong> : This [choice no 3] [choice no 4] [choice no 1] [choice no 6] [choice no 1] [choice no 3] [choice no 1], custom word flerfom [choice no 4] inserted [choice no 4] confuse [choice no 5] [choice no 4] [choice no 6] [choice no 5] on somewhere
  <br><br>(Note: of course [choice no 3] will be shortened to [3] to save bytes and also maybe we can do much better in some cases by sending the first letter of the word)
  <br><br>of course it means that the client side neural network has to be static or only updated in a predictable fasion, so the server knows for sure that the client neural network's predictions will follow the given choice orders. I tried example with <a href="https://demo.allennlp.org/next-token-lm" rel="nofollow noreferrer">https://demo.allennlp.org/next-token-lm</a>, but the prediction is not that good. maybe gpt-3 can do better . but its too heavy for use in a normal pc / mobile device</p>
  <p>In more details, the process is</p>
  <p>Deploy the same model on both sides<br>
  Predict the next word after the starting word<br>
  Keep the prediction limit say 100<br>
  For any word which have more than 2 characters we do the prediction<br>
  If the current word is predicted within the top 100 predictions of the model , we can essentially replace it with a numeric char between 0-99 (inclusive) so we are replacing a say , 5 character word with a 2 character numerical char..<br>
  And if the word is not predicted in top 100 we send the word as it is..<br>
  As much better the model predicts, that much better the compression<br>
  And under no scenario it will work worse than the existing method..<br></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-27T00:07:49.587Z
Comments.Text: So, essentially, the twin models have [complete information](https://en.wikipedia.org/wiki/Complete_information) on the decision making process of the the other?  It's an interesting idea, and, in theory, should work.  (One issue could be transmission noise, but that can be mitigated with redundant signals per [information theory](https://en.wikipedia.org/wiki/Information_theory).)
Comments.UserDisplayName: DukeZhou
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-27T09:38:40.967Z
Comments.Text: @DukeZhou well yes. I mean if you have same weights and other things same, then predictions should be always same right ?
Comments.UserDisplayName: sktguha
Tags: <natural-language-processing><reference-request><prediction><game-theory><information-theory>
Title: Compressing text using AI by sending only prediction rank of next word
UserDisplayName: sktguha
ThreadText: # Compressing text using AI by sending only prediction rank of next word
  
  (Asked by: username_0 on 2020-08-26)
  
  Is there any effort made to compress text (and maybe other media) using prediction of next word and thus sending only the order number of the word/token which will be predicted on the client side
  i.e  
  **Server text**: This is an example of a long text example, custom word flerfom inserted to confuse, that may appear on somewhere  
  **Compressed Text transmitted** : This [choice no 3] [choice no 4] [choice no 1] [choice no 6] [choice no 1] [choice no 3] [choice no 1], custom word flerfom [choice no 4] inserted [choice no 4] confuse [choice no 5] [choice no 4] [choice no 6] [choice no 5] on somewhere
    
    
  (Note: of course [choice no 3] will be shortened to [3] to save bytes and also maybe we can do much better in some cases by sending the first letter of the word)
    
    
  of course it means that the client side neural network has to be static or only updated in a predictable fasion, so the server knows for sure that the client neural network's predictions will follow the given choice orders. I tried example with <https://demo.allennlp.org/next-token-lm>, but the prediction is not that good. maybe gpt-3 can do better . but its too heavy for use in a normal pc / mobile device
  
  In more details, the process is
  
  Deploy the same model on both sides  
  Predict the next word after the starting word  
  Keep the prediction limit say 100  
  For any word which have more than 2 characters we do the prediction  
  If the current word is predicted within the top 100 predictions of the model , we can essentially replace it with a numeric char between 0-99 (inclusive) so we are replacing a say , 5 character word with a 2 character numerical char..  
  And if the word is not predicted in top 100 we send the word as it is..  
  As much better the model predicts, that much better the compression  
  And under no scenario it will work worse than the existing method..
  
  **Tags:** natural-language-processing, reference-request, prediction, game-theory, information-theory
  
  > **Comments:**
  > **username_1**: So, essentially, the twin models have [complete information](https://en.wikipedia.org/wiki/Complete_information) on the decision making process of the the other?  It's an interesting idea, and, in theory, should work.  (One issue could be transmission noise, but that can be mitigated with redundant signals per [information theory](https://en.wikipedia.org/wiki/Information_theory).)  
  > **username_0**: @username_1 well yes. I mean if you have same weights and other things same, then predictions should be always same right ?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-08-26. Score: 1
  
  If you have a fixed predictor, then yes. If the predictor is not fixed but deterministic, the feasibility depends on the effort needed to update the predictor, and ensuring that messages include a time stamp to ensure the correct version of the predictor is used for compression and inflation.
  
  You get a really nice property if the prediction order is in order of probability that the word occurs, and if you use fewer bits for lower numbers. You would get something that is pretty close to [Shannon coding](https://en.wikipedia.org/wiki/Shannon_coding), which is not optimal but is still valid.
  
  > **Comments:**
  > **username_0**: Thanks for the reply. I guess the main challenge is to make it fast yet, effective. i guess having a limit of 100 predictions can work. I wonder if there is any research , research paper or program/demos etc done for this concept.  
  > **username_0**: In more detail the approach is  Deploy the same model on both sides Predict the next word after the starting word Keep the prediction limit say 100 For any word which have more than 2 characters we do the prediction If the current word is predicted within the top 100 predictions of the model , we can essentially replace it with a numeric char between 0-99 (inclusive)  
  
  ---
  
  