Body: <p><strong>Assumption</strong></p>
  <p>Let's assume we collect a high quality amount of training data for machine translation for example parallel corpus data from the european parlament combined with other data. We store these texts in a database, like a graph data base, which is a fast operation even for data of size 100 gb.</p>
  <p>Now, when we have an input text that we want to translate into an other language, we lookup the relevant training data in the graph and limit it to some constant size. Maybe, we also additionally collect some other constant size of texts to learn the general grammar.</p>
  <p>Now, we have collected relevant training data in constant runtime and can use this <em>to train a model at inference time</em> using slow state of the art solutions that become fast enough for small inputs.</p>
  <p>We have created a fast to train and adaptive model for machine translation that will be slower at inference time but fast enough.</p>
  <p><strong>Question</strong></p>
  <ul>
  <li>What are the downsides of this approach?</li>
  <li>Can it nearly achieve the quality like training the whole data set
  <ul>
  <li>for machine translation</li>
  <li>for other text related problems</li>
  <li>for other problems like image-related</li>
  </ul>
  </li>
  </ul>
  <p>Thanks in advance!</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-01T07:06:32.270Z
Comments.Text: What is your reference implementation for "a fast to train and adaptive model for machine translation". The question title mentions neural networks. These are usually very slow to train, so I would like to see some example of the kind of system you have in mind. Also what kind of response time you are targeting - is it under a second, a few seconds, up to an hour?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-01T07:15:49.513Z
Comments.Text: With "fast" I mean fast enough for a normal user, like 2 minutes, at worst 10 minutes, at best less than 60 seconds. The model should use a normal state of the art approach like neural-network-based or seq2seq (I'm no expert here, but we could just look one up based on new papers and comparisons for the specific task). Since the training data is guaranteed to be constant time and rather relevant due to graph retrieval, the model is assumed to have train+inference in constant time and yield a quite ok answer. We assume an average notebook hardware.
Comments.UserDisplayName: user1232048
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-01T07:39:40.527Z
Comments.Text: So, to be clear, you don't know of any constant-training-time neural network suitable for the task, you are simply hoping that one can be made? You are also using "constant time" really loosely, when you mean "fast", for all the components.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-01T07:45:30.127Z
Comments.Text: I think you misunderstand the title. Constant time does _not_ mean that the neural network training is constant time. Insertion Sort is the slowest possible sorting algorithm for average N, but the fastest for small N. Training a neural network (obviously) requires O(N*p) runtime where p depends on the parameters. Since N is guaranteed to be constant, the training time is constant and independent on the size of our training data. Thus, I say, it does not matter at all, which model we use, we just use some slow state of the art model, which becomes fast (1 to 10 minutes) for small N.
Comments.UserDisplayName: user1232048
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-01T07:50:13.560Z
Comments.Text: Let's just assume that runtime is not a problem (it won't be a problem). My question is mainly about the quality of the results (answers). It's also ok if someone can give me the name of this approach maybe with a link to a paper that shows the downsides. In my Data Science courses I never heard this method. We learned: train all data first to one model, wait a few days, than have fast inference. I don't want to "wait a few days".
Comments.UserDisplayName: user1232048
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-01T07:53:23.297Z
Comments.Text: OK, so this is similar to the idea of using kNN (where all training data is memorised and a subset collected during inference time - BTW this search will also not be constant time in the normal meaning of that phrase, and potentially not fast, but likely good enough for purpose if your goal is on the order of minutes - e.g. you don't care that lookup time might scales as log(N)), and a refinement at the end. I cannot answer the question, and I doubt anyone can unless a similar system has already been built.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-01T07:54:21.283Z
Comments.Text: Thanks for the answer! That's already quite helpful. Indeed I assume that it might be novel.
Comments.UserDisplayName: user1232048
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-01T07:54:24.553Z
Comments.Text: There is precedent for the architecture for planning refinement in e.g. Alpha Go. But I have never heard of such an architecture being applied to translations or vision problems. Perhaps someone has.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-01T07:56:03.510Z
Comments.Text: BTW, it is unlikely that your last stage NN would be very small and trained from scratch. More likely to be a pre-trained model, where you could use the search results from the corpus to refine its tuning - because refining a medium-sized seq2seq model to improve results should be less work training a new small model from scratch, and probably more accurate. However, that is just a gut feeling/guess. You would probably have to do the experiment
Comments.UserDisplayName: Neil Slater
Tags: <machine-learning><training><training-datasets><machine-translation>
Title: Downsides of training a neural network in constant time
UserDisplayName: user1232048
ThreadText: # Downsides of training a neural network in constant time
  
  (Asked by: username_0 on 2024-07-01)
  
  **Assumption**
  
  Let's assume we collect a high quality amount of training data for machine translation for example parallel corpus data from the european parlament combined with other data. We store these texts in a database, like a graph data base, which is a fast operation even for data of size 100 gb.
  
  Now, when we have an input text that we want to translate into an other language, we lookup the relevant training data in the graph and limit it to some constant size. Maybe, we also additionally collect some other constant size of texts to learn the general grammar.
  
  Now, we have collected relevant training data in constant runtime and can use this *to train a model at inference time* using slow state of the art solutions that become fast enough for small inputs.
  
  We have created a fast to train and adaptive model for machine translation that will be slower at inference time but fast enough.
  
  **Question**
  
  * What are the downsides of this approach?
  * Can it nearly achieve the quality like training the whole data set
    + for machine translation
    + for other text related problems
    + for other problems like image-related
  
  Thanks in advance!
  
  **Tags:** machine-learning, training, training-datasets, machine-translation
  
  > **Comments:**
  > **username_1**: What is your reference implementation for "a fast to train and adaptive model for machine translation". The question title mentions neural networks. These are usually very slow to train, so I would like to see some example of the kind of system you have in mind. Also what kind of response time you are targeting - is it under a second, a few seconds, up to an hour?  
  > **username_0**: With "fast" I mean fast enough for a normal user, like 2 minutes, at worst 10 minutes, at best less than 60 seconds. The model should use a normal state of the art approach like neural-network-based or seq2seq (I'm no expert here, but we could just look one up based on new papers and comparisons for the specific task). Since the training data is guaranteed to be constant time and rather relevant due to graph retrieval, the model is assumed to have train+inference in constant time and yield a quite ok answer. We assume an average notebook hardware.  
  > **username_1**: So, to be clear, you don't know of any constant-training-time neural network suitable for the task, you are simply hoping that one can be made? You are also using "constant time" really loosely, when you mean "fast", for all the components.  
  > **username_0**: I think you misunderstand the title. Constant time does _not_ mean that the neural network training is constant time. Insertion Sort is the slowest possible sorting algorithm for average N, but the fastest for small N. Training a neural network (obviously) requires O(N*p) runtime where p depends on the parameters. Since N is guaranteed to be constant, the training time is constant and independent on the size of our training data. Thus, I say, it does not matter at all, which model we use, we just use some slow state of the art model, which becomes fast (1 to 10 minutes) for small N.  
  > **username_0**: Let's just assume that runtime is not a problem (it won't be a problem). My question is mainly about the quality of the results (answers). It's also ok if someone can give me the name of this approach maybe with a link to a paper that shows the downsides. In my Data Science courses I never heard this method. We learned: train all data first to one model, wait a few days, than have fast inference. I don't want to "wait a few days".  
  > **username_1**: OK, so this is similar to the idea of using kNN (where all training data is memorised and a subset collected during inference time - BTW this search will also not be constant time in the normal meaning of that phrase, and potentially not fast, but likely good enough for purpose if your goal is on the order of minutes - e.g. you don't care that lookup time might scales as log(N)), and a refinement at the end. I cannot answer the question, and I doubt anyone can unless a similar system has already been built.  
  > **username_0**: Thanks for the answer! That's already quite helpful. Indeed I assume that it might be novel.  
  > **username_1**: There is precedent for the architecture for planning refinement in e.g. Alpha Go. But I have never heard of such an architecture being applied to translations or vision problems. Perhaps someone has.  
  > **username_1**: BTW, it is unlikely that your last stage NN would be very small and trained from scratch. More likely to be a pre-trained model, where you could use the search results from the corpus to refine its tuning - because refining a medium-sized seq2seq model to improve results should be less work training a new small model from scratch, and probably more accurate. However, that is just a gut feeling/guess. You would probably have to do the experiment  
  
  ---
  
  ## 0 answers
  
  