Answers.Body: <p>If you prefer to use Python (rather than Java, which is used to implement <a href="https://moa.cms.waikato.ac.nz/" rel="nofollow noreferrer">MOA</a>, which is suggested in <a href="https://ai.stackexchange.com/a/12958/2444">the other answer</a>), you might want to have a look at the Python's <a href="https://pypi.org/project/creme/" rel="nofollow noreferrer"><code>creme</code></a> library, whose API is described at <a href="https://creme-ml.github.io/api.html" rel="nofollow noreferrer">https://creme-ml.github.io/api.html</a>, which is a library for incremental and online learning. In particular, you might be interested in the class <a href="https://creme-ml.github.io/generated/creme.multiclass.OneVsRestClassifier.html#creme.multiclass.OneVsRestClassifier" rel="nofollow noreferrer"><code>OneVsRestClassifier</code></a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-20T18:08:00.453Z
Answers.LastActivityDate: 2019-07-20T18:08:00.453Z
Answers.UserDisplayName: nbro
Answers.Body: <p>I suggest using Data Stream algorithms to try on your problems since you are asking for "online learning with singular values or minibatches as the dataset is too big too load all the training data into memory."</p>
  
  <p><a href="https://moa.cms.waikato.ac.nz/" rel="nofollow noreferrer">MOA</a> is a good choice for these algorithms. <a href="https://www.cs.waikato.ac.nz/~abifet/MOA/API/classmoa_1_1classifiers_1_1trees_1_1_hoeffding_tree.html" rel="nofollow noreferrer">Hoeffding Trees</a> is also a good first  choice to try.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-20T13:04:01.453Z
Answers.LastActivityDate: 2019-06-20T13:04:01.453Z
Answers.UserDisplayName: Atilla Ozgur
Answers.Body: <p>The way I would approach this (and which I already applied in the past for similar a similar task) is based on multiple points. Let's suppose we're working with PyTorch.</p>
  <ol>
  <li><p>I'd start normalizing every input in <span class="math-container">$[-1, 1]$</span>. Since every feature of the input is in <span class="math-container">$\mathbb{Z}$</span> and <span class="math-container">$\in [-63, +63]$</span>, normalizing it and casting it to fp16 can lower the overall dimension of the batches in the memory, and can allow for automatic mixed precision training. Matter of fact, you should really consider normalizing the values in order to obtain smaller weights in the network.</p>
  </li>
  <li><p>Same thing for the output: Since your task is effectively a bounded regression (those 127 values are not classes), you'd be better off with a single output neuron that can predict values <span class="math-container">$\in [-1, 1]$</span> using a <em>tanh</em> activation function. This would rescale the last layer of your network from <span class="math-container">$Q*H$</span> (not including biases) to <span class="math-container">$Q$</span> weights, where <span class="math-container">$Q$</span> is the number of neurons in the last hidden layer and <span class="math-container">$H$</span> is the number of output neurons. During inference you can multiply your output by <span class="math-container">$63$</span>, round it to the nearest integer and obtain your prediction.</p>
  </li>
  <li><p>Since you only have 4 features ranging <span class="math-container">$[-63, 63]$</span> you have only <span class="math-container">$260144641 = 127^4$</span> values, and it'd be interesting to study the distribution of each single feature, pair of features and triplets of feature, in order to understand if there are some kind of repeated values or collinearities.</p>
  </li>
  <li><p>Usually, earlier layers of the neural networks tend to learn more basic patterns, while deep layers handle more complex, fine details. I'd still suggest, based on my experience and the previous point, to add more neurons to early layers, since you have loads of data (the magnitude of your dataset makes me think it's some sort of scraped values with lots of noise) and lots of updates are gonna hit the weight space in those dimensions. I can sense that the distribution of those features are not uniform, thus you'd avoid handling problems like small differences in input leading to big differences in output with less neurons in the early layers.</p>
  </li>
  <li><p>Yes, the whole dataset can't fit inside the <em>memory</em>, but it can fit inside your <em>disk</em> (if I understood correctly, you are training with a large, offline dataset rather than an online stream of data). There is no need to load it at once though! You can serialize batches of processed data on your disk, and load/unload the batches you need for training, focusing on the biggest batch you can fit in the remaining memory in order to obtain the best results. Of course you're going to account for a time complexity overhead but that was your problem for the get-go. With this solution you will train a neural network without using exotic libraries or frameworks, and focus only on hyperparameter tuning and metrics.</p>
  </li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-08-23T15:23:44.257Z
Answers.LastActivityDate: 2024-08-23T15:23:44.257Z
Answers.UserDisplayName: Nope
Body: <p>I have the following problem. We have <span class="math-container">$4$</span> separate <em>discrete</em> inputs, which can take any integer value between <span class="math-container">$-63$</span> and <span class="math-container">$63$</span>. The output is also supposed to be a discrete value between <span class="math-container">$-63$</span> and <span class="math-container">$63$</span>. Another constraint is that the solution should allow for online learning with singular values or mini-batches, as the dataset is too big to load all the training data into memory.</p>
  
  <p>I have tried the following method, but the predictions are not good.</p>
  
  <p>I created an MLP or feedforward network with <span class="math-container">$4$</span> inputs and <span class="math-container">$127$</span> outputs. The inputs are being fed without normalization. The number of hidden layers is <span class="math-container">$4$</span> with <span class="math-container">$[8,16,32,64]$</span> units in each (respectively). So, essentially, this treats the problem like a sequence classification problem. For training, we feed the non-normalized input along with a one-hot encoded vector for that specific value as output. The inference is done the same way. Finding the hottest output and returning that as the next number in the sequence.</p>
  
Tags: <machine-learning><ai-design><classification><multilayer-perceptrons><online-learning>
Title: Which online machine learning technique to use for multi-class classification problem with multiple inputs?
UserDisplayName: John Mayer
ThreadText: # Which online machine learning technique to use for multi-class classification problem with multiple inputs?
  
  (Asked by: username_0 on 2019-06-20)
  
  I have the following problem. We have $4$ separate *discrete* inputs, which can take any integer value between $-63$ and $63$. The output is also supposed to be a discrete value between $-63$ and $63$. Another constraint is that the solution should allow for online learning with singular values or mini-batches, as the dataset is too big to load all the training data into memory.
  
  I have tried the following method, but the predictions are not good.
  
  I created an MLP or feedforward network with $4$ inputs and $127$ outputs. The inputs are being fed without normalization. The number of hidden layers is $4$ with $[8,16,32,64]$ units in each (respectively). So, essentially, this treats the problem like a sequence classification problem. For training, we feed the non-normalized input along with a one-hot encoded vector for that specific value as output. The inference is done the same way. Finding the hottest output and returning that as the next number in the sequence.
  
  **Tags:** machine-learning, ai-design, classification, multilayer-perceptrons, online-learning
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2019-07-20. Score: 0
  
  If you prefer to use Python (rather than Java, which is used to implement [MOA](https://moa.cms.waikato.ac.nz/), which is suggested in [the other answer](https://ai.stackexchange.com/a/12958/2444)), you might want to have a look at the Python's [`creme`](https://pypi.org/project/creme/) library, whose API is described at <https://creme-ml.github.io/api.html>, which is a library for incremental and online learning. In particular, you might be interested in the class [`OneVsRestClassifier`](https://creme-ml.github.io/generated/creme.multiclass.OneVsRestClassifier.html#creme.multiclass.OneVsRestClassifier).
  
  
  ---
  
  ### Answer by username_2 on 2019-06-20. Score: 0
  
  I suggest using Data Stream algorithms to try on your problems since you are asking for "online learning with singular values or minibatches as the dataset is too big too load all the training data into memory."
  
  [MOA](https://moa.cms.waikato.ac.nz/) is a good choice for these algorithms. [Hoeffding Trees](https://www.cs.waikato.ac.nz/~abifet/MOA/API/classmoa_1_1classifiers_1_1trees_1_1_hoeffding_tree.html) is also a good first choice to try.
  
  
  ---
  
  ### Answer by username_3 on 2024-08-23. Score: 0
  
  The way I would approach this (and which I already applied in the past for similar a similar task) is based on multiple points. Let's suppose we're working with PyTorch.
  
  1. I'd start normalizing every input in $[-1, 1]$. Since every feature of the input is in $\mathbb{Z}$ and $\in [-63, +63]$, normalizing it and casting it to fp16 can lower the overall dimension of the batches in the memory, and can allow for automatic mixed precision training. Matter of fact, you should really consider normalizing the values in order to obtain smaller weights in the network.
  2. Same thing for the output: Since your task is effectively a bounded regression (those 127 values are not classes), you'd be better off with a single output neuron that can predict values $\in [-1, 1]$ using a *tanh* activation function. This would rescale the last layer of your network from $Q\*H$ (not including biases) to $Q$ weights, where $Q$ is the number of neurons in the last hidden layer and $H$ is the number of output neurons. During inference you can multiply your output by $63$, round it to the nearest integer and obtain your prediction.
  3. Since you only have 4 features ranging $[-63, 63]$ you have only $260144641 = 127^4$ values, and it'd be interesting to study the distribution of each single feature, pair of features and triplets of feature, in order to understand if there are some kind of repeated values or collinearities.
  4. Usually, earlier layers of the neural networks tend to learn more basic patterns, while deep layers handle more complex, fine details. I'd still suggest, based on my experience and the previous point, to add more neurons to early layers, since you have loads of data (the magnitude of your dataset makes me think it's some sort of scraped values with lots of noise) and lots of updates are gonna hit the weight space in those dimensions. I can sense that the distribution of those features are not uniform, thus you'd avoid handling problems like small differences in input leading to big differences in output with less neurons in the early layers.
  5. Yes, the whole dataset can't fit inside the *memory*, but it can fit inside your *disk* (if I understood correctly, you are training with a large, offline dataset rather than an online stream of data). There is no need to load it at once though! You can serialize batches of processed data on your disk, and load/unload the batches you need for training, focusing on the biggest batch you can fit in the remaining memory in order to obtain the best results. Of course you're going to account for a time complexity overhead but that was your problem for the get-go. With this solution you will train a neural network without using exotic libraries or frameworks, and focus only on hyperparameter tuning and metrics.
  
  
  ---
  
  