Answers.Body: <p>Though you've already accepted an answer, from your comments it seems you're still somewhat unclear and need an example. Take your example of (binary) classification, usually balanced training data means you have more or less equal number of samples for each class label. Any due to the intrinsic random nature of each mini-batch during gradient descent (MBGD), it's very likely some mini-batches data will not be representative in the sense that it may become imbalanced. But so long as MBGD converges, it's generally converge faster and more computationally efficient leveraging parallelism than full batch GD, the impact of possibly many imbalanced mini-batches is typically minimized as the model updates its parameters based on the entire dataset and over multiple epochs.</p>
  <p>Here's a simple binary logistic classification example for a balanced dataset with two class labels of equal number of samples (50 for each) and two features, code is provided at then end. Total dataset has 100 samples, mini-batch size is set to 5, and epoch is tuned to 1000. As you can see from the output many mini-batches are not representative for the whole dataset (many have 0%, 20%, 60%, 80% class_1 data), but the final result is still good and close to the full batch result as you can easily verify.</p>
  <p><a href="https://i.sstatic.net/7jmbg.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/7jmbg.png" alt="enter image description here" /></a></p>
  <pre><code>output result:
  i=0, minibatch_size=5 class_1%=0.6
  i=5, minibatch_size=5 class_1%=0.4
  i=10, minibatch_size=5 class_1%=0.4
  i=15, minibatch_size=5 class_1%=0.8
  i=20, minibatch_size=5 class_1%=1.0
  i=25, minibatch_size=5 class_1%=0.6
  i=30, minibatch_size=5 class_1%=0.0
  i=35, minibatch_size=5 class_1%=0.4
  i=40, minibatch_size=5 class_1%=0.8
  i=45, minibatch_size=5 class_1%=0.2
  i=50, minibatch_size=5 class_1%=0.4
  i=55, minibatch_size=5 class_1%=0.6
  i=60, minibatch_size=5 class_1%=0.6
  i=65, minibatch_size=5 class_1%=0.8
  i=70, minibatch_size=5 class_1%=0.8
  i=75, minibatch_size=5 class_1%=0.6
  i=80, minibatch_size=5 class_1%=0.4
  i=85, minibatch_size=5 class_1%=0.2
  i=90, minibatch_size=5 class_1%=0.2
  i=95, minibatch_size=5 class_1%=0.2
  i=0, minibatch_size=5 class_1%=0.6
  i=5, minibatch_size=5 class_1%=0.2
  i=10, minibatch_size=5 class_1%=0.2
  i=15, minibatch_size=5 class_1%=0.8
  i=20, minibatch_size=5 class_1%=0.4
  ...
  i=95, minibatch_size=5 class_1%=0.2
  ----
  import numpy as np
  import matplotlib.pyplot as plt
  
  np.random.seed(42)
  
  # Class 0: Generate points in a cluster around (-1, -1)
  num_class_0 = 50
  X_class_0 = np.random.randn(num_class_0, 2) - 1 * np.ones((num_class_0, 2))
  y_class_0 = np.zeros((num_class_0,))
  
  # Class 1: Generate points in a cluster around (1, 1)
  num_class_1 = 50 
  X_class_1 = np.random.randn(num_class_1, 2) + 1 * np.ones((num_class_1, 2))
  y_class_1 = np.ones((num_class_1,))
  
  X = np.concatenate((X_class_0, X_class_1), axis=0)
  X_b = np.c_[np.ones((X.shape[0], 1)), X]
  y = np.concatenate((y_class_0, y_class_1))
  
  # Sigmoid function for logistic regression
  def sigmoid(z):
      return 1 / (1 + np.exp(-z))
  
  # Initialize parameters for logistic regression (bias and weights)
  theta = np.random.randn(X_b.shape[1], 1)
  
  # Define hyperparameters
  learning_rate = 0.2
  batch_size = 5
  n_iterations = 1000
  m = len(X)
  # Mini-batch gradient descent
  for iteration in range(n_iterations):
      shuffled_indices = np.random.permutation(m)
      X_shuffled = X_b[shuffled_indices]
      y_shuffled = y[shuffled_indices].reshape(-1, 1)
      
      for i in range(0, m, batch_size):
          xi = X_shuffled[i:i+batch_size]
          yi = y_shuffled[i:i+batch_size]
          print(f'i={i}, minibatch_size={np.size(yi)} class_1%={np.sum(yi)/np.size(yi)}')
          gradients = 1/batch_size * xi.T.dot(sigmoid(xi.dot(theta)) - yi)
          theta -= learning_rate * gradients
  
  # Function to predict classes
  def predict(X):
      return (sigmoid(X.dot(theta)) &gt;= 0.5).astype(int)
  
  # Plotting the decision boundary
  x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
  y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
  xx, yy = np.meshgrid(np.linspace(x_min, x_max, 1000), np.linspace(y_min, y_max, 1000))
  X_new = np.c_[np.ones((len(xx.ravel()), 1)), xx.ravel(), yy.ravel()]
  Z = predict(X_new).reshape(xx.shape)
  print(f'theta={theta}')
  
  plt.contourf(xx, yy, Z, alpha=0.3)
  plt.scatter(X_class_0[:, 0], X_class_0[:, 1], label='Class 0')
  plt.scatter(X_class_1[:, 0], X_class_1[:, 1], label='Class 1')
  plt.xlabel('Feature 1')
  plt.ylabel('Feature 2')
  plt.title('MBGD Logistic Regression on Imbalanced Data')
  plt.legend()
  plt.show()
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-18T22:36:35.867Z
Answers.LastActivityDate: 2023-11-18T22:36:35.867Z
Answers.UserDisplayName: cinch
Answers.Body: <p>The problem is, what mathematical definition do you give to &quot;balanced&quot; and &quot;representative&quot;?</p>
  <p>Balanced usually means that classes are uniformly distributed, but then you are screwing up your prior, but that's nothing that an importance sampling coefficient might not fix</p>
  <p>However, the representative part is ill defined. By itself, you want enough data such that:
  <span class="math-container">$$
  \nabla L^*(\theta)^T \nabla L(\theta) \ge 0
  $$</span>
  <span class="math-container">$L$</span> being the stochastic loss, and <span class="math-container">$L^*$</span> the actual &quot;full&quot; loss... in other words, you want that your stochastic gradient is pointing &quot;pretty much&quot; in the same direction as the full gradient.</p>
  <p>Now, how do we know that this is true? we don't... however, the more &quot;stochastic&quot; it is, the more likely is this to happen (thus increasing batch sizes usually fixes this)</p>
  <p>So, can a a batch of data be balanced and be not representative? Yes, you can pretty easily build an example where this is true</p>
  <hr />
  <p>Since it seems that the words were not enough, here you are an example</p>
  <p>Consider the following setting (dotted line is the decision boundary)</p>
  <p><a href="https://i.sstatic.net/FJK7p.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/FJK7p.png" alt="enter image description here" /></a></p>
  <p>If you sample the two RED outliers, the gradient from that minibatch will still drive the decision boundary towards the optimal one (hope we agree on this point), thus</p>
  <p><span class="math-container">$$
  \nabla L^\text{all}(\theta)^T \nabla L^\text{minibatch RED}(\theta) \ge 0
  $$</span></p>
  <p>However, now consider the following decision boundary</p>
  <p><a href="https://i.sstatic.net/XYam1.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/XYam1.png" alt="enter image description here" /></a></p>
  <p>Clearly, the loss induced by this decision boundary, is much lower by the previous one... however, now if you sample the RED minibatch, the gradient will say &quot;nope, totally disagree, you should completely flip the prediction since all O are predicted as X and all X are predicted as O&quot;, in other words:
  <span class="math-container">$$
  \nabla L^\text{all}(\theta)^T \nabla L^\text{minibatch RED}(\theta) \le 0
  $$</span></p>
  <p><strong>Note</strong>: in all cases the ratio between O/X in the whole dataset and in the minibatch is preserved (1:1 ratio), thus the minibatch is indeed <strong>balanced</strong>, yet it's <strong>not representative</strong></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-18T01:08:23.983Z
Answers.Text: very hard to make a connection to the asked question.
Answers.UserDisplayName: DSPinfinity
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-18T01:11:14.633Z
Answers.Text: @DSPinfinity what's not clear?
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-18T01:15:52.107Z
Answers.Text: @DSPinfinity btw, your definition of balanced is not correct
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-18T01:21:44.367Z
Answers.Text: Could you please correct it? I will be happy if you can give a simple definition for representativeness.
Answers.UserDisplayName: DSPinfinity
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-18T01:29:39.670Z
Answers.Text: @DSPinfinity I already gave the usual definition for being balanced and for being representative in the answer, being uniformly distributed and to have correlated gradients it's pretty clear to me
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-18T10:45:58.413Z
Answers.Text: @DSPinfinity From a statistical perspective, a sample (mini-batch) of a population (dataset) is representative if its statistics are pretty much the same of the population. Let's say you have a dataset $X$ with 1 feature, with mean $\mu_X=1$. Now you're mini-batches should have approximately the same mean value, therefore not something very different like $2$ or $-1$, for example. Anyway, I think that the important part is that the samples are representative if they follow the same distribution of the dataset. So balancing classes makes the "class" distribution uniform, without modes (peaks).
Answers.UserDisplayName: Luca Anzalone
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-18T13:20:45.397Z
Answers.Text: @LucaAnzalone, perfect answer. Could you please give an example like this where the data is balanced (for a classification problem) but not representative?
Answers.UserDisplayName: DSPinfinity
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-19T00:34:11.543Z
Answers.Text: @LucaAnzalone No,  consider logistic regression, take 2 linearly separable classes, and add one "outlier" per class in the opposite cluster. Now, if your minibatch is composed by only these 2 outlier points, your batch is still balanced, but it's not representative, as the direction of the update will be almost opposite to the true gradient (and (1) your minibatch can be seen as a single sample MC estimate of your parameters (2) these two points might definitely come from the exact distribution of your data).
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-19T00:36:45.910Z
Answers.Text: @LucaAnzalone To conclude, it makes little sense to define "representativeness" as closeness in the distribution statistics space, because then you should define a measure of closeness, and some threshold    over which is considered "not representative", where instead the definition with the gradient has no "subjectivity" innit, and as said earlier, you can most likely have a non representative sample, and it gets more likely the closer you get to your local minima
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-19T00:46:23.467Z
Answers.Text: @LucaAnzalone added some illustrations on the answer, hope it helps
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-19T00:46:30.470Z
Answers.Text: @DSPinfinity added some illustrations on the answer, hope it helps
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-18T00:47:38.183Z
Answers.LastActivityDate: 2023-11-19T00:46:02.853Z
Answers.LastEditDate: 2023-11-19T00:46:02.853Z
Answers.UserDisplayName: Alberto
Body: <p>When we construct mini-batches for stochastic gradient, it is important to ensure that the different mini-batches are balanced (for example, in case of classification they contain the same ratio for each class) and representative for the whole dataset.</p>
  <p>Can mini-batches be balanced but not representative? Any example?</p>
  
Tags: <machine-learning><deep-learning>
Title: Can mini-batches for stochastic gradient be balanced but not representative of the training data?
UserDisplayName: DSPinfinity
ThreadText: # Can mini-batches for stochastic gradient be balanced but not representative of the training data?
  
  (Asked by: username_0 on 2023-11-17)
  
  When we construct mini-batches for stochastic gradient, it is important to ensure that the different mini-batches are balanced (for example, in case of classification they contain the same ratio for each class) and representative for the whole dataset.
  
  Can mini-batches be balanced but not representative? Any example?
  
  **Tags:** machine-learning, deep-learning
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2023-11-18. Score: 3 | Accepted answer
  
  The problem is, what mathematical definition do you give to "balanced" and "representative"?
  
  Balanced usually means that classes are uniformly distributed, but then you are screwing up your prior, but that's nothing that an importance sampling coefficient might not fix
  
  However, the representative part is ill defined. By itself, you want enough data such that:
  $$
  \nabla L^\*(\theta)^T \nabla L(\theta) \ge 0
  $$
  $L$ being the stochastic loss, and $L^\*$ the actual "full" loss... in other words, you want that your stochastic gradient is pointing "pretty much" in the same direction as the full gradient.
  
  Now, how do we know that this is true? we don't... however, the more "stochastic" it is, the more likely is this to happen (thus increasing batch sizes usually fixes this)
  
  So, can a a batch of data be balanced and be not representative? Yes, you can pretty easily build an example where this is true
  
  ---
  
  Since it seems that the words were not enough, here you are an example
  
  Consider the following setting (dotted line is the decision boundary)
  
  [![enter image description here](https://i.sstatic.net/FJK7p.png)](https://i.sstatic.net/FJK7p.png)
  
  If you sample the two RED outliers, the gradient from that minibatch will still drive the decision boundary towards the optimal one (hope we agree on this point), thus
  
  $$
  \nabla L^\text{all}(\theta)^T \nabla L^\text{minibatch RED}(\theta) \ge 0
  $$
  
  However, now consider the following decision boundary
  
  [![enter image description here](https://i.sstatic.net/XYam1.png)](https://i.sstatic.net/XYam1.png)
  
  Clearly, the loss induced by this decision boundary, is much lower by the previous one... however, now if you sample the RED minibatch, the gradient will say "nope, totally disagree, you should completely flip the prediction since all O are predicted as X and all X are predicted as O", in other words:
  $$
  \nabla L^\text{all}(\theta)^T \nabla L^\text{minibatch RED}(\theta) \le 0
  $$
  
  **Note**: in all cases the ratio between O/X in the whole dataset and in the minibatch is preserved (1:1 ratio), thus the minibatch is indeed **balanced**, yet it's **not representative**
  
  > **Comments:**
  > **username_0**: very hard to make a connection to the asked question.  
  > **username_2**: @username_0 what's not clear?  
  > **username_2**: @username_0 btw, your definition of balanced is not correct  
  > **username_0**: Could you please correct it? I will be happy if you can give a simple definition for representativeness.  
  > **username_2**: @username_0 I already gave the usual definition for being balanced and for being representative in the answer, being uniformly distributed and to have correlated gradients it's pretty clear to me  
  > **username_3**: @username_0 From a statistical perspective, a sample (mini-batch) of a population (dataset) is representative if its statistics are pretty much the same of the population. Let's say you have a dataset $X$ with 1 feature, with mean $\mu_X=1$. Now you're mini-batches should have approximately the same mean value, therefore not something very different like $2$ or $-1$, for example. Anyway, I think that the important part is that the samples are representative if they follow the same distribution of the dataset. So balancing classes makes the "class" distribution uniform, without modes (peaks).  
  > **username_0**: @username_3, perfect answer. Could you please give an example like this where the data is balanced (for a classification problem) but not representative?  
  > **username_2**: @username_3 No,  consider logistic regression, take 2 linearly separable classes, and add one "outlier" per class in the opposite cluster. Now, if your minibatch is composed by only these 2 outlier points, your batch is still balanced, but it's not representative, as the direction of the update will be almost opposite to the true gradient (and (1) your minibatch can be seen as a single sample MC estimate of your parameters (2) these two points might definitely come from the exact distribution of your data).  
  > **username_2**: @username_3 To conclude, it makes little sense to define "representativeness" as closeness in the distribution statistics space, because then you should define a measure of closeness, and some threshold    over which is considered "not representative", where instead the definition with the gradient has no "subjectivity" innit, and as said earlier, you can most likely have a non representative sample, and it gets more likely the closer you get to your local minima  
  > **username_2**: @username_3 added some illustrations on the answer, hope it helps  
  > **username_2**: @username_0 added some illustrations on the answer, hope it helps  
  
  ---
  
  ### Answer by username_1 on 2023-11-18. Score: 2
  
  Though you've already accepted an answer, from your comments it seems you're still somewhat unclear and need an example. Take your example of (binary) classification, usually balanced training data means you have more or less equal number of samples for each class label. Any due to the intrinsic random nature of each mini-batch during gradient descent (MBGD), it's very likely some mini-batches data will not be representative in the sense that it may become imbalanced. But so long as MBGD converges, it's generally converge faster and more computationally efficient leveraging parallelism than full batch GD, the impact of possibly many imbalanced mini-batches is typically minimized as the model updates its parameters based on the entire dataset and over multiple epochs.
  
  Here's a simple binary logistic classification example for a balanced dataset with two class labels of equal number of samples (50 for each) and two features, code is provided at then end. Total dataset has 100 samples, mini-batch size is set to 5, and epoch is tuned to 1000. As you can see from the output many mini-batches are not representative for the whole dataset (many have 0%, 20%, 60%, 80% class\_1 data), but the final result is still good and close to the full batch result as you can easily verify.
  
  [![enter image description here](https://i.sstatic.net/7jmbg.png)](https://i.sstatic.net/7jmbg.png)
  
  ```
  output result:
  i=0, minibatch_size=5 class_1%=0.6
  i=5, minibatch_size=5 class_1%=0.4
  i=10, minibatch_size=5 class_1%=0.4
  i=15, minibatch_size=5 class_1%=0.8
  i=20, minibatch_size=5 class_1%=1.0
  i=25, minibatch_size=5 class_1%=0.6
  i=30, minibatch_size=5 class_1%=0.0
  i=35, minibatch_size=5 class_1%=0.4
  i=40, minibatch_size=5 class_1%=0.8
  i=45, minibatch_size=5 class_1%=0.2
  i=50, minibatch_size=5 class_1%=0.4
  i=55, minibatch_size=5 class_1%=0.6
  i=60, minibatch_size=5 class_1%=0.6
  i=65, minibatch_size=5 class_1%=0.8
  i=70, minibatch_size=5 class_1%=0.8
  i=75, minibatch_size=5 class_1%=0.6
  i=80, minibatch_size=5 class_1%=0.4
  i=85, minibatch_size=5 class_1%=0.2
  i=90, minibatch_size=5 class_1%=0.2
  i=95, minibatch_size=5 class_1%=0.2
  i=0, minibatch_size=5 class_1%=0.6
  i=5, minibatch_size=5 class_1%=0.2
  i=10, minibatch_size=5 class_1%=0.2
  i=15, minibatch_size=5 class_1%=0.8
  i=20, minibatch_size=5 class_1%=0.4
  ...
  i=95, minibatch_size=5 class_1%=0.2
  ----
  import numpy as np
  import matplotlib.pyplot as plt
  
  np.random.seed(42)
  
  # Class 0: Generate points in a cluster around (-1, -1)
  num_class_0 = 50
  X_class_0 = np.random.randn(num_class_0, 2) - 1 * np.ones((num_class_0, 2))
  y_class_0 = np.zeros((num_class_0,))
  
  # Class 1: Generate points in a cluster around (1, 1)
  num_class_1 = 50 
  X_class_1 = np.random.randn(num_class_1, 2) + 1 * np.ones((num_class_1, 2))
  y_class_1 = np.ones((num_class_1,))
  
  X = np.concatenate((X_class_0, X_class_1), axis=0)
  X_b = np.c_[np.ones((X.shape[0], 1)), X]
  y = np.concatenate((y_class_0, y_class_1))
  
  # Sigmoid function for logistic regression
  def sigmoid(z):
      return 1 / (1 + np.exp(-z))
  
  # Initialize parameters for logistic regression (bias and weights)
  theta = np.random.randn(X_b.shape[1], 1)
  
  # Define hyperparameters
  learning_rate = 0.2
  batch_size = 5
  n_iterations = 1000
  m = len(X)
  # Mini-batch gradient descent
  for iteration in range(n_iterations):
      shuffled_indices = np.random.permutation(m)
      X_shuffled = X_b[shuffled_indices]
      y_shuffled = y[shuffled_indices].reshape(-1, 1)
      
      for i in range(0, m, batch_size):
          xi = X_shuffled[i:i+batch_size]
          yi = y_shuffled[i:i+batch_size]
          print(f'i={i}, minibatch_size={np.size(yi)} class_1%={np.sum(yi)/np.size(yi)}')
          gradients = 1/batch_size * xi.T.dot(sigmoid(xi.dot(theta)) - yi)
          theta -= learning_rate * gradients
  
  # Function to predict classes
  def predict(X):
      return (sigmoid(X.dot(theta)) >= 0.5).astype(int)
  
  # Plotting the decision boundary
  x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
  y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
  xx, yy = np.meshgrid(np.linspace(x_min, x_max, 1000), np.linspace(y_min, y_max, 1000))
  X_new = np.c_[np.ones((len(xx.ravel()), 1)), xx.ravel(), yy.ravel()]
  Z = predict(X_new).reshape(xx.shape)
  print(f'theta={theta}')
  
  plt.contourf(xx, yy, Z, alpha=0.3)
  plt.scatter(X_class_0[:, 0], X_class_0[:, 1], label='Class 0')
  plt.scatter(X_class_1[:, 0], X_class_1[:, 1], label='Class 1')
  plt.xlabel('Feature 1')
  plt.ylabel('Feature 2')
  plt.title('MBGD Logistic Regression on Imbalanced Data')
  plt.legend()
  plt.show()
  
  ```
  
  
  ---
  
  