title: How to implement mini-batch gradient descent in python?
body: <p>I have just started to learn deep learning. I found myself stuck when it came to gradient descent. I know how to implement batch gradient descent. I know how it works as well how mini-batch and stochastic gradient descent works in theory. But really can't understand how to implement in code.</p>
  
  <pre><code>import numpy as np
  X = np.array([ [0,0,1],[0,1,1],[1,0,1],[1,1,1] ])
  y = np.array([[0,1,1,0]]).T
  alpha,hidden_dim = (0.5,4)
  synapse_0 = 2*np.random.random((3,hidden_dim)) - 1
  synapse_1 = 2*np.random.random((hidden_dim,1)) - 1
  for j in xrange(60000):
      layer_1 = 1/(1+np.exp(-(np.dot(X,synapse_0))))
      layer_2 = 1/(1+np.exp(-(np.dot(layer_1,synapse_1))))
      layer_2_delta = (layer_2 - y)*(layer_2*(1-layer_2))
      layer_1_delta = layer_2_delta.dot(synapse_1.T) * (layer_1 * (1-layer_1))
      synapse_1 -= (alpha * layer_1.T.dot(layer_2_delta))
      synapse_0 -= (alpha * X.T.dot(layer_1_delta))
  </code></pre>
  
  <p>This is the sample code from ANDREW TRASK's blog. It's small and easy to understand. This code implements batch gradient descent but I would like to implement mini-batch and stochastic gradient descent in this sample. How could I do this? What I have to add/modify in this code in order to implement mini-batch and stochastic gradient descent respectively? Your help will help me a lot. Thanks in advance.( I know this sample code has few examples, whereas I need large dataset to split into mini-batches. But I would like to know how can I implement it)</p>