Stack Exchange
Delta is growing instead of going down in simple one neuron neural network
Trying to implement simple neural network using Delta rule without step function having almost no experience in programming and machine learning. So far I have next code.
%matplotlib inline
from random import choice 
from numpy import array, dot, random
import matplotlib.pyplot as plt
training_data = [(array([20,10,1]), 10), 
             (array([10,12,1]), 8), 
             (array([11,20,1]), 5), 
             (array([13,16,1]), 9), ]
w = random.rand(3)
a = []
mu = 0.001
n = 50
for i in xrange(n): 
    x, expected = choice(training_data)
    result = dot(w,x)
    delta = abs(expected - result)
    a.append(delta)
    if delta < a[-1]:
        w += mu * delta * x
    else: w -= mu * delta * x
plt.plot(a)

The problem with this code is that the delta rule is growing up instead of going down.You may see that from this graph. I am not sure where I made a mistake with it. Any help is appreciated.
EDITED VERSION OF MY FOR LOOP
for i in xrange(n): 
x, expected = choice(training_data)
result = dot(w,x)
delta = abs(expected - result)
if delta < a[-1]:
    w += mu * delta * x
else: w -= mu * delta * x
a.append(delta)

also, I have stated a=[0]. new results

If you append delta to the end of a, then compare delta to the last element of a (a[-1]), you are comparing delta < delta, and this branch will never be visited.

thanx, dumb me :) I am now thinking how to compare those starting from the second iteration?

I have moved  a.append(delta) to the bottom and changed a = [] to a = [0], but stil, the effect is the same.

This results suggests that one of your conditions (< or > a[-1]) is always satisfied : expected and x are constant, so to have a big delta, result has to grow, so w has to continue growing. Think about why a new delta is always bigger than the previous one. print() helps a lot when debugging code ! Also, try reversing the inequality, just to see.
