Hi Adrian,



 Adrian.1:

Does this mean swapping

Q[s, a] += alpha * ( reward + (gamma * Q[s_, a_] ) - Q[s, a] )

with a neural network where the network inputs are state (s) and the output values are a (action) ?


I think it’s a typo but you are missing a max for Q[s_, a_] values, since you need to find state-action pair with the maximum value for all actions.
The neural network works as a function approximator here, so instead of looking up a table you can use the network to find Q values for all actions in that state. When you predict those values (by inputing state to network), i.e:
a_vector_of_Q_values_for_all_actions_in_s__ = model(s_)
you can find the max of them to find Y = reward + gamma * maxQ(s_, a_)
Now if X = model(s) then backpropagating loss(X[a],Y) constitutes the learning part of the algorithm which replaces the Q-value update in tabular Q-learning.
I also found this post very useful to understand the transition from tabular Q-learning to DQN: https://towardsdatascience.com/why-going-from-implementing-q-learning-to-deep-q-learning-can-be-difficult-36e7ea1648af 21