backpropagation DQN double q learning

After many research, I still can't find a neat answer about this question:

Let's assume 'lo' is our loss for a state-action pair calculated with the Bellman eq. I don't understand wich one here is correct:

1. Should I backprop the same loss for every output Q(s,a) in my network?

2. Should I ONLY backpropagate the loss for the specific output neuron I chose an action from?(not backprop the rest of the output neurons. Meaning that if we choose action 3 in for example 10 possible actions, we only backprop from the output neuron 3).

3. Should I calculate for every Q(sn,an) it's Q*(sn,an) and each time backpropagate the loss of these 2? This is not correct as far as I understood.

Thnx for helping me out!

• This question would probably be better asked at datascience.stackexchange.com. Jun 17, 2020 at 3:18
• Your question has also received a number of downvotes so I would also suggest retracting the question. When you ask it again in the datascience part of the site, you should provide more background to the question so that the answers you receive will be better. Right now, there is too little context for someone to provide you a useful answer. Jun 17, 2020 at 16:05

1 Answer

I think you're over-thinking this. Your Q network is simply a function approximator that you're using for regression. Upon a transition (s, a, r, s'), your input is Q(s, a) and your label is (r + gamma * max Q'(s', a')), where the max is over the actions a' and Q' is your target Q network. You calculate your loss between the input and the label, and simply backpropagate. I'm assuming you're using some autograd library, so you don't really need to worry much more about it. However, if you want to know what your gradients look like, remember your input Q(s, a) depends only on one of the output neurons (the one corresponding to a). So your gradients only flow through paths that pass through that neuron.

• Thnx a lot, makes everything more clear. I'm actually implementing my whole neural network from scratch! ;) Pain in the ass I must say. I'm using inputs from 0-1 and using ReLu as activation function for my hidden layers. Keeping my output layer without activation function (raw numbers to calc the loss). Is relu a good choice? and does it makes sense to have input values between 0-1? Jun 17, 2020 at 21:24
• Ah. A PITA indeed, but something every ML enthusiast must do once. And yes, your choices make sense to me. Jun 17, 2020 at 23:00