I built a neural net, and planned on optimizing the weights using a genetic algorithm. I was informed though, that this isn't a good idea, and to look into backpropagation.

I searched around, and found either very superficial overviews of the algorithm (which conceptually isn't that difficult), or very in-depth guides of the math. All the math I've seen appears to be calculus (although I wouldn't know, as I don't yet know calculus), so it's difficult for me to see what's going on.

I looked up the concept in Artificial Intelligence: A Modern Approach, but as with everything else in the book, the math looks arcane.

I understand the process is basically:

  1. Propagate the input towards the output.
  2. Once it reaches the output, compare the result to what you expected it to be.
  3. The error (expected - actual) is calculated.
  4. Starting at the output, the error is propagated back through the network, and the weights are adjusted accordingly.

My issue is point 4. How can you know a hidden node's effect on the output?

Can anybody explain backpropagation of a neural net, with math that can be understood at a sub-calculus level?


"Hidden" nodes aren't really hidden like a black box - it is just a layer in between your input and output nodes. You program will have the values of all weights and the signal values propagated through the neural network.

Once you have the output error, you can use all that information with the "archaic" (do you mean arcane?) mathematics.

I found this PDF through wikipedia which has more accessible examples: https://www4.rgu.ac.uk/files/chapter3%20-%20bp.pdf

So, the hidden layer node errors are calculated from the output error multiplied by the weight of the connection between the hidden node and output node - and then the weights from input to hidden node can be modified from this error. In the simple case of a single output node:

δα = outα (1 - outα) (Targetα - outα) 
output error = actual output * (1 - actual output) * (output we want - actual output)

δA = outA (1 – outA) (δαWAα)
hidden node error = output of node * (1 - output of node) * (output error * weight between hidden and output node)

Programmatically, you want to calculate all the node error values before adjusting any weights.

  • Thanks. In your last bit with the code, what are the weird looking 'a's and the ' o's with the tails coming out of the top? I haven't learned about them yet. – Carcigenicate Jul 16 '15 at 12:27
  • well - the α is just a label - it's the greek letter alpha - and its only use here is to indicate the output neuron - as for the δ - this is the greek letter delta, and it's used in maths to indicate a change in value - so, δα is the error in neuron α – HorusKol Jul 16 '15 at 22:35
  • I'm sorry, I completely forgot about this answer until this morning. I ended up using the pdf you linked to (which explains it very nicely) and have a working neural net that uses backpropagation! Again, sorry I neglected your answer. – Carcigenicate Sep 11 '15 at 11:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.