2

The values of all of the neurons in my neural network are initialized to 0 and the connection strengths between the neurons are set to randomly generated floats, between 0 and 1. I have seen other people setting their ranges from 0 to .5, however. Are there any benefits of one over the other? and Should the neuron values be randomly set to either 1 or 0, instead?

2

Randomizing the nodes and connections, increases the initial entropy of the network.

Creating stronger biases in the network at the beginning (before training), would presumably facilitate the formation of some of the neural paths faster.

When all other factors are equal for an input, this will amortize worst cases during training (this should happen rarely, but in a network training session it may happen a few times).

In practice the difference in efficience is probably infinitesimal throughout the training period (if it matters at all) - so in the end, it is a matter of prefference.

Similarly, if you implement qsort and choose the pivot position randomly, (instead of picking the middle of each sorted segment) the sorting tends to have on average, a faster execution time (observable usually only over many repetitions).

1
  • So is it safe to say that the neurons are like flip/flops and their values determined by whether the weights on the connections are + or -? I guess what I am trying to figure out is that if the state of the neurons determines what gets fed forward, but they are determined by the weight values, then why not forget mutating neuron states altogether and simply return neuron behavior in terms of a function that takes weights as inputs? – kurofune Aug 11 '15 at 22:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.