2

TL; DR:

  • Is the output from a feed-forward neural a direct result of the activation function?

  • I.e: If the activation function is the sigmoid function, will the output always be between 0 and 1?

I'm having difficulty understanding the activation function, and the output of a feed-forward neural net.

From what I understand, the propagation of a signal through the net follows this pattern:

  • The input values are fed to the nodes of the first hidden layer.

  • Each node of the layer multiplies each input by the associated weight, and gets a sum of all the weighted inputs.

  • The activation function (I'm using a sigmoid function) is applied to the sum, and the result becomes the activation for the node.

  • The first hidden layer becomes the new input, and the above repeats for it and the second hidden layer.

  • This continues until the output layer has received from the last hidden layer.

Question 1: Is my above understanding of the process correct?

My misunderstanding is about the resulting activations of the output layer. Since the sigmoid activation function was applied to the output, the output will always be within the range of 0 to 1 (assuming a sigmoid function was used).

Question 2: Is this correct? Should the output values be a direct result of the sigmoid function?

I'm asking because this throws a wrench in my simple first test of my net. I was using this to train my net (it should be self explanatory enough to not require more code examples/context). The fitness function is inside:

    public List<Double> trainNet(List<Double> input, List<Double> expectedOutput, GeneticAlgorithm<Double> ga, int gens) {

            ga.setFitnessFunction( genes -> {
                assert output.sizeWithoutBias() == expectedOutput.size();
                setWeights( genes.getSequence() );

                //"Fire" net
                List<Double> output = update();

                int fitness = Integer.MAX_VALUE;
                for (int i = 0; i < output.size(); i++) {
                    fitness -= 100 * Math.abs( output.get(i) - expectedOutput.get(i) );
                }

                return fitness;
            } );

            resetInput(input);

            ga.simNGens(gens);

            return output.getActivationsWithoutBias();
        }

Basically, I'm just subtracting difference between the output and and expected output from the fitness. This "worked" (seemingly) until I realized that I forgot about the sigmoid function. So I added it (trimmed excerpt):

public class Node {

    private List<Double> weights;
    private double activation = 0;

    public void addWeightedInputs(List<Double> inputs) {
        if (inputs.size() != weights.size()) throw new IllegalArgumentException(
            "Input size (" + inputs.size() +
                ") doesn't match weight's size (" + weights.size() + ").\nThis: " + this );

        double sum = 0;

        for (int i = 0; i < inputs.size(); i++) {
            sum += weights.get(i) * inputs.get(i);
        }

        activation = sigmoidFunction(sum);
        System.out.println("Act: " + activation + ", Sum: " + sum);
    }

    private static double sigmoidFunction(double activation) {
        return 1.0 / (1 + Math.pow(Math.E, -activation));
    }
}

The problem is, now, the expected output is in a "raw" form ([1,2,3]), and the output from the net is in "sigmoid form" ([0.9993,0.8764,0.9966]). Obviously now the fitness function is broken, since the output and the expected out are very different.

Question 3: Which is correct: the fitness function's expectation, or the output from the net? If the "sigmoid output" is correct, how do I use it in that form?

2

Question 1: Is my above understanding of the process correct?

Yes; for the feed-forward process. It should be noted that the process can be computed with a matrix multiply for each layer.

Question 2: Is this correct? Should the output values be a direct result of the sigmoid function?

Yes, you generally apply the activation function to output layer. It should be obvious that the range of your output will match the range of your activation function. In some cases you may need to map the output for it to be useful.

N.b. the activation function doesn't need to the same for every layer, you could configure your network however you want/need. For example NNs for image recognition usually start with alternating layers of convolution and pooling layers, before they get into any fully connected layers. You just need to apply the correct derivative to the correct layer when back propagating.

Question 3: Which is correct: the fitness function's expectation, or the output from the net? If the "sigmoid output" is correct, how do I use it in that form?

This question is difficult to answer, it appears somewhat malformed, i.e. the second sentence doesn't refer to anything in the first. The short answer is sigmoid and map; the long answer is:

A neural network (NN) is a universal function approximator. The goal of training the network is to get its output to match the output of the function (F()) you are approximating. The output is supposed to be getting closer and closer to the expected value as you train, so it gets more and more correct as you train. It won't be correct until your training is complete and the weights have converged.

The loss function (a fitness function is a type of loss function) tells you how far off the output is from the expected value. The loss function also isn't 'correct' since it is the error signal for your network, not the output. When your loss is near zero, it indicates that the output is very close to correct.

A very common network output is a one hot vector where all the values are 0, except for one which is 1. This is useful for classification, where each element (in the output vector) represents the probability the input is of a given class.

The one hot output lines up very will with the range of a sigmoid, all values are between 0 and 1 (ideally exactly zero or one).

If your function has a different range you will have to map it, and use the mapped value to compare with the output. Or use a different activation function, like rectified linear. Obviously if the desired range doesn't correspond to the range of the activation function you will have to map, or swap your activation function.

Also, I don't know where you got your values for sigmoid([1,2,3]) the correct values would be [0.731059, 0.880797, 0.952574] not [0.9993,0.8764,0.9966] as you have.


I noticed that you are using a genetic algorithm for optimization instead of back-propagation. Due to the high dimensionality of NNs a genetic weight optimization is almost certainly going to be under-sampling your error space. This will lead to a slowly converging network. You also lose out on many of the advancements in back-propagation, like momentum. Some enhancements like drop-out should still be able to be applied, however.

Much of the power of NNs comes from the back-propagation scheme, and most NNs are trained with gradient descent via back-propagation. Genetic optimization sort of misses the point of NNs (they are not quite orthogonal, however.)

Genetic algorithms are often used to optimize the architecture of a NN, i.e. how many layers and of what size.

Genetic optimization does get around the vanishing gradient problem, which generally limits sigmoid networks to 2 hidden layers at maximum.

Genetic optimization is, however, preferable when you have a plethora of local minima in your error space that gradient descent is likely to get stuck in and/or have trouble moving past. In this case the genetic optimization will probably preform better, since it doesn't 'care' about the shape of your error space.


N.b. writing your own neural network is generally only recommended as a learning exercise, if you intend to do actual work you should use an existing library, as it will already be debugged for you; debugging NNs is notoriously difficult. Bugs tend to be in two categories, ones that just completely break the network, and ones that just hinder the network, making it slower (to converge); if the bug doesn't break the network, the network still learns, just slower, or not as well as it should. The second class of bugs can be very difficult to find!


Your initial tests of the network should be something that is easy to verify without a network, ex. linear regression:

  • can be done without a NN
  • only takes 3 nodes(2 in, 1 out) to do with a NN
  • can verify the answer without a network.

Also a small network is easy to calculate an update by hand to verify the math is correct as well; if you write a NN from scratch you will probably need verify a single pass (forward and back-prop) by hand at some point during debugging!

  • Awesome, thank you. I was avoiding backpropagation because the math looked really difficult, but I guess I just need to suck it up. – Carcigenicate Jul 15 '15 at 13:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.