4

I heard that one of the main problems applying neural style to high resolution images is the huge amount of memory that would use.

Also I just configured a network using tiny-cnn

This is my calculation for number of weights and number of neurons per layer in my example:

conv_layer_# height width depth filter_height filter_width neurons_(h*w*d) weights
1            512    512   3     3             3            786432          108    
2            256    256   6     3             3            393216          216    
3            128    128   12    3             3            196608          432    
4            64     64    24    3             3            98304           864    
5            32     32    48    3             3            49152           1728   
6            16     16    96    3             3            24576           3456   
7            8      8     192   3             3            12288           6912   

                                                           1560576         13716  

If we give every value that has to be stored 8 bytes (single precision floating point format), we end up with 6297168 bytes, i.e. about 12 MB. But the net allocates over 1GB of RAM when I train it. What is all that memory needed for?

14
  • 2
    Aren't there a hundred millionish synapses in your configuration? Jul 11, 2016 at 7:41
  • Are you aware that a neural network needs to store the state of every single neuron and every single synapse somewhere?
    – Philipp
    Jul 11, 2016 at 7:58
  • @Philipp: Yes, I am aware of this. I added my calculation of number of neurons and weights to my question. Jul 11, 2016 at 13:15
  • @TobiasHermann 4 bytes? Really? With 4 bytes you can't even find the neuron you're looking for on a 64-bit OS, let alone store data about it.
    – Ordous
    Jul 11, 2016 at 13:23
  • @Ordous: Why should I need pointers to single neurons? Or why should a neuron need more than one single floating point value? Jul 11, 2016 at 13:42

2 Answers 2

2

I found out some of the factors that may contribute to the effect.

1) At least in tiny-cnn, some of the buffers are allocated not once but once per worker thread. On a machine with 8 CPU threads, this can increase the memory usage a lot. In debug mode using MS VC++ 2015 the following two lines in the code base allocate a big chunk, both related to worker threads: ith_in_node(i)->set_worker_size(worker_count); and ith_out_node(i)->set_worker_size(worker_count);.

2) Additionally to the values for the neurons and weights listed in my question, also gradients and some other stuff for the backward passes and the optimization have to be stored.

3) Not sure if this is relevant fot tiny-cnn, but many frameworks seem to use an operation called im2col. This makes the convolution much faster, by expressing it as a matrix multiplication. But in case of filters with 3*3 in height and width, this scales the number of values from the input to a convolution up by a factor of 9. Justin Johnson explains it in the lecture "S231n Winter 2016 Lecture 11 ConvNets in practice" starting at 36:22.

4) There was an error in my initial calculation. When a volume of 512*512*3 is convolved with 6 3*3 filters and then send into an average pooling layer, the result volume is 256*256*6, but in between it is 512*512*6, also contributing with a factor of 2.

5) There was another error in my initial calculation. I demonstrate it on the last conv layer (7). It takes a volume of 16*16*96 to a volume of 8*8*192 with filters of size 3*3. This means every filter has 3*3*96 weights, and there are 192 of them, resulting in 165888 (3*3*96*192) weights overall for this layer, not 6912.

So numerologically multiplying only the first three factors (8, 9 and 2) we end up with a factor of 144, which seems enough to explain the high memory consumption.

1

Your calculation for the amount of memory used appears to be related to the number of neurons in the network and storing a double for each, but that isn't the only storage that is required -- each neuron will also contain a number of weights, each of which is likely to need at least a float. This is the last column in your output, and (at least if I understand the program you're using correctly) is the number of weights stored per neuron. Assuming each weight is a float, this adds up to a little over 2GB.

1
  • Thanks for thinking this through. But the way I understand CNNs the number of weights is not per neuron but per layer. In a convolutional layer the weights are shared, which is different from a fully connected layer. see cs231n.github.io/convolutional-networks Jul 12, 2016 at 10:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.