Can I train further a trained neural network only with some new character?

Question

I would like to make a semi-automatic OCR software for offline handwritten documents, where the OCR tries to recognize the words and the user has the ability to correct the fails of the recognizer by defining the misrecognized characters.

I found a similar issue but doesn't really answered my question: https://stackoverflow.com/questions/13117761/encog-neuroph-save-neural-network

My first approach is the following: I train a neural net before everything and run the first recognition on the selected document, then when the user define some specific character (for example the user selects one character on the image and says it's an "a" letter) it just modifies the weights in the neural net(runtime) and restarts the recognition on the document. Is it possible to "quick-train" the whole net only with some new spec characters so that the original characters could be still recognizable?

My second approach: I create two independent neural nets. The first is the universal as above and the second is an empty. After "universal" net finished the recognition I just add the new specified characters to train the second empty neural net and then restart the recognition with only this NN.

Answer 1

It's possible so long as you've done nothing to entrench the training.

A neural network starts out weighed randomly. As you train it the weights gradually take on values that produce your desired output (through a feedback loop). Once you're satisfied you stop training.

Now you want to add new things for the net to recognize. Fine. Either start randomly again or start where you are now. It's hard to tell which one will get you there sooner.

Also long as you've not pruned down the number of neurons to the minimum needed it should be possible to train it with a new character.

However, don't think that your old characters can simply be removed from the training set. With enough training they will be forgotten.

Answer 2

Though training and recognition are quite different, it is theoretically possible to continue training after the initial, even on the end user device. However, NN training works by consuming huge data sets. It seems suspicious that the end user will be able to supply a sufficiently large data set for the training adjustment such that the NN becomes fully updated to that user. Compare, for example, the original training set size to what the end user might do over the course of a few months, probably a difference in data set size on the scale of more than a few orders of magnitude.

Answer 3

There are techniques, such as stochastic gradient descent, which perform online training (i.e. you train the network one example at a time, and you can always add more). The problem is that neural networks are fundamentally statistical beasts, which means that if the neural network thinks that X is A, while it might be possible to retrain it to classify X as B, it might require a bit of convincing (multiple examples of X being B) - this also depends on how well represented is X in the training set or how confident is the network that X is A.

You can make it more flexible to new examples by twitching the hyperparameters.

Provided that the user makes enough corrections, it will eventually improve accuracy.

Long time I would consider ways of improving the original network. One suggestion would be to store user's corrections, collect them as anonymous data and add them to your training set.

I don't think this is useful in your case, I going to mention knowledge transfer [1], since it might be worth investigating. It has been observed that the first layers of a network are quite identical (for example, in image recognition, the first layer is almost always a Gabor filter or color blob). Knowledge transfer implies taking the first n layers of a network, randomly initializing the last m layers and retraing the whole network. This might require a smaller data set and less computational power, so you might be able to do it on user's machines.

[1] - http://www.evolvingai.org/files/2014-YosinskiEtAl-HowTransferableAreFeaturesInDNNs-NIPS.pdf