I have a python module
dataProcessor.py which initialises a large amount of data into memory (approximately 3GB) I want to use this module in different processes which are running simultaneously.
But the problem is there is not enough Memory on machine to run everything at same time due to
dataProcessor.py loading data into memory for every process (3GB for each process, so for 3 processes a total of 9GB Memory).
I tried using server-client model to initialise data only once and and serve all processes but this model is too slow. Is there any method to load data only once and have other processes access the methods in module
The module I am talking about is Spacy which is written in Cython. The data can be any Python object, and won't change once written. It is OK if the solution is a C extensions to Python.
Is there any alternative to server-client or subprocess model which shares memory.