3 grammatical error, file extension marked as code, multiplication symbol
source | link

I am working with MATLAB on a model reduction algorithm. It is basically a data processing pipeline.

ckt = generate_ckt(ckt_properties);
freq = generate_fpoints(fconfig);
result = freq_dom_sim(ckt,freq);
red_ckt = run_PRIMA(ckt, red_order);

Each of these are potentially time consuming activities, being that the data I work with is pretty big (10000 x× 10000 matrices). So in a previous implementation I had all of these as separate scripts that I had to manually execute one by one (ormanually or run a master script) all of them. Each of these stored the data in .mat.mat files. The next program would read from this and write its own result in another directory. And so on.

What I would like to use is a framework that can store the dependencies between various pieces of data, such that at any point of time I can just ask it to generate the output.

It should :

  1. Check if the variable is present in the workspace.
  2. If it is, check if its consistent with the expected properties (check with the config data)
  3. If not, load from file (the exact path to the file will be pre-specified).
  4. Check if its consistent with the expected properties.
  5. If not, compute it from the command associated with it. (pre-specified)

I would like this to be recursive, so that effectively I run the last module and it automatically runs checks and actually computes only those pieces of data that are not already available and consistent.

Can you give some suggestions on how to design this? If it is already called something (I assume it must) please point me to it.

I am working with MATLAB on a model reduction algorithm. It is basically a data processing pipeline.

ckt = generate_ckt(ckt_properties);
freq = generate_fpoints(fconfig);
result = freq_dom_sim(ckt,freq);
red_ckt = run_PRIMA(ckt, red_order);

Each of these are potentially time consuming activities, being that the data I work with is pretty big (10000 x 10000 matrices). So in a previous implementation I had all of these as separate scripts that I had to manually execute (or run a master script) all of them. Each of these stored the data in .mat files. The next program would read from this and write its own result in another directory. And so on.

What I would like to use is a framework that can store the dependencies between various pieces of data, such that at any point of time I can just ask it to generate the output.

It should :

  1. Check if the variable is present in the workspace.
  2. If it is, check if its consistent with the expected properties (check with the config data)
  3. If not, load from file (the exact path to the file will be pre-specified).
  4. Check if its consistent with the expected properties.
  5. If not, compute it from the command associated with it. (pre-specified)

I would like this to be recursive, so that effectively I run the last module and it automatically runs checks and actually computes only those pieces of data that are not already available and consistent.

Can you give some suggestions on how to design this? If it is already called something (I assume it must) please point me to it.

I am working with MATLAB on a model reduction algorithm. It is basically a data processing pipeline.

ckt = generate_ckt(ckt_properties);
freq = generate_fpoints(fconfig);
result = freq_dom_sim(ckt,freq);
red_ckt = run_PRIMA(ckt, red_order);

Each of these are potentially time consuming activities, being that the data I work with is pretty big (10000 × 10000 matrices). So in a previous implementation I had all of these as separate scripts that I had to execute one by one (manually or run a master script). Each of these stored the data in .mat files. The next program would read from this and write its own result in another directory. And so on.

What I would like to use is a framework that can store the dependencies between various pieces of data, such that at any point of time I can just ask it to generate the output.

It should :

  1. Check if the variable is present in the workspace.
  2. If it is, check if its consistent with the expected properties (check with the config data)
  3. If not, load from file (the exact path to the file will be pre-specified).
  4. Check if its consistent with the expected properties.
  5. If not, compute it from the command associated with it. (pre-specified)

I would like this to be recursive, so that effectively I run the last module and it automatically runs checks and actually computes only those pieces of data that are not already available and consistent.

Can you give some suggestions on how to design this? If it is already called something (I assume it must) please point me to it.

2 added new tags
| link
1
source | link

How do I transparently cache data in intermediate stages of processing?

I am working with MATLAB on a model reduction algorithm. It is basically a data processing pipeline.

ckt = generate_ckt(ckt_properties);
freq = generate_fpoints(fconfig);
result = freq_dom_sim(ckt,freq);
red_ckt = run_PRIMA(ckt, red_order);

Each of these are potentially time consuming activities, being that the data I work with is pretty big (10000 x 10000 matrices). So in a previous implementation I had all of these as separate scripts that I had to manually execute (or run a master script) all of them. Each of these stored the data in .mat files. The next program would read from this and write its own result in another directory. And so on.

What I would like to use is a framework that can store the dependencies between various pieces of data, such that at any point of time I can just ask it to generate the output.

It should :

  1. Check if the variable is present in the workspace.
  2. If it is, check if its consistent with the expected properties (check with the config data)
  3. If not, load from file (the exact path to the file will be pre-specified).
  4. Check if its consistent with the expected properties.
  5. If not, compute it from the command associated with it. (pre-specified)

I would like this to be recursive, so that effectively I run the last module and it automatically runs checks and actually computes only those pieces of data that are not already available and consistent.

Can you give some suggestions on how to design this? If it is already called something (I assume it must) please point me to it.