Design: multiple algorithms on the same large data sets

Question

I have several algorithms that I would like to test against the same data sets to compare their results. I don't know how to design it so there is maximum readability and maximum efficiency.

I have considered creating a class for each algorithm, and giving it a copy of the data to work with, but it doesn't seem that that is the right answer:

1. Each data set is fairly large (10,000 float numpy array), so I don't want to copy each one ~30 times.
2. Many of the algorithms have similar pre-processing routines (thus repeating them for each algorithm seems wasteful)
3. Some algorithms have nearly identical code, except a few parameters which are different.

At the same time, having one function call per algorithm also seems wrong: as per (2), many will call the same preprocessing functions, and then it becomes very difficult to tell who is calling who.

I want to be able to allow the user (which will be me) to easily call a variety of algorithms on the data, while keeping the code as clear as possible.

I just keep thinking I need the inverse of a class; where each objects of a class will have the same methods but different data, I need something where each member will have the same data but different methods.

Answer 1

Basically you want to create the large data once and cache it between invocations of your different algorithms.

For this to work, the data you want to share should not be altered by the algorithms you run.

How about a caching factory and some dependency injection?

class DataFactory(object):

  def __init__(self, ...):
    self._raw_data = None  # will be lazy-loaded and cached
    self._preprocessed_with_foo = None  # ditto
    self._preprocessed_with_bar = None  # ditto

  @propery
  def raw_data(self):
    if not self._raw_data:
      self._raw_data = loadDataSomehow(...)
    return self._raw_data

  @propery
  def foo_data(self):
    if not self._preprocessed_with_foo:
      self._preprocessed_with_foo = fooTransform(self.raw_data)
    return self._preprocessed_with_foo

  # etc

class Algo1(object):

  def __init__(self, data_factory):
    self.data_factory = data_factory

  def calculateOne(self):
    return someAlgorithm(self.data_factory.raw_data)

  def calculateFooOne(self):
    return anotherAlgorithm(self.data_factory.foo_data)

class Algo2(...): ...  # by the same pattern

It works like this:

factory = DataFactory(...)  # nothing much happens

algo1 = Algo1(factory)  # dependency on factory is injected into algo1

print aglo1.calculateOne()  # factory loads raw data
print algo1.calculateFooOne() # factory reuses raw data, calculates foo data

algo2 = Algo2(factory)  # note: the same factory

print algo2.calculateTwo()  # assumedly reuses raw data
print algo2.calculateFooTwo()  # reuses both raw data and foo data

Answer 2

> a = np.random.randn(10000)
> %timeit a.copy()
100000 loops, best of 3: 3.9 us per loop

So copying that 30 times takes an imperceptible amount of time. While I would like more information about what these algorithms are (and heartily recommend %timeit, along with the rest of ipython), I doubt that the copies will be the largest of your concerns.

While it is tempting to share initialization blocks, you already say you have most of the meaningful code separated out already. I would suggest writing each algorithm as straightforwardly as possible, above almost any other consideration.

Answer 3

you say you want to "test" different algorithms. If you want to test them to pick the "best" one, I would keep one class per object. At first, some of the code will be duplicated, but at the same time, modifying one algorithm won't have any impact on the other algorithms, and separating them at the end will only be easier as well as adding new algorithms to your test.

If certain algorithm share the same pre processing by design, there's no reason you could not use inheritance for that family of algorithms and cache the result of that preprocessing. It would slightly flaw the comparison between this family of algorithms and other completely unrelated algorithms (memory consumption, CPU cost of caching...).

Now, I say "best" and I don't know what it means. If it's just a matter of performances, an array copy is probably going to be much cheaper than running the algorithm on the whole dataset. And that's something you probably won't have to do once you've selected your best one.

Also, you say you don't want to copy the dataset 30 times. If you don't modify it in your algorithm (which you should not), you don't have to copy it in the first place.

Answer 4

Just to start off, one should never pre-optimize code; code which is easier to understand and more logical is preferable 98% of the time. But since you're bringing it up I suspect you are having problems with it.

Since you're doing a benchmark, you should have one:

class Benchmark(object):
    def __init__(self, data):
        self.data = data
    def __call__(self, algos):
        return (algo.test(self.data) for algo in algos)

benchmark = Benchmark(get_data())
benchmark([Algo1(), Algo2(), ..]

However, I realize upon reading the question again that you would prefer to have something more interactive.

When it comes to the common code between algorithms, you should have proper inheritance to sort all of them out (It's normally bad practice to copy code at all, so work against that by pushing as much code as possible high up in the tree). For the "some algorithms" being identical with changed parameters, one could inject via the constructor like this:

[Algo1(n_states=16), Algo1(n_states=8), Algo2(), ..]

If you want to make a new class out of it you could do

from functools import partial
Algo1a = partial(Algo1, n_states=16)

It's hard to give tips on the problem with common code without more knowledge on your particular algorithms.

For the particular problem where the preprocessing is common for almost all algorithms, I still don't have a straight answer: it depends a bit on your code. But each class should have the responsibility to choose how to do the preprocessing (Single responsibility principle), although a DataFactory is effective.

I wouldn’t ruin a logical code to gain speed by not recomputing the preprocessing, but then again I don't know your particular problem.