I asked this question already at stackoverflow together with a serialization related part and at codereview for the design part only. Since the design related part receives no answers or comments on stack exchange and was off-topic on codereview I'd like to ask about it here.
The problem: I have a complicated class hierarchy in which the classes are similar to each other but every class contains a bunch of more or less complicated stateful variables. To give you an impression, please have a look at this minimal working example:
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import numpy as np
class OptimizableVariable:
"""
Complicated stateful variable. It is intended to return mainly
floating point values. Status determines whether an optimization
later shall change the value ("V") or not ("F"). The status "P"
is for a variable which depends on a number of other variables.
"""
def __init__(self, name="", status="F", **kwargs):
self.name = name
self.status = status
self.parameters = kwargs
self.eval_dict = {"F": self.eval_fixed,
"V": self.eval_variable,
"P": self.eval_pickup}
def eval_fixed(self):
return self.parameters.get("value", 0.0)
def eval_variable(self):
return self.parameters.get("value", 0.0)
def eval_pickup(self):
fun = self.parameters.get("function", None)
optvar_arguments = self.parameters.get("arguments", tuple())
if fun is not None:
call_args = tuple([ov.evaluate() for ov in optvar_arguments])
return fun(*call_args)
def evaluate(self):
return self.eval_dict[self.status]()
def setvalue(self, value):
if self.status == "F" or self.status == "V":
self.parameters["value"] = value
class OptimizableVariableContainer:
"""
Class which contains several OptVar objects and nested OptVarContainer
classes. Is responsible for OptVar management of its sub-OptVarContainers
with their respective OptVar objects.
"""
def __init__(self, name="", **kwargs):
self.name = name
for (key, value_dict) in kwargs.items():
setattr(self, key, OptimizableVariable(name=key, **value_dict))
def getAllOptVars(self):
def addOptVarToDict(var,
dictOfOptVars={},
idlist=[],
reducedkeystring=""):
"""
Accumulates optimizable variables in var and its linked objects.
Ignores ring-links and double links.
@param var: object to evaluate (object)
@param dictOfOptVars: optimizable variables found so far (dict of dict of objects)
@param idlist: ids of objects already evaluated (list of int)
@param reducedkeystring: to generate the dict key (string)
"""
if id(var) not in idlist:
idlist.append(id(var))
if isinstance(var, OptimizableVariableContainer):
for (k, v) in var.__dict__.items():
newredkeystring = reducedkeystring + var.name + "."
dictOfOptVars, idlist = addOptVarToDict(v,
dictOfOptVars,
idlist,
newredkeystring)
elif isinstance(var, OptimizableVariable):
newreducedkeystring = reducedkeystring + var.name
dictOfOptVars[newreducedkeystring] = var
# here other elifs are also possible, checking for
# OptimizableVariables in lists or dicts within the
# class instance
return dictOfOptVars, idlist
(dict_opt_vars, idlist) = addOptVarToDict(self, reducedkeystring="")
return dict_opt_vars
def getStatusVVariables(self):
"""
Get all OptimizableVariables with status "V"
"""
optvars = self.getAllOptVars()
return [ov for ov in optvars.values() if ov.status == "V"]
def getVValues(self):
"""
Function to get all values of status "V" variables
into one large np.array. Interface to scipy.optimize functions.
"""
return np.array([a.evaluate() for a in self.getStatusVVariables()])
def setVValues(self, x):
"""
Function to set all values of active variables to the values
in the large np.array x. Interface to scipy.optimize functions.
"""
for i, var in enumerate(self.getStatusVVariables()):
var.setvalue(x[i])
class LocalCoordinates(OptimizableVariableContainer):
"""
Specific implementation of class OptVarContainer
"""
def __init__(self, name=""):
super(LocalCoordinates, self).__init__(name=name,
**{"x": {"status": "F", "value": 0.},
"y": {"status": "F", "value": 0.},
"z": {"status": "F", "value": 0.}})
class Surface(OptimizableVariableContainer):
"""
Specific implementation of class OptVarContainer
"""
def __init__(self, name=""):
super(Surface, self).__init__(name=name,
**{"curvature": {"status": "F", "value": 0.0}})
self.lc = LocalCoordinates(name=name + "_lc")
class System(OptimizableVariableContainer):
"""
Specific implementation of class OptVarContainer
"""
def __init__(self, name=""):
super(System, self).__init__(name=name,
**{"some_property": {"status": "F", "value": 1.0},
"some_other_property": {"status": "F", "value": 0.0}})
self.surf1 = Surface(name="surf1")
self.surf2 = Surface(name="surf2")
def surf1_lc_on_circle(mysys):
"""
Scalar function to verify that x, y of lc of surf1 are on a unit circle
"""
return (mysys.surf1.lc.x.evaluate()**2 +
mysys.surf1.lc.y.evaluate()**2 - 1)**2
# optvars = mysys.getStatusVVariables()
# numpy_vector = np.asarray([ov.evaluate() for ov in optvars])
def main():
# creating OptimizableVariableContainer with some nested
# OptimizableVariableContainers.
my_sys = System(name="system")
# It is intended behaviour to access the OptimizableVariable objects via
# scoping within the class hierarchy.
print(my_sys.surf1.lc.x.evaluate())
my_sys.surf1.lc.x.parameters["value"] = 2.0
print(my_sys.surf1.lc.y.parameters)
my_sys.surf1.lc.z = OptimizableVariable(name="z", status="P",
function=lambda x, y: x**2 + y**2,
arguments=(my_sys.surf1.lc.x,
my_sys.surf1.lc.y))
my_sys.surf1.lc.x.status = "V"
my_sys.surf1.lc.y.status = "V"
print(my_sys.surf1.lc.z.evaluate())
# The following construction is quite ugly:
# a) due to the dict: order is not fixed
# b) due to recursion (and maybe lexical sorting) it is not fast
# c) goal: hand over these optvars into some numerical optimization
# via a numpy array => should be fast
optvarsdict = my_sys.getAllOptVars()
for (key, ov) in optvarsdict.items():
print("%s(%s): %f" % (key, ov.status, ov.evaluate()))
# Optimization of status "V" variables due to a scalar optimization
# criterion is an important point in the code. This is done manually in
# the following:
print(surf1_lc_on_circle(my_sys)) # check value of scalar function (>0)
print(my_sys.getVValues()) # get numpy array of V variables
my_sys.setVValues(np.array([0., 1.])) # this pair is on a unit circle
# Notice that since the values come from a dict, it is not guaranteed
# that the array is [x, y] nor is guaranteed that the order is preserved
print(surf1_lc_on_circle(my_sys)) # check value of scalar function (==0)
# Notice that my_sys got changed (which is intended behaviour):
# Notice also that my_sys.surf1.surf1_lc.z changed since it is of
# status "P" (also intended behaviour)
optvarsdict = my_sys.getAllOptVars()
for (key, ov) in optvarsdict.items():
print("%s(%s): %f" % (key, ov.status, ov.evaluate()))
if __name__ == "__main__":
main()
The question is now: How to access the OptimizableVariable objects the best. I already thought about to use some kind of pool object and use some proxy within the class hierarchy to have a link to the pool. This pool should not be a singleton, since it should be possible to manage more than one pool at a time. Up to now the access to the OptimizableVariable variables within my own code is done via the getAllOptVars function. This is quite ugly and only considered temporary. Is there a better alternative to this function?
To summarize and clarify my goals:
- The class hierarchy is ok and reflects the model context of our problem
- The OptimizableVariables belong to each object in the hierarchy and should also stay there also due to model context
- The access and post processing of the OptimizableVariables (i.e. collecting them from the hierarchy and handing them over to an optimizer as a numpy array) is not considered to be optimal. There I need some suggestions on doing better (i.e. getting rid of isinstance and id queries).
- A "nice to have" would be: Decouple serialization of the OptimizableVariables and version management of classes from the object hierarchy (i.e. it would be nice to have an interface to make snap shots of certain objects in the hierarchy)
- Another "nice to have" would be: How to incorporate shared OptimizableVariables and "singleton" OptimizableVariables in this class hierarchy best?
I am aware that no unique solutions exist for these design problems, but I need further input on that.
PS: If the minimal working example is too long and does not show the goal of the code, I'm sorry and I could reduce it.
