-1

I am amazed by the way APIs of some machine learning packages are designed, namely Chainer's and Pytorch's "define-by-run" API. Even Optuna, a hyper parameter tuning library has "define-by-run" API.

How to achieve such design of API? Can anyone guide me towards it with some code?

2
  • 1
    Sharing your research helps everyone. Tell us what you've tried and why it didn't meet your needs. This demonstrates that you've taken the time to try to help yourself, it saves us from reiterating obvious answers, and most of all it helps you get a more specific and relevant answer. Also see How to Ask – gnat Feb 13 at 9:04
  • @gnat Hey buddy. I didn't find any answer regarding define by run. I am completely ignorant on this field. Is it bad to post questions on the platform that is meant to clarify it in the first place? I have already worked with Optuna and was impressed by its API and hence wanted to learn how to implement it. Please don't downvote genuine questions in the name of triage. – Manishankar Singh Feb 15 at 17:21
2

Define-by-run is just a neat application of symbolic programming. Python does not support symbolic programming the way that Lisp, Mathematica, or R do. But Python can emulate it with operator overloading.

The idea is that an expression x + y will not compute the addition of two variables, but produce an object that represents the addition of two variables. We can later manipulate this object as a data structure. In the context of ANN, it is particularly important that we can perform back-propagation.

As a simpler example, let's consider such a symbolic system that supports partial derivatives. First, we need a base class for symbolic operations:

class Symbolic:
  def eval(self, bindings: Dict[str, float]) -> float:
    raise NotImplementedError
  def derivative(self, x: Variable) -> Symbolic:
    raise NotImplementedError
  def __add__(self, other) -> Symbolic:
    return Addition.of(self, symbolify(other))
  def __radd__(self, other) -> Symbolic:
    return Addition.of(symbolify(other), self)
  def __mul__(self, other) -> Symbolic:
    return Multiplication.of(self, symbolify(other))
  def __rmul__(self, other) -> Symbolic:
    return Multiplication.of(symbolify(other), self)

def symbolify(x) -> Symbolic:
  if isinstance(x, Symbolic):
    return x
  if isinstance(x, (float, int)):
    return Constant(float(x))
  raise TypeError

We can create variables that are not yet assigned a value:

class Variable(Symbolic):
  def __init__(self, name: str) -> None:
    self.name = name

  def __repr__(self) -> str:
    return self.name

  def eval(self, bindings: Dict[str, float]) -> float:
    return bindings[self.name]

  def derivative(self, x: Variable) -> Symbolic:
    if self.name == x.name:
      return Constant(1)
    else:
      return Constant(0)

We can create constants with fixed values:

class Constant(Symbolic):
  def __init__(self, value: float) -> None:
    self.value = value

  def eval(self, bindings) -> float:
    return self.value

  def __repr__(self) -> str:
    return str(self.value)

  def derivative(self, x: Variable) -> Symbolic:
    return Constant(0)

And we can create a symbolic object representing addition and multiplication, with some rules for simplification.

class Addition(Symbolic):
  def __init__(self, left: Symbolic, right: Symbolic) -> None:
    self.left = left
    self.right = right

  def __repr__(self) -> str:
    return f"({self.left!r} + {self.right!r})"

  @staticmethod
  def of(left: Symbolic, right: Symbolic) -> Symbolic:
    if isinstance(left, Constant) and left.value == 0:
      return right
    if isinstance(right, Constant) and right.value == 0:
      return left
    return Addition(left, right)

  def eval(self, bindings) -> float:
    return self.left.eval(bindings) + self.right.eval(bindings)

  def derivative(self, x: Variable) -> Symbolic:
    return self.left.derivative(x) + self.right.derivative(x)

class Multiplication(Symbolic):
  def __init__(self, left: Symbolic, right: Symbolic) -> None:
    self.left = left
    self.right = right

  def __repr__(self) -> str:
    return f"({self.left!r} * {self.right!r})"

  @staticmethod
  def of(left: Symbolic, right: Symbolic) -> Symbolic:
    if isinstance(left, Constant):
      if left.value == 0:
        return left
      elif left.value == 1:
        return right
    if isinstance(right, Constant):
      if right.value == 0:
        return right
      elif right.value == 1:
        return left
    return Multiplication(left, right)

  def eval(self, bindings) -> float:
    return self.left.eval(bindings) * self.right.eval(bindings)

  def derivative(self, x: Variable) -> Symbolic:
    # product rule
    return self.left.derivative(x) * self.right \
         + self.left * self.right.derivative(x)

We now have sufficient infrastructure to define and manipulate a symbolic expression:

>>> x = Variable('x')
>>> y = Variable('y')
>>> expr = x + 2 * y
>>> expr
(x + (2.0 * y))
>>> expr.eval(dict(x=3, y=5))
13.0
>>> dy = expr.derivative(y)
>>> dy
2.0
>>> (x * y).derivative(x)
y

So this isn't magic, just lots of tedious code. There are limitations of this approach, e.g. such symbolic expressions don't work well with control flow: you can't assign a true or false value to these expressions until they are evaluated.

You mention Optuna which doesn't have this limitation, since it doesn't overload Python expressions as Chainer or Numpy would do. Instead, the programmer explicitly requests values from a context which can then be stored:

def objective(trial):
  x = trial.suggest_uniform('x', 0, 10)
  y = trial.suggest_uniform('y', -10, 10)
  # x and y are normal Python numbers
  if y > x:
    return x + 2 * y
  else:
    return 3 + x * y

study = optuna.create_study()
study.optimize(objective)

After the objective is executed, the trial contains a black-box view onto the result, for example:

trial_input = {
  'x': 3.0,
  'y': 5.0,
}
trial_output = {
  'result': 13.0,
}

The optimizer does not know about the algebraic relationship between the various values, but can build a black-box model about their interactions. After many iterations, the optimizer might “learn” about the correct interactions and will thus be able to find the optimal values for x and y.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.