How to achieve inheritance when using just modules and vanilla functions in Python?

Question

Python doesn't enforce any paradigm. It gives you freedom. Python provides encapsulation at module level.

If I have a module A and have a module B with same interface. How do I sort of inherit B from A and override some functionality provided by B ?

Answer 1

When You import Y module, that imports functions from module X with from X import *, all functions from X are available as they were in Y ( You can thing of that, like pasting content of X into Y ).

And, when you have multiple defs, Python will take the last one. So to override functions You have just to redefine those function after you import X.

Also, when You want to use original function from X in Your Y, you can add aditional import X and access it with X.functionName().

---

Example:

module1.py

def foo():
   print "this will be overridden"
def bar():
   print "this will be preserved"

module2.py

from module1 import *
import module1

def foo():
   print "This was foo:"
   module1.foo()
   print "But it was overridden."

run.py

import module2

module2.foo()
module2.bar()

Example's result:

-> % python run.py 
This was foo:
this will be overridden
But it was overridden.
this will be preserved

Answer 2

What is the point of modules?

Modules are a mechanism to structure an application into distinct, reusable parts. E.g. a command line utility could factor out the code that does the argument parsing into a separate module that can then be used in other command line tools as well. Since a module has a clear public interface and can hide implementation details within the module, the overall application also becomes less tightly coupled.

Can I swap out a module for another module with the same interface?

Not necessarily, and that is not the point of modules. However, there are various techniques to implement modules. One is to define a structure of function pointers that defines the type of the whole module. When a module is loaded, it provides an instance of this module. This technique is frequently used to mimic modules in JavaScript:

// a mathModule may be anything that supplies "add" and "sub" functions
var mathModule = (function () {
  var privateVar = "secret";
  function add(a, b) { return a + b; }
  function sub(a, b) { return a - b; }
  function privateFunction() { return "very" + privateVar; }

  // return an “object” (Python: dict) that defines the public interface
  return {
    add: add,
    sub: sub,
  };
})();

This technique can also be adapted to C to provide real modules, and can be easily translated to Python. However, neither language has lambdas and C lacks closures, so the translated example would be more restricted.

Since we are dealing with function pointers here, we have an important level of indirection. In principle, we could have multiple instances of the same module interface at once, and we can easily switch to a different implementation if it offers the same interface.

This sounds very much like object-oriented programming, but a few important details are missing: for OOP, the instance structure must also contain data in addition to methods, and must pass the instance structure as an argument back into any method that was invoked. I discuss creating a JavaScript object system out of closures in a blog post in more depth.

Do Python's modules work like this?

Kind of. In fact, Python's modules act very much like a regular object except that we don't have a self argument. However, Python misses the part where we actually define a clear interface for our module. Where the above JavaScript example can explicitly choose whether to expose a function through the public interface, Python makes any symbol in file scope accessible when importing a module. There are tricks that limit this issue (such as defining functions in helper modules and then selectively importing the public interface into the module's __init__.py), but in Python we end up defining the interface through our documentation, not through the code.

So, long story short, Python has namespaces that it calls “modules”, but don't offer any encapsulation.

So, how can I inherit a module?

The concept of “inheritance” makes no sense when discussing modules, as it's more of an OOP term. There, “Subclass inherits from Base” means (a) that Subclass is a subtype of Base and (b) that Subclass instances can somehow use the implementation defined by Base. As you might know, reusing the implementation can be done via two mechanisms: implementation inheritance (where the method lookup mechanism for Subclass is initialized with the methods for Base) and composition (where Subclass delegates to a Base instance).

With the module-as-dict encoding, this can be illustrated as such:

# The original module.
# Can be instantiated like `mathModule = mathModule_load()`
def mathModule_load():
  private_var = "secret"
  def add(a, b): return a + b
  def sub(a, b): return a - b
  def private_function(): return "very" + private_var

  return {
    'add': add,
    'sub': sub,
  }

# Inherit the mathModule via composition
# Overrides "add" to type-check for int arguments
def checkedMathModule_load():
  base = mathModule_load() # the module instance we inherit

  def add(a, b):
    if type(a) is not int or type(b) is not int:
      raise TypeError("give me ints!")
    return base['add'](a, b)

  def sub(a, b):
    return base['sub'](a, b)

  # return our own module instance that meets the expected interface
  return {
    'add': add,
    'sub': sub,
  }

# Inherit the mathModule by changing the module definition
# Overrides "add" to emit debug info
def chattyMathModule_load():
  module = mathModule_load() # the module instance we inherit

  original_add = module['add']
  def my_add(a, b):
    print("DEBUG: add(%d, %d)" % (a, b))
    return original_add(a, b)
  module['add'] = my_add

  return module

OK, can we now do that with built-in modules?

*grumble* Python may formally be multi-paradigm, but it very clearly favours OOP solutions to problems like this. We can rework the above example to use Python's built-in modules, but it isn't pretty. Note that Python's modules don't have a comparable concept of “instances”, as modules are global.

Instead of the explicit dict that stores the function of the module, we will use regular Python syntax to add symbols to the file scope.

The modules are organized in the following file hierarchy:

mathModule/
  __init__.py
  detail.py
checkedMathModule/
  __init__.py
  detail.py
chattyMathModule/
  __init__.py
  detail.py

mathModule/__init__.py:

from .detail import add, sub
del detail

mathModule/detail.py:

private_var = "secret"

def add(a, b):
  return a + b

def sub(a, b):
  return a - b

def private_function():
  return "very" + private_var

checkedMathModule/__init__.py:

# this module offers the same interface as mathModule
from .detail import add, sub
del detail

checkedMathModule/detail.py:

import mathModule as base

def add(a, b):
  if type(a) is not int or type(b) is not int:
    raise TypeError("give me ints!")
  return base.add(a, b)

def sub(a, b):
  return base.sub(a, b)

chattyMathModule/__init__.py:

from mathModule import *

# now import the function we want to override
from .detail import my_add as add
del detail

chattyMathModule/detail.py:

from mathModule import add as original_add

def my_add(a, b):
  print("DEBUG: add(%d, %d)" % (a, b))
  return original_add(a, b)

I am not sure that the del detail won't break anything, but without it users could do mathModule.detail.private_function() and get something 'verysecret', which completely breaks encapsulation.

These techniques kind of work, but are a lot of unnecessary effort for something that could be done much more easy with regular objects. Also, Python is special because modules are represented as module objects that can be passed around as values, whereas the module system of most languages is far more static. These techniques are not representative of modular programming, but rather of a “I want to do OOP without using the OOP parts of my language” mindset that is not constructive for anything except understanding OOP better.