0

I have a flask server which uses a controller-service-resource model.

For simplification it is like this:

Controller(parses requests, dispatches to A or B):
  ServiceA
  ServiceB

However there are cases where I want to use ServiceB inside of ServiceA, and ServiceB inside of ServiceA, however this leads to circular imports. We have looked at subdividing the services, but it gets impractical.

An architecture I am considering is to use a singleton object which is accessible by all.

from __future__ import annotations
from typing import TYPE_CHECKING

if TYPE_CHECKING:
    from service_a import ServiceA
    from service_b import ServiceB

class ServiceClassDispatcher:

    def __init__():
        self.service_a_class: ServiceA | None = None
        self.service_b_class: ServiceB | None = None

MasterDispatch = ServiceClassDispatcher()

In Controller

from service_a import ServiceA
from service_b import ServiceB
from service_class_dispatcher import MasterDispatch

MasterDispatch.service_a_class = ServiceA
MasterDispatch.service_b_class = ServiceB

class Controller:
   ....

In service_a

from service_class_dispatcher import MasterDispatch

class ServiceA:

   def some_method_needing_service_b(self, body):
       service_b = MasterDispatch.service_b_class(self.some_input_needed)
       ...

In service_b

from service_class_dispatcher import MasterDispatch

class ServiceB:

   def some_method_needing_service_a(self, body):
       service_a = MasterDispatch.service_a_class(self.some_input_needed)
       ...

This will give me a type hints on the object and avoid the circular import conundrum. Are there any problems with this or simpler solutions?

EDIT

I appreciate the answers so far, however the underlying nature is that serviceA and serviceB are both providing information for a non-linear differential equation that is iterative, and hence circular in nature.

Starting from seed values, service A and service B each provide values to each other creating more accurate values. The calculation is both circular and convergent, which makes it deterministic for all practical purposes. In other words we have software tests and consistently pass the software tests and system is robust in the field.

The actual application is in aerodynamics where a large amount of existing data can be used to improve the accuracy of the initial data.

System testing is performed by inputing seed data, and ensuring results which are floating point numbers are within tolerance. So it always requires the controller and service A and B. Creating these mockups is not difficult.

In a simpler case, you would have two methods inside the class, or two classes within a file. As the data is convergent, the iteration can stop when within tolerance.

In our case the information accumulates over time and is stored in a database, and involves two different types of clients.

The existing structure can be simplified as:

Controller ServiceA SimplifiedServiceB ServiceB SimplifiedServiceA

The actual structure ends up a bit more complicated with more than one simplified service. We end up with either duplicated code or a complicated inheritance.

It has been put forward that Node for example, can handle this circular service arrangement in a very simple manner. It is very easy to make calls to other services without worrying about circular imports. However, the general layout of a flask server does not lend itself well to this.

2
  • 1
    In some cases, you can untangle imports by deferring them. E.g. move the actual import into a function. You can also separate interface from implementation, e.g. by writing a Protocol describing one service. The other service can then import the protocol, and is given the actual implementation at runtime (compare the "Stairway Pattern"). But in most cases, cyclic imports can be solved by writing smaller, more cohesive modules.
    – amon
    Commented May 27 at 21:55
  • We are trying something similar to this. However, several members on the team have pointed out that the underlying algorithm is circular, and the entire pattern is to avoid python's circular imports. In node for example, we can have independant services that can call each other without any problem and results in far simpler and more elegant code. Commented May 28 at 13:12

3 Answers 3

5

Fix the root cause of this problem, which is circular dependency.

We have looked at subdividing the services, but it gets impractical.

Why? That's exactly what you should do. Subdivide the services, untie this knot, get rid of circular dependencies. Now I understand it might be costly, but avoiding the true problem is not a solution. Don't go down that road. Circular dependencies is a huge red flag. 99% of time this is bad design that will back fire. And I've never seen a production Python code that actually needs circular dependencies.

So instead of solving core problem, you wrap those services with this thing. And you've made it worse, because the circular relationship is still there, except it is hidden now and implicit. In order for those services to work I have to import the controller first? I mean, it doesn't matter that the initialization is in controller. Regardless of where you put it, when I import something I expect it to work. But with this design it won't, I need an additional import. Just import, the imported thing is not going to be used. Unless you put it in a controller, for another implicit hidden side effect. Counterintuitive and unclear. And what is worse, that import cannot be in service_a and service_b files, because of circular imports. So you need that import, but you have to put it somewhere else...

I mean, look at this design: how am I supposed to test it? I don't even know where to start, and how to do it properly. How can I reason about it? And now imagine your team doing such designs over and over again for two years. At that point leaving the company is the only way to stay sane.

I'm sorry for being bluntly honest, maybe even rude, but this is an abomination. Fix those circular dependencies.

5
  • Hear, hear! I have worked on large production code bases where someone snuck in a circular import dependency, and it only becomes more nightmarish as we hire more folks who come to believe "oh, I should write my new code just like that old code". We wind up with conditional imports, we can no longer alphabetize imports because the order matters, we can't import Z unless we've already imported A, B, C, it just goes on and on. You have a buggy design. Fix the bug, and then implement the improved design.
    – J_H
    Commented May 27 at 21:24
  • Thank you for your answer, I have provided a bit more information as to the root of the circular dependencies, and why they are required. Commented May 28 at 13:16
  • 1
    @VectorVictor sorry, but your edit doesn't address the "why you need circularity?" question at all. If those services are tightly related then maybe they should be one service. Which I doubt very much. Otherwise subdivide them. Either way circular dependencies (explicit or not) is a mistake.
    – freakish
    Commented May 28 at 13:37
  • 3
    Btw, it sounds like you need an event bus. And then your two services simply accept events and emit events. You can then indeed make the process circular. Which is not the same as circular dependency in code.
    – freakish
    Commented May 28 at 13:44
  • @VectorVictor You have 3 services: the parts of A that don't depend on B, the parts of B that don't depend on A, and C that depends on both A and B
    – Caleth
    Commented May 31 at 11:48
2

The circular import error can be avoided by using the import structure below

No.

This isn't accurate.

Please don't do this, and especially please do not encourage others to do this.

Do the right thing: break out a dataclass, dict, or similar DTO, which services {A, B} accept as input and will iteratively mutate to their heart's content.


Here is OP's proposed solution.

service_a.py

import service_b

class ServiceA:
    def __init__(self, body):
        self.some_input_needed = body

    def __repr__(self):
        return f"<ServiceA {self.some_input_needed}>"

    def some_method_needing_service_b(self, body):
        service_b_instance = service_b.ServiceB(self.some_input_needed)
        return body + ' world'


if __name__ == "__main__":
    print('main A')
    b = service_b.ServiceB()
    print(b.some_method_needing_service_a('hi'))

service_b.py

from service_a import ServiceA


class ServiceB:
    def __init__(self):
        self.some_input_needed = 'initial B'

    def some_method_needing_service_a(self, body):
        service_a_instance = ServiceA(self.some_input_needed)
        return service_a_instance


if __name__ == "__main__":
    print('main B')
    b = service_b.ServiceB()
    print(b.some_method_needing_service_a('greetings'))

service_b_test.py

import unittest
import service_b
from service_a import ServiceA

class ServiceBTest(unittest.TestCase):
    def test_b(self):
        b = service_b.ServiceB()
        self.assertEqual('<ServiceA initial B>',
                         repr(b.some_method_needing_service_a('hello')))

Let's run it the way typical developers expect they should be able to, using python 3.12.3.

$ python -c 'import service_b'
$
$ python service_a.py
main A
<ServiceA initial B>
$
$ python -m unittest service_b_test.py
.
----------------------------------------------------------------------
Ran 1 test in 0.000s

OK

Wow, works great! Guess we have a winner here, right?

No.

We flipped a coin and happened to get lucky. Here is the obverse side.

$ python -c 'import service_a'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/service_a.py", line 1, in <module>
    import service_b
  File "/tmp/service_b.py", line 1, in <module>
    from service_a import ServiceA
ImportError: cannot import name 'ServiceA' from partially initialized module 'service_a' (most likely due to a circular import) (/tmp/service_a.py)

Ouch! It hurts.

Let's debug things with an interactive script, as nearly all developers will at some point try to do.

$ python service_b.py
Traceback (most recent call last):
  File "/tmp/service_b.py", line 1, in <module>
    from service_a import ServiceA
  File "/tmp/service_a.py", line 1, in <module>
    import service_b
  File "/tmp/service_b.py", line 1, in <module>
    from service_a import ServiceA
ImportError: cannot import name 'ServiceA' from partially initialized module 'service_a' (most likely due to a circular import) (/tmp/service_a.py)

Those sure are some sharp edges!

Let's swap the order of the last two import statements in the unit test. Oh, too bad! It's cyclic.

It gets worse. People will add services C, D, E to the project, inducing their own (tree-like!) dependency graph and consuming A. And then according to the details of import order / alphabetized module names, when F calls into E it may or may not work. It all depends on whether we won the coin toss, and on whether the project documentation and training was sufficient for F's author to notice the pitfall and avoid it.

Do we have a "solution"? No. We have a land mine, lying in wait. Not a pretty situation. I have worked on production projects like this for more than a year before we managed to clear the land mines.

A project that exhibits such behavior will have trouble with

  • training staff
  • retaining existing staff
  • recruiting new staff
  • routine refactors
  • alphabetizing imports
  • renaming modules

Don't do it. Don't encourage your friends to do it. There's a reason the DTO was invented. It's simple; it has no dependencies.

-2

I will answer my own question as I found what I was looking for on Stack Overflow - https://stackoverflow.com/a/22210807/2422685

The circular import error can be avoided by using the import structure below with no additional files needed:

In service_a.py

import service_b

class ServiceA:

   def some_method_needing_service_b(self, body):
       service_b_instance = service_b.ServiceB(self.some_input_needed)
       ...

In service_b.py

import service_a


class ServiceB:

    def some_method_needing_service_a(self, body):
        service_a_instance = service_a.ServiceA(self.some_input_needed)

The other answer posted and the comments ask the question of why a circular dependency is needed.

Many modelling situations have distinct and separate inputs and outputs, these can almost always be ordered into a linear code with hierarchal dependencies.

However, this is not always the case, many complex equations can only be solved iteratively. To solve problems like these, recursion is a valid design pattern, an can be an efficient solution for solving complex equations. In our case, we have a recursive algorithm which involves a server and client and thus far larger than a single file.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.