How compatible are data science notebooks with clean architecture?

Question

Clean architecture decouples an app's core from the presentation/UI layer. The UI is just a plugin, replaceable (eg, web-based to desktop) without impacting the core.

Many data science apps mix code, user inputs, text, graphics and other outputs in one notebook, eg, Jupyter. Everything seems coupled: the domain, UI, presentation, persistence.

Q: How to design such an app cleanly, with the notebook maximally decoupled? Or are notebooks inherently incompatible with clean architecture?

Perhaps I could have an independent module with core functionality. The notebook would call this module, without defining any non-trivial functionality. Would this, however, allow enough decoupling or even fit with a notebook?

Why:

I'll be developing an app for a client who's only used Excel. The app will predict cost effectiveness of medical treatments and will need MCMC simulations, regression and other stats.

I plan to implement it in Python with Jupyter or the nteract notebook, pushed by Netflix https://medium.com/netflix-techblog/tagged/nteract. However, this may eventually prove unsuitable for the client, as Jupyter is mainly used by those who program it themselves. There're other potential pitfalls, eg, https://docs.google.com/presentation/d/1n2RlMdmv1p25Xy5thJUhkKGvjtV-dkAIsUXP-AL4ffI/edit#slide=id.g362da58057_0_1. Ideally, I could easily swap between notebook types or change over to a desktop GUI.

Answer 1

Jupyter already has a fairly modular architecture. You have to distinguish:

• The notebook format which contains input and output cells. This is essentially just a JSON file.
• The kernel which executes code in a particular environment and language. For example you might have an R kernel, a Python kernel, and another Python kernel for code in a particular virtualenv.
• A notebook viewer or console which displays a notebook and/or allows interaction with a kernel. For example, there's the Jupyter web interface but there are also a couple of native programs. GitHub has a built-in notebook viewer.

Incidentally those three components fit a MVC architecture quite well. In particular, kernels are easy to change and do not depend on the other components!

You might want to consider whether you can already mix and match suitable components to meet your needs. However, the notebook format is the one thing you cannot easily change because the notebook is a common data model between multiple parts of a software. Similarly, all the other components (user interfaces, persistence) depend on the model in the clean architecture.

Answer 2

I plan to implement it in Python with Jupyter or the nteract notebook, pushed by Netflix

Here's your use case for Clean Architecture right here. How much of your code has to KNOW which notebook you decided to use? Does this choice infect every line of your code?

Architecture called Clean, Onion, Ports and Adapters, or whatever, teaches you how to put a firewall between a detail like which notebook you used and your hard won business logic that, if you are very careful, doesn't care which notebook you used.

The notebook provides you with services you need. Look hard at what those needs are and strip them of any hint about which notebook, if any, is providing them. Don't bother with services you don't need. That creates your firewall against changing notebooks.

So long as there are places in your code that can't tell what notebook you used then they don't care when you change what you use. That's really powerful. It's just not free. It's work.

You have to decide if being able to swap notebooks is worth putting in that work.