When making changes to large systems, I often face the problem that some piece of functionality needs to get some data from another piece, but they're in different parts of a deep and branching call tree, possibly flowing through event listeners, deferred calls, etc. In this way a simple change can balloon quickly.

A related quote from Yossi Kreinin's blog post at http://www.yosefk.com/blog/i-want-a-struct-linker.html:

You have some kind of data structure that you pass around a lot. Soon, the most valuable thing about the structure isn't the data it keeps, but the fact that it's available all the way through some hairy flow of control.

Global variables are one classic way to let code "shout out" to distant code, but they're known to be problematic. Dynamically scoped variables are a more restricted way, but they're problematic as well.

Is there any programming language research aimed at solving this problem? Can we make it easier to add unanticipated flows of data to a large codebase, while still having static checking, easy unit testing and other goodies?

  • The way you are phrasing your question I guess you have data flow in a single process in mind, no interprocess communication. So what kind of problem do you see which cannot be solved by standard event sender/listener mechanisms? – Doc Brown Feb 26 '13 at 17:25
  • A contrived example: imagine that deep within your system there's some code that sends the user a text message. And you get a new requirement that the text of the message should depend on the current time in the user's timezone. The callstack looks like this: some code that knows the user's timezone, calls a method that calls a method that (...repeat 15 times) calls a method that generates the message text. This is a simple example by my standards, because it involves communication only downward, but still you must change the signatures of 15 methods to make your trivial change. – Vladimir Slepnev Feb 26 '13 at 18:02
  • Well, I guess what may help is to model the data flow explicitly, and separate components from data flow. The german software engineer is writing a lot about this topic, most articles in german. Here is an entry article in english of him: geekswithblogs.net/theArchitectsNapkin/archive/2011/03/19/… – Doc Brown Feb 26 '13 at 18:33
  • I think a singleton internal API could help. It would be accessible all over the application and would encapsulate all the data-retrieving logic. – superM Feb 26 '13 at 19:07

You are referring to CDI (Context Dependency Injection) AKA IoC (Inversion of Control). Java JSF and Spring Framework are some examples. ASP.NET MVC has plugins like Unity. Javascript is beginning to have organized structures using libraries like RequireJS, which has injection behavior seen in many modern JS frameworks. That is for wiring up local and remote applications.

For loose coupling across networks, companies like to use Web Services with SOAP, REST, AJAX, or regular remote method calling with RPC. In Java you can use JAX-WS or .NET WCF to build distributed services. Then you line them up in a service bus or "data flow" from any language or platform as a client. Ruby, Python, Scala, Java, C#, ... anything.

Loose coupling allows you to divide and conquer problems, and services are often the entry point to a database for pulling data. Stepping up the ladder we have the beast called Message Queue. That road leads to enterprise and infrastructure type frameworks.

If your project insists on no network, though, there are languages like Scala, Akka, NodeJS etc.. that are designed for a high flow of data within a single application. They also work with some or all of the previously mentioned technologies for complex projects. For example, Scala can be used with JAX-RS REST services for pulling sort of "global data" from a data source, and have Spring for the IoC internal wiring. There are also many business execution or workflow frameworks in JBoss, .NET, and GUI tools like MuleESB. In development, Eclipse and Netbeans lets you drag and drop services in a visual flow chart screen.

Finally, Java has Singleton beans still. For adjusting your methods at run-time, use proxy or reflection frameworks. But honestly, that is so 1999.

If you're making that many calls to send a user a message based on their timezone, in my opinion, there's probably a 2-step way to achieve the same effect that the user sees. But yeah, CDI frameworks are worn by existing languages like a coat that gives them all of the flexible powers you mentioned. I like to call it my program's subconscious, taking care of dirty work seamlessly.

  • Message Queues may be overkill but messaging is the perfect way to get events fired across the board. Java uses Message Driven Beans (MDB) and that should allow your program to send or receive 'conversation' with each other. You can do it this way for the asynchronous bonus. – Senor Developer Mar 2 '13 at 20:06
  • Thanks for the pointers! It definitely makes me wonder how a language could look like if it was designed from the ground up to support dependency injection and similar patterns. – Vladimir Slepnev Mar 5 '13 at 15:14

The simplest way to do this over a large scale is in fact to use some sort of data encapsulation API. This could be a NoSQL store or it could be an encapsulated RDBMS (or it could in fact be both in different times and places in the same application-- there isn't any reason why you can't have an RDBMS handling long-term storage and a NoSQL db handling short-term state control). It could even be a series of singleton objects.

Your data structures can then be made available in some sort of neutral space, in a somewhat managed way. This is the approach we take with LedgerSMB (but with a few semi-global variables for what are essentially stashed singletons, but again these are managed, we chose directly stashing the object because it made management of the variables a little easier but then there are all of 4 of them).

Of course any approach has tradeoffs and you cannot get around those tradeoffs. The key is to look at what the tradeoffs are (management vs performance vs cleanliness of code vs potential coding pitfalls) and make a decision based on what is best for your application.

  • Thanks for the answer! It seems to me that programming language research could help with this problem. For example, if the code reads some data from a global database or from stashed singletons, there could be a static/declarative guarantee about which data it requires. – Vladimir Slepnev Mar 1 '13 at 15:53

If you use (or quote) the words

hairy flow of control

then I assume that your code really is a mess. You should drop it immediately. If you use modularization/separation of concerns, there is no such thing as a "hairy flow of control". Your code simply lacks simplicity which is also inferrable by the fact that you referred to global variables :-).

  • Why the downvote? The quote was missing the introduction which exactly supports my view: "(it's probably classified as an "Antipattern" or a "Code Smell" and as such has a name in the appropriate circles, but I wouldn't know, so I'll leave it nameless)" – user127749 Mar 2 '13 at 23:16
  • 2
    This isn't really an answer to the question, that's probably the reason for the downvote – Daniel Gratzer Mar 2 '13 at 23:24
  • Then let me rephrase the question: is there any magic trick to undo the mess of code that violates one of the most basic principles of software design: KISS (keep it simple, stupid)? The trick is not magic, it is either a programmer that cannot get replaced because he knows all not-so-obvious details (which will kill the company in the long term), or to restructure the code base. Unfortunately, many companies initially don't care for proper code design or don't even understand the consequences, then having to rewrite their code at least once, many even rewrite it many times over... – user127749 Mar 3 '13 at 0:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.