We have a project that is meant to do signal processing. If all we had to do with the software was that one job, we would have a pretty clean architecture. Unfortunately, we've run into an issue most other projects standard enterprise projects don't ever deal with. We need to generate a qualitative scientific analysis of our algorithms and software in a way that interfaces with domain experts that aren't software engineers. This analysis is used to develop new novel algorithms to improve the signal processing chain further.

The "not being software engineers" isn't the issue here; the tools we create for them also end up helping us developers a lot. We also are not at odds with the importance of these metrics, IE, the entire software development team, agrees that these metrics need to be exposed to better our software. Not exposing these metrics is not an option.

We have to dig deep into the implementation details of our signal processing chain to get the metrics we need for analysis, with many callbacks and configuration parameters that are completely ignored when our application is not run under a development environment. It creates massive coupling issues and complicated code paths. For example, we originally would have a function signature that looks like this:

 SignalProcessingChainCommunicator(output_port, signal_processing_configuration...)

but with the added metrics, becomes something like:

 SignalProcessingChainCommunicator(output_port, signal_processing_configuration..., enable_publish_a, ... enable_publish_z, log_results_a, ... log_results_z)

And resulted in lots of conditionals inside the code and required adding callbacks to internal dependency injection objects. We decided we needed to refactor when we started getting dozens of parameters that obfuscated the real code underneath. We created a "CallbackConfigurator" object which dealt with enabling the publishing and logging, and assigning callbacks to the DI (we couldn't remove the callback members, we still needed the internal signal processing objects to send out the information we needed) turning the above into this:

  SignalProcessingChainCommunicator(output_port, signal_processing_configuration..., callback_configurator)

and turning this:

 if enable_publish_a:
     object_a.callback = publish_a
 if enable_publish_z:
     object_z.callback = publish_z
 if enable_log_a:
     old_callback = object_a.callback
     def new_callback(obj):
     object_a.callback = new_callback
 if enable_log_z:
     old_callback = object_z.callback
     def new_callback(obj):
     object_z.callback = new_callback

into this:


and an additional update function (which passes new values from the SignalProcessingChainCommunicator to the internal callbacks, i.e., what the current signal processing iteration is)

It cleaned up the noise in our signal processing chain, but it didn't remove the need to configure internal messaging of implementation details. It moved this to a new class.

Is this the only solution to this issue? Is there another way we could solve this problem of ever-growing analysis outputs that depend on implementation details?

  • I think that you should probably investigate DTrace and its ilk. The idea of such tools is that you run your code in an instrumented environment rather than create an instrumented variant of your code. It's a while since I had any hands-on experience of DTrace and it's not really something I could show you how to use in an answer here. But you might care to do some research until you get an answer. – High Performance Mark Nov 8 at 23:33
  • 1
    What language is this? – candied_orange Nov 9 at 4:40
  • This sounds like a use case for aspect oriented programming. I cannot tell you precisely if and how you can apply this to your case, but if I were in your shoes, I would definitely find an AOP framework and try to utilize it. – Doc Brown 2 days ago
  • @candied_orange I wasn't sure that the language was important, but it's python – opa 2 days ago
  • It is possible to log/publish everything and then just cherypick data you want afterwards? Or is the config toggling about not doing unecessary and expensive computation that is only used for logging and metrics? – Euphoric 2 days ago

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.