0

This is not (supposed to be) an opinion question, but it is a newbie question, so if there is just a resource I haven't found that I need to read, point me there :)

I am in the design stages of a microservice-based ETL system. One service in particular seems to be a natural candidate for functional programming over oop, but I didn't want to be a solution in search of a problem.

The service would receive a complex data object in the form of a GraphFragment (think json)

[
   {
      "type":"node",
      "label":"Product",
      "properties": {}
   },
   {
      "type":"node",
      "label":"Factor",
      "properties": {},
   },
   {
      "type":"Edge",
      "label":"INCLUDES_FACTOR",
      "properties": {},
      "from": 0, // Product
      "to": 1, // Factor
   },
]

Except much more complex with hundreds (maybe thousands, but no more) of nodes and edges, each with many properties.

The microservice, then, interrogates that graph fragment and produces a report (also in a json-like object). In that report would be two different "kinds" of fields:

  1. Attributes (like a list of all components of that product by traversing the graph to a depth of n)
  2. Calculated scores based on a complex formula that takes into account the depth of traversal, properties, and weights.

The service itself has no responsibility to write to anything until the end, when it puts the final product onto a kafka queue. It should not need anything other than the graph fragment.

It seems like a good choice for a pipeline with a collection of functions that each add an attribute to the report object. Each function would receive the entire graph, and should be blind to the output of the previous and future functions.

Some other requirements would be fast and resource efficient. This specific process will be used far more than the others and must be as performant as possible. Running multiple functions in parallel is attractive.

So, I guess I have two questions:

  1. Is this a good use case for functional programming, maybe in Kotlin or Scala (or Python), which are the three languages most used in the application as a whole.

  2. If the data structures are immutable, how does one function add an attribute to the report? Is it handed the current state of the report and then returns a copy of that report plus the new attribute.

Why does that sound memory wasteful to me?

I appreciate any thoughts and would be happy to add clarification.

  • Well, these are some great answers. I recognize, now, that this should have been two different questions. One about the use of functional programming and one about how to add to the state of the report. The answers given touch on those separate questions. – Apollo Jul 19 '17 at 22:42
0

Is this a good use case for functional programming, maybe in Kotlin or Scala (or Python), which are the three languages most used in the application as a whole.

Naively, it seems fine. Walking a graph collecting state seems decidedly functional (even better if you could map/reduce the nodes rather than walking the graph serially).

If the data structures are immutable, how does one function add an attribute to the report? Is it handed the current state of the report and then returns a copy of that report plus the new attribute.

Possibly. A better solution (if you actually need to collect all of the results before writing them to the queue) may be returning a linked list sort of object that is the new attribute and a pointer to the "previous" state of the report. Once the processing is finished, it would return the last attribute, which points to the previous, which points to the previous, and so-on.

  • I like the idea of a linked list, but part of what excited me about the functional programming (other than learning something I've been wanting to learn) was running these functions in parallel. I realize, now, that my original thought would not work in parallel either. – Apollo Jul 19 '17 at 22:46
0

If the data structures are immutable, how does one function add an attribute to the report? Is it handed the current state of the report and then returns a copy of that report plus the new attribute.

In a functional language, you can return a function that modifies your report, and fold the report over the functions returned (Scala):

type ReportModifier = Report => Report
type PipelineFunction = GraphFragment => ReportModifier

def executePipelineFunctionsInParallel(
    report: Report, 
    pipelineFunctions: List[PipelineFunction], 
    graphFragment: GraphFragment
): Future[Report] = {
    // Run our pipeline functions in parallel
    // Collect their results into Future[List[ReportModifier]]
    Future.traverse(pipelineFunctions)(pipelineFunction => Future {
        pipelineFunction(graphFragment)
    }).map(reportModifiers => reportModifiers.foldLeft(report) {
        case (report, reportModifier) => reportModifier(report)
    })
}
0

It seems like a good choice for a pipeline with a collection of functions that each add an attribute to the report object. Each function would receive the entire graph, and should be blind to the output of the previous and future functions.

Don't add attributes one at a time. Use a map-like operation to create the final dictionary. Something like this in python:

# start with a dictionary mapping the attribute to the function
# that should produce that attribute
functions_to_apply = {
   'foobar': foobar_function,
   'goat': goat_function
}

# send the function and graph parameter to an executor who will run
# the operation in parallel and return a future.
with_futures = {key: executor.submit(value, graph) for key, value in functions_to_apply.items()}

# resolve all of the future to get your final object
final_report = {key: value.get() for key, value in with_futures.items()}
  • Forgive my ignorance, but this seems like an oop approach and less a functional programming approach. I say that because of the for loop. However, in python, I guess this would be functional. – Apollo Jul 19 '17 at 22:44
  • @Apollo, python took the comprehension idea from haskell: wiki.haskell.org/List_comprehension. So I think it qualifies as functional. But without comprehensions you'd call some sort of map function to do the same thing. – Winston Ewert Jul 19 '17 at 22:53
  • Good point. I'm still learning, and not at all hell-bent on doing something super pure or academic. Random question, have you played with github.com/evhub/coconut for functional programming in python? – Apollo Jul 19 '17 at 22:58
  • @Apollo, no I haven't. I generally shy away from hybrids like that because I think they end up being the worst of both worlds. – Winston Ewert Jul 20 '17 at 14:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.