I'm wondering how to design a fairly simple class whose properties are complex to compute. Also, the properties depend upon each other for computation.
An example using graphs and graph processing (think nodes and edges, not charts or scatterplots) to motivate the problem:
CommunityGraphSet is a collection of
CommunityGraphSet is initialized with an input graph (some large social network graph), along with some basic parameters describing what is known about the graph. A
Community is a subgraph of the input graph, along with some structural descriptors.
After initialization, the
CommunityGraphSet contains no
Community subgraphs. After several major processing steps, it contains a set of
Community subgraphs. Each major processing step (i.e. function call) triggers a GUI for a human to tune the parameters of the algorithm being run in that step. The human sets parameters, the processing results are visualized, and they continue to refine the parameters until the results are acceptable. After they "accept the parameters" with a button click, the results they saw are returned by the method that triggered the GUI.
input_graph = read("graph_file.csv") number_communities = 3 sparseness_thresh = (0.05, 0.10) # MAJOR PROCESSING STEP # get parameters for key community member identification based on overall graph structure key_ident_params = compute_key_identification_params(input_graph) # initialize the graph set graph_set = CommunityGraphSet(input_graph, number_communities, sparseness_thresh) # MAJOR PROCESSING STEP # mark where community subgraphs definitely are not. partition_mask = isolate_potential_communities(graph_set) # use mask to extract subgraphs for refinement by different algorithms. potential_community_subgraphs = graph_set.extract_subgraphs(partition_mask) for community in community_list: # MAJOR PROCESSING STEP # key community member extraction key_members = extract_key_members(community, key_ident_params) # MAJOR PROCESSING STEP # use identified key members and community sparseness to trim community to final subgraph final_community = refine_community(community, key_members, sparseness_thresh) # put the refined community in the final set graph_set.add_community(final_community) save(graph_set.serialize(), "communities.json")
My problems with this design:
CommunityGraphSetis stateful. It contains
Communitysubgraphs only after the right function calls in the right order.
- The processing pipeline is monolithic. As the processing pipeline grows and changes, maintaining it will become unweildy as I'm faced by a growing number of variables used in increasingly many places.
Are these indeed problems? Or are they inherent in data processing pipelines?
If they are problems, how can I solve them?
- None of
CommunityGraphSetis not coupled to GUI logic, it is coupled to an
IterativeOptimizerinterface for each step. The described GUI or a clever optimization algorithm could implement that interface.