Flexible (dynamic) user input vs statically typed language c#

Question

Problem domain:

There are different types of financial products and I am trying to simulate them in different scenarios. Each of the product types requires different input data. Suppose the input data are organized with a database table, then the input data for a certain product may have its own total number of columns, and the columns may also have different business meaning compared with other types. In addition, for each product type, there can be a large number of actual product for the type (such as 500,000), and each product would be a data record in the table.

For example, input data for product type A may include fields "a_1", "a_2", "a_3"..."a_20". Input data for product B may include "b_1", "b_2", "b_3"..."b_25". It's not possible to clearly define and exhaust the input data structure for all product types in advance. A user may introduce new product types with new input data requirements.

While the input data vary in terms of the number of data elements and their business meanings, the data elements across product types can be abstracted into a few groups and each group can be handled in the same way, using common calc engine implemented using c#.

Users prepares an input data file for a product type. Users then separately use GUIs to tell the calc engine what groups of data are included in the input file. The calc engine uses the info from GUI to interprets the data from input the file and then perform the calculation.

I am trying to pass the dynamic, varying input data to the calc engine.

Attempted solutions:

Attempt 1:

Have a one-size-fits-all data structure for all product types. This will require a large large number of data fields to hopefully meet the needs of the most complex data type. In addition, I am also trying to use objects to represent the data records. With this approach, the object would also include unused data fields. Considering the large number of data records (and therefore objects)for a product type, this approach can be challenging.

This approach will also require generic field names, such as "field_1" and "field_2", instead of "a_1" and "a_2". The generic field names can be confusing and additional mapping to actual user field name may be necessary.

Attempt 2:

I was thinking maybe I can have a range of generic data structures, like "a-few-sizes-fit-all" instead of one-size-fits-all. Maybe I can have a data structure 1 to meet the needs of up to 20 data elements, a data structure 2 to meet the needs of up to 30 data elements, and so on. This may help clean the data table and reduce the size of objects. In this case, the calc engine will need to deal with different data classes, with the classes are still known at compile time.

Attempt 3:

Dynamically generate classes for the different product type, and have the calc engine deal with the classes generated at run time. This may have performance concerns. It also makes the calc engine more complex by asking it to deal with unknown data classes.

Question:

I wonder, if my thoughts (with the attempted solutions) about the input data structure and the use of objects to represent data records are reasonable. Is there a better solution? Thanks!

Answer 1

It generally depends on the requirements but, more often than not, it depends on how much time you have at your disposal, because when lots of conflicting requirements arise, you may have to actually begin implementing and build from there, improving your architecture as you go (which may include throwing code away).

First of all, remember, multiple databases with lots of tables and loads of records per table is nothing special for most good database implementations and good ones already have your back "covered" in that respect anyway, so this is not something you should worry about in advance.

Then, you should (usually) not design your architecture and reason about it on the basis of the implementation. You should design and reason on the basis of the abstraction. You are talking about (and consequently thinking of) classes and fields rather than the actual objects of your domain. I see you mention very specific numbers such as "20 data elements" and "30 data elements", but have you got them "down" yet, those specific 20-30 data elements (as an example)? Note down 20-30 data elements (less will also do) and check what they have in common for your use case. Financial products probably have a price, maybe a few other properties in common.

Remember, however, that while the representation of a domain may be almost completely detached from potential use cases, it's the use cases that have to be implemented. On many occasions, it can assist to conceptualize the relationships and clarify (to yourself) the true reasons why you are looking at it in that specific way. Then you can fine-tune your own reasoning by oscillating back-and-forth between domain logic and use case.

For example, financial products are, in a sense, interesting in terms of their "financial" properties, which should be common (e.g. price), then a name/description and, potentially, one or more depictions/images, so that one can manage a group of objects, facilitate purchasing and various other activities, all of which share the common ground of simply representing simple or complex operations on a database.

But take a look at your described use case. Rows and columns, lots of them, acting as a virtualization of the content of a database. Then, objects appear to have lots of properties. I will make the unpopular statement that, whatever your "calc engine", odds are extremely low that all these properties are relevant for that engine. If, for some reason, they are and you need them all, and they are also not common ground, then you should start refactoring your calc engine in the following simple way:

The way you imagine it right now, all your objects have specific types and you can use dynamic type inference or type checking to perform things in the calc engine. This is not too bad, it would serve you well to separate your objects in strong types and use some type checking in the calc engine than to play with field names to see if an object is of a given type, etc.

The way I would suggest, though, is to reconsider what it is that this calc engine does. If it calculates the price, for example, based on the properties, and your objects are wildly different in terms of characteristics (and, of course, have multiple, quite disparate, properties), you can simply put some "blood" into the object classes (try to look into anemic domain models and how they don't help much in the end) and give them functionality. Let them calculate the price on their own, so they implement a common interface, say IFinancial, which has a method GetPrice(). This way, the calculation takes place INSIDE the object and you do not need to type-check anything where the use-case code is organized. You just ask the objects.

Oh, and about these rows and columns that have to be filled with the rest of the (non-common?) properties, well, if there is nothing "financial" about them, then I do not see why you need these properties in your calc engine. If they are there just for reference, you can model them as key-value pairs. A Dictionary<string, string> will be all you need.

Few-sizes-fit-all solutions arise when either trying to model an incomplete domain, or trying to incompletely model a domain. They are better avoided of course, although I am no expert on that, plus the real world is too complicated to deal in absolutes!

Because developers like to see code:

public class FinancialObject : IFinancial
{
   //Common properties of IFinancial

   Currency GetPrice(); //Currency is some proper abstraction.

   //Maps property code-string (key) to property value.
   Dictionary<string, string> AdditionalProperties { get; }

   //Maps property code-string (key) to property user-friendly description/name.
   Dictionary<string, string> AdditionalPropertyDescriptiveNames { get; }
}