Suppose you have an application that reads an input file, on a language that does not support the concept of None. The input is read, parsed, and the contents are stored on a structure for later use.

Now, in general you want to keep into account transformation of the data from the input, such as adding default values when not specified, or adding full path information to relative path specified in the input. There are two different strategies to achieve this.

The first strategy is to perform these transformations at input file reading time. In practice, you put all the intelligence into the input parser, and your application has no logic to deal with unexpected circumstances, such as an unspecified value. You lose the information of what was specified and what wasn't, but you gain in black-boxing the details. Your "running code" needs that information in any case and in a proper form, and is not concerned if it's the default or a user-specified information.

The second strategy is to have the file reader a real one-to-one mapper from the file to a memory-stored object, with no intelligent behavior. unspecified values are not filled (which may however be a problem in languages not supporting None) and data is stored verbatim from the file. The intelligence for recovery must now go into the "running code", which must check what was specified in the file, eventually fall back to a default, or modify the input properly before using it.

I would like to know your opinion on these two approaches, and in particular which one you found the most frequently implemented.


This is, of course, a false dichotomy. The real answer is not this simple.

Also the presence or absence of None or Null isn't interesting (or even relevant).

The first strategy is to perform these transformations at input file reading time.

In the long run, this becomes a maintenance nightmare. "black-boxing the details" is a bad policy.

The second strategy is to have the file reader a real one-to-one mapper from the file to a memory-stored object, with no intelligent behavior.

That's a very weird way to put it, but that's one small part of a good architecture.

None. None or Null is what you can call a domain independent null -- a universal nothing value. Some languages (like C) lack a domain independent null. That doesn't mean anything, because you have at least two ways forward.

  1. Domain-specific null. Input values like zero-length fields, or 999-99-9999 for a Social Security Number and stuff like that are domain-specific null values. You may have this kind of thing.

  2. Flags. The embedded SQL folks decided that the best way to handle SQL NULL in languages which don't have a domain-independent NULL is to provide an array of flags as null indicators. Applications in C (or COBOL or even FORTRAN) would have to check the null indicator for the presence of a SQL NULL. The actual field could have any random junk in it if the indicator said it was really a NULL.

Transformation. For information on this, read up on ETL (Extract Transform Load) processing.

Applications that process files naturally form a pipeline.

  1. Input reading. This byte-to-character processing. The least you can do to read. If you have Unicode decoders, they go here. Nothing more.

  2. Some kind of lexical scanning -- the basis for syntax recognition. If it's a CSV file, this is the comma or tab recognition.

  3. Some kind of file object parsing. This creates Domain Objects based on the file representation. For CSV or tab-delimited files, it creates a list of row values.

    For XML files, 1, 2 and 3 are combined into an XML parser.

  4. Some kind of application object parsing. For CSV files, this is where you'd transform a sequence of objects into something more usable by your application.

    For JSON, YAML or files, this creates the objects. Often, 2, 3 and 4 are all done by the load operation.

At this point you have an object in your application domain. Processing from this point forward is independent of the initial input format. In the applications I support, all of the previous processing is a separate library that handles XLS, XLSX, CSV and a few other formats for the inputs.

Once you have a application domain object, you then have to apply your application's rules. This varies a lot, of course. But here's the strategy.

  1. Write down the "outer" processing loop. for object in source(): process( object ). Maybe it involves decision making. for object in source(): if valid( object): process( object ). Whatever the high-level, end-user, easy-to-understand summary is.

  2. Odds are good that none of the input objects are actually suitable for this kind of processing. You have things like cleansing, filtering, validation, derived data calculation, foreign-key reference lookups and the final stage of persistence (update/insert/delete) processing. This forms a natural pipeline.

    Define this pipeline from "back to the front".

    What must be true of the objects so they can fit with your ideal outer loop?

    Write down all the conditions that must be true. (cleaned this way and that way, filtered for this that and the other, calculations done for this and that, etc., etc.)

    What's the last thing that has to be done to prepare for the persistence step?

    What's the last thing that has to be done to prepare for that?


    Until you've run out of "last thing to be done to prepare" and you're looking at the input objects and a pipeline of steps to prepare them for their final persistence.

  3. For each stage in the pipeline, design the simplest transformation function you can get away with to do just that operation on each input object, cleaning, filtering, validating, deriving data, doing lookups and finally doing whatever persistence thing must be done.

    Each of these stages will have to make use of the null indicators, BTW.

    Some of these will be generic and reusable. Some won't be. Don't think about reuse. Discover reuse after you've written the same code 3 or more times.

Object Updates. There are at least two ways to handle the transformations.

  1. Update the objects as you go. This is classic "OO" design. Stateful objects.

  2. Create new objects from old objects. This is more of the functional programming design. Immutable objects. In some languages, this requires more memory management. On the other hand, if each object is immutable, the pipeline becomes simpler because each stage expects a specific data type and creates an object of a different specific data type.


Proper encapsulation of the problem would suggest that neither solution you presented is correct. An alternative would be to follow choice 2 up to the point that you litter the code with data domain transformations. This should be factored out as a post processing step to be applied after the file is read.

This separation provides a clean way to revise the implementation domain data mappings.

Not the answer you're looking for? Browse other questions tagged or ask your own question.