I am currently facing a situation where I am not quite sure how to best proceed.


I am planning a new data import process, which will import personnel and salary information from a text file produced by a different system. We already have an import process for the data in question, but the format of the import file will change completely. Thus, while the data that is read will remain the same (employee names, pay grades, past salary payments...), it is expressed very differently in the new file format (different format, different fields, different coding of data, different representation of special cases such as unpaid leave or retroactive changes of salary).

How to reuse?

Now I would like to re-use as much of the existing import code as possible. The problem is that most of the existing classes and methods have parts that could be reused (mostly the parts that deal with writing the data into our own database, which has not changed), but also have many details of the old import format hardcoded, so direct reuse is impossible.

I now see two alternatives:

  1. try to refactor and rework the existing code to be more independent of the particular import format, so it can handle both old and new format
  2. cherry-pick the parts of the old code that still apply (extract them to separate methods or similar) and reuse that

My problem

I'm finding it difficult to judge which approach is best. On the one hand, I believe reuse is important, and 1) seems to allow me to reuse more. On the other hand, from looking through the code I have a gut feeling that at least 30%-50% of the code no longer apply, so I fear 1) might be a lot of refactoring work, and in the end I'd only reuse little more than with approach 2).

The problem is (I feel ) that I'll have to decide this upfront, because it's difficult to switch approach once I have started: If I start with 1), and after a few day of refactoring I find that little actual "meat" remains in the generalized functions, I have to throw away all the refactorings and start afresh with separate code that calls the old code as needed.

So how do you go about solving new problems when you have code that is broadly similar, but differs in many details to what you need? How do you judge which approach to code reuse makes more sense?

  • From what you describe, the current importer's quality in terms of design is severely lacking (dependencies, tight coupling, etc). Is the quality in terms of code good? For instance, has it been tested/used enough that you know it's robust? – jv42 May 27 '11 at 11:44
  • @jv42: Good point. Yes, the design has, let's say, evolved over time. I'm not perfectly happy with it, but I could not justify spending a lot of time improving it, because it works quite well. And yes, it has both unit and (automated) integration tests, and it has been in production with numerous clients for several years. – sleske May 27 '11 at 12:12
  • 1
    And BTW, I am the main author (though part was written by a less experienced version of me). Maybe that's why I feel I should reuse it... – sleske May 27 '11 at 12:13
  • This sounds like a job for Refactor Man. – Rein Henrichs May 27 '11 at 16:26
  • @Rein Henrichs: Excellent idea. Can you put me in touch with him? :-) – sleske May 28 '11 at 10:34

So you have an old import process, and need a new one, for a different data format. The old code is not decoupled from the exact input format.

My advice is to start from scratch, and copy-paste whatever parts are useful. Make sure your new code does decouple the input and output function, and has solid test cases for both. Then, next time you or your successor need a similar function it will be much easier to reuse the code. If possible, use the old code to create an input filter for the old format, testing your decoupling.

Code reuse is a great idea, but reusing bad decisions not so much.

| improve this answer | |
  • I really want to avoid copy-paste whereever possible. But factoring out what I can reuse into seperate methods, then using these should not be a problem. Otherwise, thanks for a well-reasoned argument. – sleske May 27 '11 at 12:14
  • As to "decouple the input and output function": I fear that is much easier said than done. The parts that are clearly separate are decoupled, but in many cases there is a "natural" coupling (e.g. the output function must produce special values for certain input data), and I don't see an easy way around this. – sleske May 27 '11 at 12:18
  • 1
    +1 What I was trying to hint at with my comment. If current design is bad, but code is good, redesign and copy whatever useful bits you have. – jv42 May 27 '11 at 12:22

To start answering the problem, this is what I believe your solution would look like. The system would have three functional parts. The core part would be a (reasonably abstract) in memory representation of the data. The second part would be the output logic that would be able to output your data to the database. The third (and most relevant in this case) would be the input logic.

The input logic reads the file and creates the in memory representation. This part is dependent on the input framework. There may be a parsing framework involved (if the format is complex enough to warrant that), which could be reused, but otherwise this part would need to be rewritten.

What you should do depends on your existing code. If it is a big spaghetti that is almost impossible to refactor into such a framework (possibly with parsing utility code), then you would probably have to do a rewrite and cherry pick what is salvageable. If refactoring could work, do that and as added benefit your program can now handle both input formats at the same time, and in a way that makes it fairly straightforward to add another format.

| improve this answer | |
  • As I explained in another comment: The two formats are so dissimilar that I find it hard to come up with an internal representation that would work for both. The only representation that would work would be our internal representation (i.e. the target of the import), which is what the code is already doing. Otherwise, a good breakdown of a possible structure for the code. – sleske May 28 '11 at 10:55
  • If they are that dissimilar, probably you would indeed have to rewrite the entire parser part. Depending on the output format, you could reuse an in memory representation of your output (if it isn't worth it you wouldn't) – Paul de Vrieze Jun 10 '11 at 10:40
  • Yes, that's effectivefly what I did. I did reuse various helper classes that let me store an manipulate data in the internal format - that was the "cherry-picking" solution I mentioned in my question. – sleske Jun 10 '11 at 15:28

My advice is don't go overboard with code reuse. Sometimes (more often than you think) you think that you should reuse the code in a new place, but in reality it is only here and now that the code you need is the same as the code you have available in the older part of the system. If two components are logically coupled and will develop together (say, different variants of the same business problem), it is usually fine to do some effort to reuse code. If they only accidentally look similar, belong to two different problem domains and may develop along different paths in the future, coupling them via code reuse is going to bring more problems than benefits. In this situation, do not hesitate to Ctrl+C, Ctrl+V.

| improve this answer | |
  • +1 for "logically coupled". Yes, I feel that much of the similarity is more incidental, rather than being caused by some shared characteristic. – sleske May 27 '11 at 12:19
  • 1
    Still, I don't think copy-paste is ever acceptable for non-trivial parts of code, so I'd still factour out common code to reuse it. The exception might be if the code needs to change significantly to allow reuse; then copy-paste might be acceptable as a starting point. – sleske May 27 '11 at 12:21

Why don't you just write something to reformat the new data into the existing input format that you know already works?

| improve this answer | |
  • Yes, I considered this, but the new format is much more flexible (and more logical) than the old format, so doing this would mean all kinds of contortions, and I'd have to throw away / simplify some data. I feel this would be a bad solution because it carries the limitations of the old format over to the import of the new format. Plus there is often no obvious one-to-one mapping between the old and the new format, so I'd probably introduce additional errors. – sleske May 27 '11 at 14:33

I'm going to say go with option #1, but approach it as if it's option #2. You write a completely new system from scratch, and copy the good parts of your old code base. As someone already mentioned, make the input/output format pluggable. Then write your new input/output format.

And then, write an input/output format for your OLD format. Voila. You have option #1 complete. The nice thing about doing it this way is that you can skip the last step (porting old data format) if you're stretched for time.

| improve this answer | |
  • Interesting approach. However, I don't think "porting over" the reading of the old format into the new import code is worthwile. The old format is unlikely to ever change significantly (is is kind of deprecated already), so putting so much effort into it seems questionable. Of course, it might make sense if both interfaces were expected to evolve further. – sleske May 27 '11 at 14:57
  • I think it makes sense from the perspective that you can throw the old code away and never look at it again. I'm always surprised at how often bugs pop up in deprecated and unlikely to change code. – Kevin May 27 '11 at 22:58

I do recommend refactoring the code. You essentially have two input formats you need to deal with, and change having a habit of being constant, there will likely be new formats in the future. There's a couple ways of organizing the target code.

  1. Refactor and move the business logic into a set of utility functions. That keeps the reusable bits available for both file formats, and depending on how simple/hard those reusable bits are can be the quickest solution. This works best if each function operates directly on the data provided to it. If you need to reference data in another part of the file, you may need to look at option 2.

  2. Refactor to separate parsing and consuming data. In other words, have an internal representation of the data that isn't going to change unless you have new requirements. That can have the reusable bits in it, so you can deal with the more complex logic that rely on data in different parts of the object hierarchy. The two inputs merely consume the raw format and convert it to the internal representation. To support new formats only requires adding a new front end parser.

NOTE: only refactor what you will need for the new input format. That minimizes the refactoring overhead, and lets you focus on what's important for the new format.

| improve this answer | |
  • In this case further changes seems unlikely; it's the first fundamental change of format in 20 years or so (it's from a government agency). Otherwise, good point. – sleske May 28 '11 at 10:37
  • As to "have an internal representation of the data": Also a good idea (like mentioned by Paul de Vrieze), only the two formats are so dissimilar that I find it hard to come up with an internal representation that would work for both. The only representation that would work would be our internal representation (i.e. the target of the import), which is what the code is already doing. – sleske May 28 '11 at 10:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.