Parsing Text Tokens in Script Files

Question

My team is wanting to migrate away from compiled code to dynamic code. We have a wide variety of customers and each has different business rules. Instead of having a single compiled code base, we are considering having different templated script files for each customer. However we have a lot of compiled code, so for now we are wanting to keep most of our current code and shell the script files like ruby my.rb or node my.js.

In addition, we need to get data into the scripts. So, we are thinking we will have variable declarations in the script, and those variables will be set to some character sequence that we will treat as a token. Then with another app, we'll process the document looking for the tokens replacing them with the actual data.

I have some concerns and am not sure what to think of all this, but I'm wondering if someone with more experience can shed some light on the quality of this solution and what some other alternatives might be.

EDIT: I hope this helps explain my scenario.

Some of the motivation of this is our clients are not always accessible. In addition, we have services running on the client that would need to be stopped to install new versions of .dlls or .jars. Also, our clients get fragmented on versions of software despite attempts to keep them together. The thought is the script files will ship on demand with the other files and data to which that script is valid, and the compiled software will change less often.

Answer 1

We have a wide variety of customers and each has different business rules. Instead of having a single compiled code base, we are considering having different templated script files for each customer.

I'd say that the premise for this is kind of flawed. You are after some sort of modular coding. This can be done with other languages too. Define the modules, create a Factory that serves the one you want and use that. It is probably a lot simpler than trying to do multiple temptation of the source itself.

In addition, we need to get data into the scripts. So, we are thinking we will have variable declarations in the script, and those variables will be set to some character sequence that we will treat as a token. Then with another app, we'll process the document looking for the tokens replacing them with the actual data.

This is where things get ugly.

You've got some data in some file, and some preprocessor (lets pretend its gnu m4 - if you do find yourself following down this path you might as well use a good one - the associated SO tag m4) which reads the file, identifies the template macros and modifies the source, which is then deployed out to the customer.

So now you've got something like

site = "_SITE_"
print site, "\n"

as some ruby code and then a macro file

define(`_SITE_',`programmers.SE')

and then you run it through the m4 application

$ m4 main.m4 main.rb

site = "programmers.SE"
print site, "\n"

Now you start getting a lot of moving parts in there. You've got to make sure that each file parses correctly, that some site didn't include something like a " which would be valid macro, but an invalid variable (will you catch that before it goes to the client site?).

Most of all, you had better hope that the code doesn't diverge. Once you've got something more than just running a macro against the code, but replacing the code it gets ugly. You've got to be very strict with naming conventions so that the values used in one part of the templates class don't collide with the other ones... and that you don't even use the values from one part of the template in another one (you pull just that part of the template in without the template part that sets up the data). It just gets UGLY.

Worst of all, and the thing that will burn you in the end with this, you cannot unit test the code you write. You can test the code that is generated (will you? really?), but you won't be able to actually test the code you write is doing what it should.

The fundamental flaw in this thought is mixing the data and the code. Essentially, the code you are writing in this model is data for a macro processor that spits out new code.

If you want template variables, read those out of a property file of some sort.

Store the data as data and load it as data. For example store it out in a JSON object and load it in to an object with the appropriate library (ruby python).

Then you're just deploying the different data to the different sites (with all the same code base) that indicate the values that are different between them. You will use the values in the data to call the proper factory method (that works fine on its own without any templates and can be unit tested).

Answer 2

Well the overall quality is, um, not good.

The things that seem most clearly wrong (since almost everything that is usually wrong has the odd case where it makes sense) are the idea of splitting into multiple code-bases and the idea of passing data into a script by putting in meta-tokens and then replacing those with data literals.

The multiple code-base issue is far more serious. It will probably work great for a short time, then steadily turn into a disaster as issues common to multiple customers have to be solved by making the same change in multiple separate code-bases. As they drift apart, the same logical change will more often become a different code change, so it will no longer be a tedious cut and paste, but more and more like writing the code multiple times - with attendant opportunities for multiple bugs. You may well end up having to put the code back into a single code-base, which will be far harder than it was to split them. At that point, more likely you won't, which will be worse.

If somehow you change code so infrequently that you never reach the point where this approach has more cost than benefit then I guess it could make sense. It's a bit hard to imagine a plausible scenario where it's a good idea though - perhaps if the code really is pretty close to unique for each customer and the rest of the code is shared among all the customers.

Having a lot of special behavior on a customer by customer basis is very common. The usual solution is to factor out as much common behavior as you can (this is where engineering skill comes in, but even bad engineering will probably beat the separate code-base design). If all else fails you can break up the customer specific behavior into logical pieces, and have a configuration flag for each piece, rather than testing the customer ID in the code (i.e., use feature detection, not browser sniffing). Then when a new customer wants some of those features you don't have to dig through the code to pull out features.

As far as idea of compiling values into common scripts - it's strange and seems harder than it needs to be for no real benefit, but even at worst it won't ruin your life. Generally passing variables via command line arguments, or putting them in a configuration file (possibly a configuration database) is how people would do this.

As far as the issue of stopping a service - interpreted code may help, but have you thought about what happens if files are swapped out while a process is running? If file Q has dependencies on N, O, and P, and while Q is executing an update swaps in new versions of N, O, and P, will your code work properly? Will you QA this? How do you plan to do that? If incorrect program behavior isn't a big issue this may be acceptable, but if incorrectness has serious consequences you may well end up implementing the equivalent of taking down a service running compiled code. Which kills your original motivation for using interpreted code in the first place.

Edit - I feel I should mention that multiple code-bases are acceptable in some sense - loosely coupled modules working together to make an application where each module could do something useful even in isolation is quite O.K. Copying the same code among multiple code-bases then letting them evolve individually is very rarely a good idea.

Answer 3

The problems you cite have nothing to do with complied code versus scripts. It has more to do with software design. If they think that multiple code bases will solve their problems, they and the organization would be better off if you let them go so they could find employment elsewhere and learn from a professional. Your organization should hire at least one person who can mentor and understands issues.

It is hard for me to enumerate the reasons why this idea is so bad, it is kind of like explaining why 2+2 = 4.

If I were in charge, at your place I would work towards developing a design that is highly cohesive and loosely coupled. There would be modules that were written such that they could be used by all customers. There would be some modules that would be used by only some customers. Some code would be only used by one customer.

These various parts could be assembled at compile time by some smart build/deployment scripts or by configuration files.

It is good to know that your senses are correct, however, it is a really crummy situation that you are in. Frankly you have "heavy weights" that want to play Russian roulette with 6 bullets in the chamber.

And yes, I could come up with something better. I have done so many times in my career.