3

I have a number of data servers that are pushing data to a main server as in this kind of architecture:

ServerDataArchitecture

These data points (points for short) are updated randomly from each server. On the server, there are other user-defined data points (called Tags) that are expressions based on one or more points. The server has potentially hundreds of thousands of these Tags.

For example:

Points = {DataPointA, DataPointB, DataPointC}
Tags = {DataPointA * 1.8 + 32, DataPointA * DataPointC / 2, DataPointC * DataPointC}

The server back-end is written in C# and I've decided to use JavaScript to allow the user to define the expressions that make up the Tag values (mainly because the other part of this is a web-based visualization and I only want one scripting language).

Problem

I don't want to create a single JavaScript Engine for each Tag since that would be memory intensive. I've selected JInt for the JavaScript Engine (has to be mono-compatible so V8 is out) and I can create variables (Tags) and assign values (Points) but its not thread-safe. So I would have to wait until one Tag is finished updating before starting another. This is a performance bottleneck.

So I thought about creating a pool of Engines and pass out engines as they are available, the Tag locks it while using it and then releases it when it is finished. I'm not sure this is the best approach though because during my testing I've found the single Engine to be about 25% faster (or maybe its my implementation).

How would you approach this problem? I can't delegate the calculations down to the data servers because the expressions may use tags from multiple servers and that would require the servers to talk to each other or a bi-directional exchange with a repository on the main server.

Here is some code if you'd like to look through it:

Minimal Example with No Dependencies

Minimal Example with JInt Requirement

Note about the code: There is a level of indirection here with the use of Variable, this is because the data point "names" are actually GUID's, and a GUID doesn't make a good (or valid) variable name, so they have to be aliased into something that is a valid variable name.

1
+50

I would go a bit of different route here:

  1. Setup a Node.js based service on the mainframe.
  2. Setup queuing system. (0MQ, RabbitMQ)
  3. Use your Node.js to read from the queue and process.
  4. Return the results to your main process through the queue as well.

Fully asynchronous and independent system.

Advantages: Node.js can handle millions of connections simultaniously. It's single threaded but asynchronious.

To take it a level up, spawn as many instances of of Node.js as you need and put them behind a load balancer (I doubt you will need that though).

2
  • Its an interesting concept and I might look into it, but at the moment there is a lot of other architecture that is already written in C# and I'd hate to have to redo it. – Ron Beyer Jun 26 '15 at 2:00
  • You don't have to redo it, you can use c# and Rabbit MQ to queue. The point of my response is , queue is a good alternative regardless of the language. It's just Node will be asynch but thread safe. – Alexus Jun 26 '15 at 16:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.