-2

Say I have a program using Java or C, that will interface with a hardware device and either read streaming data or write it to a database before applying a few complex mathematical equations to this data for analysis.

Where would the equations be applied (ie the data layer, application layer)? Is this possible in SQL, or does that depend on how complicated the equations are?

  • If this is done in the application layer is it done in just vanilla Java/C or do frameworks provide tools that can be leveraged?

  • If it's done in the application layer, how are large datasets handled for performance?

closed as too broad by Martin Maat, Doc Brown, gnat, kevin cline, Philip Kendall May 29 '18 at 6:03

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 2
    It depends on many things. Yes, there are libraries/modules for doing calculations and most application frameworks have one. SQL has some basic functions but whether you would want to use them in a particular scenario depends on design decisions, performance requirements, prior art and personal preference, to name but a few. – Martin Maat May 29 '18 at 5:30
3

Where would the equations be applied (ie the data layer, application layer)?

If you are developing a layered system, you will typically avoid to implement business logic in something like a data layer. "complex mathematical equations" sounds very much like business logic, so the data layer is typically not your first choice.

If your application has a classical three layered architecture (data, business logic, UI), then the business layer would probably the canonical place, assumed the core purpose of your application is to do the processing.

If, however, your application has several purposes, and the stream processing ist just one feature among many, why not introduce an "analysis layer" or a "processing layer"? Three layers is often a good start for an application, but there is no "law" or "dogma" for not introducing more layers if that makes sense.

Is this possible in SQL, or does that depend on how complicated the equations are?

SQL capabilities depend a lot on the specific SQL dialect of your database vendor, and if you include stored procedures under the term SQL, this is even more true. However, SQL and stored procedures often come to their limits when the requirements make it necessary to implement complex data types, bit-wise operations, or things like multithreading. But the first question you need to ask here is, where in your client/sever architecture do you want your processing layer or business layer to be placed?

If your whole processing layer is implemented on a client side (from the DB's point of view), it makes simply no sense to put in on the SQL/database side. If, however, you want to implement your processing layer or parts of it inside of some views or stored procedures, then make this part of your architecture's definition.

If this is done in the application layer is it done in just vanilla Java/C or do frameworks provide tools that can be leveraged?

You need to implement your processing steps if you cannot find a framework for it. If there are well-suited frameworks or tools depend on the individual requirements.

If it's done in the application layer, how are large datasets handled for performance?

This is quite unanswerable, since "large" and "performance" means different things to to different people, but, to give you a broad outline: by measuring the performance, optimizing where necessary, and/or scaling up using parallelization (more CPU cores, more machines, utilizing faster processors like GPUs). The right approach fully depends on the requirements of the task and cannot be answered in a sensible manner as long as there is a vacuum of unknown things.

Not the answer you're looking for? Browse other questions tagged or ask your own question.