Where would the equations be applied (ie the data layer, application layer)?
If you are developing a layered system, you will typically avoid to implement business logic in something like a data layer. "complex mathematical equations" sounds very much like business logic, so the data layer is typically not your first choice.
If your application has a classical three layered architecture (data, business logic, UI), then the business layer would probably the canonical place, assumed the core purpose of your application is to do the processing.
If, however, your application has several purposes, and the stream processing ist just one feature among many, why not introduce an "analysis layer" or a "processing layer"? Three layers is often a good start for an application, but there is no "law" or "dogma" for not introducing more layers if that makes sense.
Is this possible in SQL, or does that depend on how complicated the equations are?
SQL capabilities depend a lot on the specific SQL dialect of your database vendor, and if you include stored procedures under the term SQL, this is even more true. However, SQL and stored procedures often come to their limits when the requirements make it necessary to implement complex data types, bit-wise operations, or things like multithreading. But the first question you need to ask here is, where in your client/sever architecture do you want your processing layer or business layer to be placed?
If your whole processing layer is implemented on a client side (from the DB's point of view), it makes simply no sense to put in on the SQL/database side. If, however, you want to implement your processing layer or parts of it inside of some views or stored procedures, then make this part of your architecture's definition.
If this is done in the application layer is it done in just vanilla Java/C or do frameworks provide tools that can be leveraged?
You need to implement your processing steps if you cannot find a framework for it. If there are well-suited frameworks or tools depend on the individual requirements.
If it's done in the application layer, how are large datasets handled for performance?
This is quite unanswerable, since "large" and "performance" means different things to to different people, but, to give you a broad outline: by measuring the performance, optimizing where necessary, and/or scaling up using parallelization (more CPU cores, more machines, utilizing faster processors like GPUs). The right approach fully depends on the requirements of the task and cannot be answered in a sensible manner as long as there is a vacuum of unknown things.