Today we are using SQL server with multiple indexed views. Whenever we update the source tables for the view there is too long delay.
I have no experience with Spark, so the question is: Can we input the data from the source tables, create the "data view" in spark and then only select the data we need from it? but still keep the "data view" ready for reading later (not saved to disk, but as long as the spark server is running)?
very simplified SQL example of data we need (not actually code):
SELECT tbl1.id as id1,tbl2.id as id,tbl3.id as id3 from tbl1 Cross join tbl2 Cross join tbl3
So we add data from tbl1, tbl2 and tbl3 to spark, run the data transformation and then we can select data from spark like
SELECT * FROM SPARK where id1 = 74 and id2 = 85 and id3 > 45 and id3 < 90
(the question is about spark, not tuning SQL queries and sql indexes)