I have been searching about this topic for a few days and have not yet found anything on books, courses or tutorials.
What is a way to make data pipelines more scalable, that doesn't involve NoSql or major investments like hadoop clusters?
Most of our pipelines are currently made with Python. They're pretty simple in nature: The application connects to a SQL database, fetches raw data into a dataframe, transforms it, then feeds it to a production application.
My question is: where in this design is scalability defined? Or rather, if we scale up the volume of data that goes in and out, which part of this design would start giving issues? I've read about how a big reason why NoSql is more efficient than SQL at scaling is that SQL constraints take a lot of resources to enforce.
So, in my example, would this already be scalable? since SQL only participates in the raw data ingestion process, while computations are being performed outside of SQL through python/pandas (so to my knowledge, no constraints are being checked or enforced) then finally the results are fed to another app.