If relational databases don't scale, nothing does. Don't worry about scaling problems.
SQL has problems with some sorts of analysis, but it doesn't take much data to trigger the problem. For example, consider a single table with a column that references other rows based on a unique key. Typically, this might be used to create a tree structure. You can write fast SQL statements that reference the related row. Or the related row's related row. In fact you can make any specific number of jumps. But if, for each row, you want to select a field on the first related row in the chain that meets some criterion, then it gets complicated.
Consider a table of office locations at nation, province/state, county, town, and village levels, with each office referencing the office it reports to. There is no guarantee that each office's reporting office is only one level up. For a selected set of offices, not all on one level, you want to list each one's associated national office. This requires loops of SQL statments and will take a long time even today. (I used to get 30 seconds on a selection of 30 offices, but that was a long time ago--and switching to stored procedures helped a bit.)
So the alternative is to put the whole structure into one big block of data, label it, and store it. When you want to analyze the data, read all of it into memory at one go, setting up pointers to track the structure, and you can process a couple million offices in the blink of an eye.
None of this has much to do with the amount of data. The key is the nature of the data's organization. If a relational layout helps, then a RDBMS is what you want. If not, some kind of bulk storage is going to be anything from slightly to a quadrillion times faster.
Note that if one of these sets of data becomes too big to fit into memory, your non-SQL database doesn't work any more. Another problem is when you need data from more than one block at a time; you can do this if, and only if, all the blocks fit in memory at once. And the user has to wait while you load them up.
If your relational database is going to cause you problems, it will do so before you've put much data into it. The only scaling problem you might have is with your program when the block of data you are assembling for a nosql DB--if you have to use one--becomes too big for it. (Do read up on out-of-memory errors. The newer languages sometimes do strange things with memory.)