I'm designing a new application that is very basic however I expect it to grow and don't want to have to refactor down the line. My challenge is that in one method, I either need to retrieve thousands (e.g. 20,000) of records from the database and iterate through them in memory or, make multiple calls in a loop (e.g. 200 requests for 100 records each).
I saw this similar question that leans on the "the less database calls, the better" line of thinking. Is there an upper limit to how many items one should / should not iterate through in a Windows Application? Could this approach be prone to memory issues that may negate the performance benefits?
This is not limited to a specific scenario and I've had this thought before and have chosen the latter option (multiple requests for smaller datasets).
Am I overthinking this and should I just get to it?