I am creating Qt application that uses database with huge amount of data to draw some charts. Fetching data from the database is time consuming, so is blocking the application thread or worker thread creating an unpleasant delay.

I have an idea that instead of doing everything at once I can fetch data and draw charts in chunks. How would I design this functionality?

I have no idea how can I design my interfaces. I want also want there to be a place for other database stuff, like caching, in clean way.

  • Does your database layer allow you to retrieve in chunks? I know e.g. JDBC is built around this idea: the DB abstraction layer can sometimes start to retrieve data before the full data set is returned to it. – user22815 Nov 8 '15 at 17:37
  • Does QT allow data to be displayed in chunks? I know some graphing libraries are built around this idea: the graphing surface can sometimes partially render data as it is received. – Robert Harvey Nov 8 '15 at 17:42
  • Snowman: I have class that retrieve data from database directly. On top of that I want to build layer that retrieve data in chunks and send signals "hey, catch that new chunk", "hey, its done". I am looking for design pattern for such thing because I want to avoid new FooManager class. Robert Harvey: I am using QCustomPlot which is fast, so replotting when new data arrives is no problem. – Mateusz Drost Nov 8 '15 at 20:34
  • 1
    You're describing asynchronous function calls. Depending on the programming language, you can implement them using callbacks or continuations. – Robert Harvey Nov 8 '15 at 22:01

Have a look at the Batch Iterator (or Chunky Iterator) Pattern from POSA5 (chapter 6.3). There is also a book if you want more detail.

Assuming you understand the Iterator pattern, the batch iterator does the chunking based on the size of the data structure it uses (the iterator's client is unaware of how chunking occurs).

  • I think the OP's client process is blocking. If the client process is unaware that the data retrieval is batched, I think it will block, waiting for data. Does there need to be a modification to the Batch Iterator design pattern? – dcorking Nov 9 '15 at 9:42

It sounds like you need a thread pool of workers to which you can pass selection criteria (for the database call) and a reference to the chart area. Then you can have each worker query the database for a subset of the data, and have it plot its portion of the graph. You may have to mediate access to the graph area (I have no idea how QT is structured), but while the workers are busy the main thread can continue to serve user requests.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.