4

My question:

When you've got a complex converter like, that takes chunks or large result sets out of a database, converts it into a line by line file/resource in the end, should one either design it as std::basic_streambuf or should it be a std::basic_istream or something different, using C++?!

To better describe it look at the diagram below or use this text data flow:

A(ODB) -[maybe many rows]-> COSTLY CONVERSION  -[many text lines]->  ????  --> std::istream
                                                                     ????  --> std::ofstream
                                                                     ????  --> std::ostream

Note: The data volume may be large, from MB up to GB, so data could be converted in chunks and lazy from the database, which will lead to a chunk of text lines. I my assumed approach at the moment for ???? was std::basic_streambuf

My assumption:

AFAIK the foreseen way to stuff data from one std::istream source into another std::ostream sink, like std::ofstream is to just use the <<operator with sources std::streambuf argument, like

sink << source.rdbuf();

Background

I'm designing a C++ converter module, that takes custom data (i.e. measurement data), in this example out of a database mddb via the C++ library odb, converts it into another format, called df here, and now comes the important part: Makes that converted data available for at least an std::istream and also an std::fstream. This is because, the std::istream can be used for a HTTP download via Wts Wt::WStreamRessource interface and the file stream obviously, to just store the data as a file.

class diagram

7
  • Is the data, that you want to transfer from ???? to your ostream formatted or unformatted?
    – iolo
    Commented Feb 5, 2016 at 15:48
  • I'm not sure how to apply the C++ concepts of formatted input here, but it's data which white spaces should be preserved. Commented Feb 5, 2016 at 15:51
  • I am not sure i understand the question... Do you want to design an object that gives line-by-line data and under the hood extracts and converts stuff from a db?
    – iolo
    Commented Feb 5, 2016 at 15:58
  • Still your question is unclear! You seem to want to construct a resource (istream-like) and at the same time do something with output (fstream/file)?
    – iolo
    Commented Feb 5, 2016 at 16:06
  • First: It doesen't has to be just one class, I could imagine adapter classes to an intern conversion class. This module/classes should take data from a db, convert it, and then make this data usable to be saved as a local file (so stuff it into an std::ofstream) and also to be saved remote. Saved remote means for example, as an HTTP download, a library called Wt does this via the WRessourceClass that wants to access the data via an std::istream to deliver it nicely via http/tcp. Commented Feb 5, 2016 at 16:10

1 Answer 1

1

First of all, a std::basic_streambuf is basically the building block of a std::XYZstream. Using one or the other is not choosing between two implementations, it is a choice between a basic building block and a more advanced stream interface that people expect. Use a stream.

Based on your comments, it appears that you may be getting hung up on the idea of buffering.

Try to think of this as something similar to reading lines from a file, except you are reading records from a database. Instead of reading a file line by line, you are pulling data from a table record by record:

idbstream in(<DB connection info here>);
ostream out(...);
while (in) {
  db_record r;
  in >> r;
  out << r;
}

In your extraction idbstream::operator>>(db_record&) you handle your query logic, buffering enough input to be able to construct a DB record object, whatever you need. You only return once you have a full DB record and have updated that object. You could also forgo the syntactic sugar of operator>> entirely and do a move construction and return inside your stream:

db_record idbstream::readRecord() { ... }

out << in.readRecord();

For efficiency, db_record should have a move constructor and it should be created at the same time it is returned.

Finally, the details of how the DB stream performs the conversion are deliberately left unsaid. You could use an adapter, or some more complex logic to select the type of object and convert multiple record types.

Also note that the output stream is just that, an output stream. I make no assumptions about what is on the other end of that stream. As you allude to in your comments it could be a file, standard out, an HTTP connection, whatever you need. The DB record class itself would handle its serialization in an implementation-agnostic manner (the only thing that really matters is binary v. text).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.