1

I've recently joined a team working on radar observations. The team has access to a repository of software (written in C++) used for dealing with these observations, but their work is focused towards research, not software development.

There are various variables associated to reading and analysing radar information, and they are all currently stored as local variables (somewhere) in the aforementioned C++ software. It is common for a researcher to want to query some of these variables for their research. Today, I found out that the standard practice here is to check out a new branch of the code, modify it to print out what you want, and then rebuild and run the code locally. This modified code is never checked back in, since it is too localized to be useful to anyone else.

To me, this looks like very bad practice (though I've not been developing software that long, so maybe one of you will correct me). I have always held the opinion that version control (they used a modified form of SVN) should be used primarily for making changes to the code that you intend to check back in, and should never be used for making one-off queries. That is what SQL is for.

Unfortunately, I am not at all sure how easy it will be to implement an SQL front end for this program. At the moment, all the information is stored away in local C++ variables in various modules throughout the program. I have had two ideas for how to fix this:

  1. Write a function (probably involving some rewriting of the existing code) that writes all of the information that might be useful into a database in some piece of database software, using standard libraries. Then users can run this program over some data, and write SQL queries on the resulting database, using the database software's built-in parser.

The good thing about this approach is that the SQL side of things (as opposed to the database side) is entirely handled by existing software. The downside is that it requires users to build the entire database even if they only want a small query. The team frequently deals with large amounts of data coming in on an hourly basis, and it might be impractical to keep having to build enourmous databases when you're only interested in small queries. The other problem is that the organization as a whole is reluctant to install new software on company computers, and it might be difficult to get appropriate database software installed.

  1. Use a standard SQL parser library for C++ to query that somehow queries the relevant local variables in the C++ program as if they were stored in a database. So if I passed in the SQL query
SELECT RadarID, Height, Signal Strength, Wavelength FROM RadarObservations
WHERE Date = '03/07/2015'

then the program would give me out all the specified fields for radar observations on 03/07/2015, without creating the whole database, but getting the values directly from the local variables in C++ where they are stored.

Of the two, option (2) looks the more attractive, but I doubt that it's even possible. Has anyone any experience with either option, or can anyone suggest anything different? One course of action might be to have a rolling build that builds the databases each day so people can query them, but people here are interested in historical data too, and holding on to all these databases might pose storage issues.

Improve this question
4
  • 1
    Why not put configurable text logging into the C++ product. You can then write a separate program (in any language) to consume the log offline, or in real time, and populate a SQL database. This will decouple the SQL/DB requirement from your C++ code base, and you won't have to compile & link against or even install SQL on most machines.
    – Erik Eidt
    Jul 3, 2015 at 15:05
  • @ErikEidt That's a very nice idea. At the moment, queries are being done by writing ad-hoc text logging into the product for a specific case, so it wouldn't be too different from what's going on already.
    – John Gowers
    Jul 3, 2015 at 15:08
  • 2
    There is no such thing as a "standard C++ SQL parser". Option 2 is more work than the entirety of your radar observation teams work, and it's just reimplementing a wheel. The way they are doing it really isn't all that bad, for research.
    – whatsisname
    Jul 3, 2015 at 15:58
  • 1
    "... writes all of the information that might be useful... " - If its research, how do you know you have captured all the information that might be useful? How likely is it they would end up checking out, modifying, updating the database, running the query - i.e. more work than is currently being done). Another approach could to use branching in the RevControl. Maybe look into distributed system (GIT or Mecurial) with and SVN extensions so you can have local repos and branches.
    – mattnz
    Jul 4, 2015 at 6:33

1 Answer 1

Reset to default
2

Why not put configurable text logging into the C++ product. You can then write a separate program (in any language) to consume the log offline, or in real time, and populate a SQL database. This will decouple the SQL/DB requirement from your C++ code base, and you won't have to compile & link against or even install SQL on most machines.

For the product, concentrate on defining a consistent and easy to use configuration to enable logging of the various bits of interest, and, make a consistent and easy to parse logging format for all those bits. (These log entries could look like events, maybe: time-stamp, subject, property, value.)

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.