DB API for shell scripting (any shell)

Question

I am faced with some legacy shell scripts that run batch data processing jobs in Oracle using SQL+. For the most part, the data tier does not have to communicate back to the script with retrieved data to be passed for shell-level processing but in a few cases it does.

The problem is, SQL+ is really meant to be an end user app and not an API that can communicate with other clients programmaticaly. That is why people have invented APIs such as DBD::DBI for Perl, JDBC for Java, ODBC etc. The way it is done is they invoke SQL+ and then parse the output, which is clearly designed for human eye consumption, using tools like sed and awk. The whole thing is at best a hack and very prone to bugs.

Since this client is rather conservative with their technology, they don't want to scale their scripts up to Perl or Python where there are data access APIs. So I am wondering whether there are similar APIs for shell, e.g. K or bash. What I would like is if an API would return data in a 2-dimensional array or strings (for the lack of type setting) so that I can just read DB data like that.

The way they do it now is akin to parsing regular web page HTML to get a single stock quote rather than cleanly calling a web service and be done with it.

Anybody know of a product I can use? Thanks

Answer 1

When I've worked on script and database systems that sound similar we did most of the complex logic (in any way related to the database) in PL/SQL rather than the script.

So the bash script would be a relatively simple thing, the PL/SQL would have the complex logic. When it was particularly complex we would use a stored procedure and trigger that from the script.

This might be a less politically-perilous route than trying to champion Perl or Python.

Answer 2

Classic unix/linux shell tools are almost always text based - that is their strength (because of simplicity) and their weakness (because of the reasons you gave by yourself). So tools like SQLplus fit in that scheme, because they are text-based, too (though I think you are right, using this through a shell script has a high risk of becoming error prone and fragile).

I don't know any solution to your problem for K or Bash. But in the MS Windows world, there is a modern solution called Powershell which has exactly the capabilities you are asking for, and older scripting languages like VBScript can use COM based APIs like ADO / OLE-DB for database access. This may not help you for your case, since rewriting a bash script in Powershell will produce a comparable effort as rewriting it in Perl or Python.

Actually, if you cannot replace the old Bash code by no means, I first would try to live with the SQLplus solution and learn the gory details of how to deal with the weaknesses of that approach. SQLplus is old and mature, and there are solutions around for almost every problem you might encounter. Perhaps, if you have to add new features, you can try to use additional Perl or Python scripts and combine it with the existing shell scripts. For example, use Perl to call data from a web service and convert the data to a text format which can be processed furthermore within a bash script. And perhaps, if you show that solution to your client, you may convince him that bash scripts are not the best solution for everything.