3

Context

I'm about to start a new project for my company1 which architecture will comprise several elements. There are 2 mobile applications (Android and iOS), a web API (Java) and an ETL (still deciding whether I use NodeJS or Java).

The system should extract data from several OpenData repositories (web APIS), transform each dataset into a unique data model and consolidate the data.

The resulting model is the model to be published through the web API and the one consumed by the mobile apps. Additionally, the web API will provide features similar to the ones we find in social networks (likes, favourites, share links, suggestions, etc)

The system will be deployed as Docker containers. Probably in a public cloud platform.

Rationale

I have built similar systems before and I have realised that I always build the ETL with a high-level programming language (mostly in Java). I'm aware of the weakness of Java regarding serialization/deserialization and memory management. I came to the conclusion that Java has become my Golden Hammer and sometimes it could be overkill (or inadequate).

I started to consider other languages like NodeJS because it's very good at performing I/O and data transformation (JSON). The kind of ETL I have in mind can take advantage of these features and easer the implementation.

However, I have been performing DevOps' tasks recently and I fell in love with Linux shell scripting. Overall with commands such as wget, awk, grep, sed and jq. I even managed to build an ETL with these 4 commands with no need of another programming language. I have found the O.S layer to be quite comfortable, fast and efficient.

In consequence, I started to play with the idea of taking advantage of the O.S capabilities and to build the next ETL laying more extensively on the O.S layer.

Question

Due to I have never built something similar, my questions are

  • What are the risks (at least the more relevant) of relying on the O.S layer (commands and shells) as the programming platform?

  • What do I should be aware of?

  • Is it just me or we usually don't take advantage of the underlying capabilities of the O.S? If so, why? 2

Note

The idea would be to implement a lightweight web application where users can programme dynamic (e)xtractions, (t)ransformations and (l)oads, typing shell script code into text areas. Turn these text areas into executable sh files and execute them in a given order. If you are familiar with Jenkins, you probably know what I mean.

The extraction can be dealt with wget, transformation with jq, awk and sed and the load, well I hope Ubuntu has a MongoDB cli. If it has not, I could solve that part with Java or NodeJS.

1: Where I play the role of a Senior Software engineer. But for this project, it's expected of me to be the architect. I think I can do it, but I have never played fully this role before

2: I wonder if this question is (somehow) similar to the eternal discussion of whether placing business logic in the database is evil.

Improve this question
11
  • I once came upon a one-man IT department with all kinds of scripts, Perl scripts that needed to be adapted before running and such. Doing all in java (with a JavaFX menu for all jobs, input dialogs, output, info was really much better. However with well kept documentation Unix tooling with awk/sed and others is perfectly okay. At a level of some complexity, or with technology APIs (PDF, DB, ...) and their combination java is really productive and of good quality. Just the following points: Docs (Software+Tickets), TDD (unittests), VCS.
    – Joop Eggen
    Apr 11, 2018 at 10:54
  • The point is that the web app becomes a tool where you can write your own shell scripts. They can be stored as files or as database entries (transformed into files later) but you never have to maintain these files. Of course, I will have to constraint what users are allowed to execute, the user (and group) executing the process, etc.
    – Laiv
    Apr 11, 2018 at 11:08
  • I have experienced systems with business rules maintained by the end customer (the firm's departments). Disadvantages: programming craftiness, experimenting in production, bad version control, documentation & overview. Your question sounded a bit scary in that respect. But is also sounds like something powerful and versatile, so good luck.
    – Joop Eggen
    Apr 11, 2018 at 11:20
  • 2
    Since you used both kind of tooling, I would expect you already to know most of the advantages and disadvantages of those command line tools: the main advantage is you can solve a lot of problems with very few lines of code. On the disadvantage side, you find restricted error handling capabilities, almost no possibilities of creating data structures or OO code, very restricted debugging and profiling capabilties and very restricted UI capabilties. Most of these disadvantages can be mitigated by using a scripting language like Python or Perl instead.
    – Doc Brown
    Apr 11, 2018 at 11:55
  • 1
    Probably, I will gather all the issues raised in the comments, add mines and answer the question myself. Anyways, if anyone feels like elaborating an answer, I will appreciate it.
    – Laiv
    Apr 11, 2018 at 16:09

1 Answer 1

Reset to default
5

Since you used both kind of tooling, I would expect you already to know most of the advantages and disadvantages of those command line tools: the main advantage is you can solve a lot of problems with very few lines of code. On the disadvantage side, you find

  • restricted error handling capabilities
  • almost no possibilities of creating data structures or OO code
  • it is (at least in my experience) harder to implement something like input sanitizing (but YMMV)
  • very restricted debugging and profiling capabilities
  • very restricted UI capabilities.
  • each command will spawn a new process, and the communication between these tools can exclusively be done by files or pipes. This can have a performance impact.
  • shell script notation can be really convoluted and hard to maintain by inexperienced developers. Leaks in expressiveness and readability
  • suffers all the drawbacks of procedural and batch programming

OS dependency might also be an issue, and it might be easier to keep the code OS independent by using Java than by using Linux/Unix shell scripting. But even Java programs will become OS dependent if you don't be careful, and Linux shell scripts can often run on Windows by utilizing an environment like Cygwin.

Is there an alternative, which allows to write programs almost as concise as shell scripts, almost as portable as Java, but without the mentioned drawbacks?

Of course there is: several scripting languages. For example, good-old Perl was AFAIK originally designed exactly for this, and I am sure you will find a Perl module equivalent for almost every major Linux command line tool on www.cpan.org. The Python ecosystem has similar properties, but a much cleaner core language. For example, wget is also a standard Python package, awk and grep can be replaced by the capabilities of the re module, and jq by pyjq.

Since you mentioned node.js, since it is much newer than Python or Perl, I don't expect it's module system currently to be as complete as the ones for the latter languages, but I am not an expert on this, and if it suits your needs, it is surely a sensible alternative.

Improve this answer
6
  • Thank you @DocBrown I have also realised that this approach would raise some security concerns. I should be capable of not allowing users to programme malicious code, to have access to system folders and files, to download (via wget) executables or suspicious files, to have access to ENV vars, to have grants of execution for certain commands that might compromise the container (rm, format, shutdown, kill, etc), to have access to network settings, etc.
    – Laiv
    Apr 12, 2018 at 7:40
  • 1
    @Laiv: I think the security issues are not directly related to the programming platform (Java vs shell scripting), they are related to the idea of letting your users dynamically create ETL tasks. However, to handle the issues, you need techniques like, for example, input sanitizing, and that is something which I think can be much easier implemented in a full-fledged programming language like Java or Python.
    – Doc Brown
    Apr 12, 2018 at 8:01
  • Well as I commented, the idea would be to have a Java web app (or NodeJS) on top, taking care of the validations and sanitization. The web app would take care of all these concerns. The shell scripting thing is for the runnable part of the ETL. Instead of programming the steps in Java. Tthe web app (in the background) materialise the shell scripts in files, execute them and track the result. But I think this can be achieved exchanging Shell scripting by Python or Perl. In other words, allow users to type down Perl or Python code and execute them. It raises other concerns
    – Laiv
    Apr 12, 2018 at 8:09
  • 2
    @Laiv: if you allow your users directly to enter code for a general purpose platform (does not matter if it is shell scripting, Python, Perl or Javascript, or Java code) which then will be executed on a server, you create a security problem - such code can never be safely analysed if it is malicious or not, at least not with reasonable efforts. For this, better create a simplified Domain Specific Language, only containing the allowed ETL operations.
    – Doc Brown
    Apr 12, 2018 at 8:49
  • ... and this DSL then can be transformed by your application into some script (shell script or scripting language) which can be executed on the server. But make sure there is no loophole through which the user can pass some parameters through the DSL into the script which might be misinterpreted as arbitrary commands (like SQL injection).
    – Doc Brown
    Apr 12, 2018 at 10:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.