I'm about to start a new project for my company1 which architecture will comprise several elements. There are 2 mobile applications (Android and iOS), a web API (Java) and an ETL (still deciding whether I use NodeJS or Java).
The system should extract data from several OpenData repositories (web APIS), transform each dataset into a unique data model and consolidate the data.
The resulting model is the model to be published through the web API and the one consumed by the mobile apps. Additionally, the web API will provide features similar to the ones we find in social networks (likes, favourites, share links, suggestions, etc)
The system will be deployed as Docker containers. Probably in a public cloud platform.
I have built similar systems before and I have realised that I always build the ETL with a high-level programming language (mostly in Java). I'm aware of the weakness of Java regarding serialization/deserialization and memory management. I came to the conclusion that Java has become my Golden Hammer and sometimes it could be overkill (or inadequate).
I started to consider other languages like NodeJS because it's very good at performing I/O and data transformation (JSON). The kind of ETL I have in mind can take advantage of these features and easer the implementation.
However, I have been performing DevOps' tasks recently and I fell in love with Linux shell scripting. Overall with commands such as
jq. I even managed to build an ETL with these 4 commands with no need of another programming language. I have found the O.S layer to be quite comfortable, fast and efficient.
In consequence, I started to play with the idea of taking advantage of the O.S capabilities and to build the next ETL laying more extensively on the O.S layer.
Due to I have never built something similar, my questions are
What are the risks (at least the more relevant) of relying on the O.S layer (commands and shells) as the programming platform?
What do I should be aware of?
Is it just me or we usually don't take advantage of the underlying capabilities of the O.S? If so, why? 2
The idea would be to implement a lightweight web application where users can programme dynamic (e)xtractions, (t)ransformations and (l)oads, typing shell script code into text areas. Turn these text areas into executable
sh files and execute them in a given order. If you are familiar with Jenkins, you probably know what I mean.
The extraction can be dealt with
wget, transformation with
jq, awk and sed and the load, well I hope Ubuntu has a MongoDB cli. If it has not, I could solve that part with Java or NodeJS.
1: Where I play the role of a Senior Software engineer. But for this project, it's expected of me to be the architect. I think I can do it, but I have never played fully this role before
2: I wonder if this question is (somehow) similar to the eternal discussion of whether placing business logic in the database is evil.