Currently have situation between having not much data (mid single-digit to low double-digit TBs (not per data set mind you, but the average total amount)), but desiring tools similar to those that come bundled with hadoop packages, specifically Sqoop, Atlas, and Ranger (which can be easily installed via Ambari with Hortonworks).
Basically, it would cost less to get more drives to store data and RAM to process data right now than to store on HDFS and process with spark, etc than to pay for support from Hortonworks (and in this case not allowed to have HDP without paying for enterprise support), but would still like some way to catalog data sets for end/business user discovery and labeling as well as maintaining audited access to the filesystem.
Essentially, thinking about how to have a standard data lake architecture like this
https://medium.com/@rpradeepmenon/demystifying-data-lake-architecture-30cf4ac8aa07
but with Hadoop components (mainly HDFS storage, Sqoop EL, Atlas data catalog, Ranger goverance, and Zeppelin data analysis) replaced with something cheaper and non-Hadoop-based that can still run locally on/across our servers. Most importantly is to have a Sqoop alternative (which we currently use to import 100M+ rows from 100+ column-wide tables each day via virtual tunnel to a remote Oracle DB).
Any general or platform/project/architecture recommendations for this situation?
* Do let me know if there is a more appropriate SE community to post this question to.
