Questions tagged [etl]
Extract, Transform, Load - process in a database
8
questions with no upvoted or accepted answers
3
votes
1
answer
262
views
What are some design ideas for a data mapping and transformation application?
Here is a high level outline of the project:
We frequently need to convert data from a new incoming system to our in house system (sort of a basic ETL process)
We would prefer to do this dynamically, ...
- 141
2
votes
0
answers
30
views
How to manage scheduled ETL jobs that are time sensitive?
We have some ETL jobs that are scheduled to run every day, and some that are scheduled to run every week via Control-M. These types of jobs tag data with the date the job was run and perform filter ...
- 2,343
2
votes
0
answers
307
views
Data pipeline architecture: airflow triggered by message broker
Let us say we have:
a web app with a Postgres DB that produces data over time,
another DB optimized for analytics that we would like to populate over time.
My goal is to build and monitor an ETL ...
- 151
1
vote
0
answers
666
views
Parsing a JSON file from S3 using Airflow
I'm new to Airflow and I'm working on a proof of concept. The project is fairly simple... every day some 10,000 JSON files are loaded onto a folder on AWS S3. I have to get each one of them, parse ...
- 143
0
votes
1
answer
312
views
Data Integration Design Using Microsoft SSIS
I am working on a data integration project, where I need to extract data from oracle source and load it to XML file. The requirement is to get the list of customers and foreach customer create an xml ...
- 109
0
votes
0
answers
712
views
Is MapReduce a correct framework for Extract, Transform, Load of data?
EDIT
I am working on a project to update a legacy ETL infrastructure that supports a number of clients, each with a slightly different setup.
Constraints that cannot be changed:
Source data can ...
- 1,396
-1
votes
1
answer
236
views
WebApp for ETL with visual mapping - read csv and map it to data model
a few years ago I wrote a python script for reading CSV, handling the headers, filtering data, renaming stuff via RegEx...bascially to do various ETL stuff.
This was done using a exhaustive ...
- 109
-1
votes
1
answer
132
views
How to incrementally update value of features in a machine learning pipeline?
I am working on a machine learning pipeline where we have to compute certain measures on streaming data. Every day, new raw data enters our pipeline. To update our features, we have to run an ETL that ...
