Questions tagged [etl]

Extract, Transform, Load - process in a database

Filter by
Sorted by
Tagged with
1
vote
1answer
34 views

Micro-services architecture for Data Ingestion/Transformation pipeline project

I am working on designing a brand new Data Ingestion Pipeline with the Key highlights of the new project are as follows: Download and Update data to/from SharePoint using SharePoint APIs Download and ...
1
vote
2answers
59 views

Should data be pre-processed before being handled by an ETL framework?

So I was discussing coding with an associate of mine at work, and was mentioning how I was working on a project where I'd need to transform the data that was provided into a standardized format before ...
1
vote
1answer
108 views

How/when to normalize during ETL?

Let's say you're loading a denormalized flat file of purchase transactions that looks like this: | location_name | location_zip | product | product_price | |---------------|--------------|---------|--...
0
votes
1answer
55 views

WebApp for ETL with visual mapping - read csv and map it to data model

a few years ago I wrote a python script for reading CSV, handling the headers, filtering data, renaming stuff via RegEx...bascially to do various ETL stuff. This was done using a exhaustive ...
1
vote
1answer
69 views

How to incrementally update value of features in a machine learning pipeline?

I am working on a machine learning pipeline where we have to compute certain measures on streaming data. Every day, new raw data enters our pipeline. To update our features, we have to run an ETL that ...
-3
votes
2answers
144 views

Why are multiple backends in this system? [closed]

I am trying to understand the architecture of the system described in this patent about aggregating and analyzing confidential data: https://patents.justia.com/patent/20180089196. The general ...
0
votes
2answers
55 views

How to automatically test the result of an ETL tool?

If an ETL tool is being used to move data from an OLTP database into a "business intelligence reporting" database, is there any standard way of automatically testing that the data in the reporting ...
1
vote
1answer
180 views

Data Integration Design Using Microsoft SSIS

I am working on a data integration project, where I need to extract data from oracle source and load it to XML file. The requirement is to get the list of customers and foreach customer create an xml ...
2
votes
0answers
36 views

What are some design ideas for a data mapping and transformation application?

Here is a high level outline of the project: We frequently need to convert data from a new incoming system to our in house system (sort of a basic ETL process) We would prefer to do this dynamically, ...
1
vote
1answer
49 views

What does 'data coverage' mean when talking about ETL processes?

I was watching this talk about ETL's shortcomings and the solutions provided by the Kafka platform but I don't quite understand what the speaker is referring to when she says ETL tools have been ...
0
votes
0answers
109 views

Is MapReduce a correct framework for Extract, Transform, Load of data?

EDIT I am working on a project to update a legacy ETL infrastructure that supports a number of clients, each with a slightly different setup. Constraints that cannot be changed: Source data can ...
2
votes
2answers
226 views

Automating tests for ETL flow

I created am ETL that parse various files, transform file's data and then push the lines into a DB, until now I did a manual tests and check that all the values parsed correctly and all the lines (...
1
vote
2answers
182 views

Data integration from heterogeneous sources

A client has requested that we build a platform for integrating data from partner to their central data store. This will not be "Big Data" scale. The data from each partner will be accessible through ...
1
vote
1answer
1k views

How to implement ETL with MySQL?

I have a legacy MySQL Database (A), and a new reviewed structure for MySQL data base (B). Problem number one is that Database has to be alive and keeps receiving data from legacy apps. What I need is ...
3
votes
2answers
223 views

Translate data between inconsistently-matched data structures

How can my program best represent a translation between imperfectly-matched data structures? I am tasked with a one-way translation of data from one system to another. Both systems are established, I ...
0
votes
1answer
446 views

Exporting data to file share vs calling a web service to handle the export

I would like to hear opinions/best practices for the following scenario: I have an application A (C# app from a vendor that does not have an accessible database, it has a sqllite db but we don't have ...
0
votes
2answers
47 views

what is a better approach in replicating data from table to another? triggers or a third party ETL tool?

We have multiple tables which we need to retrieve data from and dump to one centralized table. Currently what we are doing is running an ETL job made from Pentaho, retrieve the records from the source ...
3
votes
1answer
858 views

How is one or more aggregate function implemented in most SQL engines?

In the book Database Fundamentals, Silberschatz. It is explained that aggregate functions can be calculated on the march. This make sense. What it means is that for calculating the maximun, average ...
4
votes
1answer
10k views

Difference between ESB and ETL

When should a ESB vs ETL tool be used? I have worked on ESB projects using Tibco Businessworks quite a few years ago. The message bus that we built used to consume messages from a source system , ...
2
votes
2answers
932 views

Using ESB for database synchronisation / replication

We're starting to look at implementing an ESB / Microservices architecture. I (think) I know about the concepts, but there's one thing I don't seem to be able to get a good idea about: data ...
4
votes
2answers
1k views

Is ReST useful in Read/Write Operations that involve over 100 Gig

I work in Healthcare and we use SAS to Extract and Transform medical and pharmacy claims data for use in downstream reporting applications. For a given Report Request(usually 40 are running at a time)...
0
votes
1answer
336 views

Coordinating a complicated data migration process

A project I'm involved in has suffered a change in scope, and before I set about trying to cook up some homegrown solution, I'm wondering if there is something out there -- some framework, for example ...
5
votes
1answer
2k views

What is the right way to process inconsistent data files?

I'm working at a company that uses Excel files to store product data, specifically, test results from products before they are shipped out. There are a few thousand spreadsheets with anywhere from 50-...
5
votes
2answers
278 views

Disagreement Concerning Data Integration (I may not understand enterprise ETL tools)

I have been in an ongoing conversation concerning a project we are about to undertake at my place at work. The project concerns data integration. Our customers want to be able to integrate our data ...
1
vote
1answer
139 views

Production or Custom Test Data for Unit Testing?

I've recently had a little disagreement with fellow developers. We're transforming various ontologies from the original source format (Pica+, RDF, etc) into our data format and have several converters ...
3
votes
2answers
682 views

Data warehouse architecture for mutating schema

I am setting up an ETL process and small data warehouse for querying the data in a few different dimensions. One issue is that the schema for the objects can mutate over time - mainly that some fields ...
3
votes
1answer
199 views

Enterprise Wide Keys [closed]

I have for a long time been working on an ODS as well as Data Warehouse. Both are integrating a wide variety of data sources from stove pipe applications. One of the uses of the ODS is to provide ...
4
votes
1answer
3k views

Designing a Content-Based ETL Process with .NET and SFDC

As my firm makes the transition to using SFDC as our main operational system, we've spun together a couple of SFDC portals where we can post customer-specific documents to be viewed at will. As such, ...
7
votes
3answers
4k views

How to do data synchronisation between two applications?

Our client has a CRM application (let's call it A) and a custom built Windows application (VB.NET) (let's call it B) that directly writes to and reads from the CRM database. Application A is used (...
5
votes
2answers
1k views

How can large data handling be a special requirement for job?

I chanced to see some job requirements of Oracle SQL programmer or C# developer jobs, who want a person who can handle terabytes of data in ETL or data migration. I have worked on data migration to ...
1
vote
3answers
495 views

ETL Software Research Question

Where I work, we use an in-house ETL solution that's homegrown and has been around for 5-10 years. I'm still new to my data analysis job, but I was wondering about the ETL tools that are out there. ...