Are Data Warehouses bespoke products that are tailored towards the specific needs of one single organisation?

Question

Are Data Warehouses bespoke projects that are developed to meet the needs of a specific organisation or can you buy off the shelf products that would meet the needs of many organisations?

I am talking about a product that will integrate information from many different systems e.g. finance system, sales system, production system etc for management reporting.

Different organisations have different systems that have different database designs. For example, organisation A may use finance system A, which has a completely different design to finance system B used by organisation B.

Therefore I believe that Data Warehouse products are bespoke products and it is not possible to develop a product that could be used by many organisations e.g. organisation A and organisation B; that have the same types of systems e.g. Finance, Sales etc.

Am I correct in this analysis?

Answer 1

Data Warehouses are not products, they are data bases with a specific architecture (logically or physically). If you read Gartner Magic Quadrant for Data Warehouse, it basically lists DBMS vendors.

There are DBMS that's more suitable for data warehouse operation (e.g. aggregation) which are more commonly known as column oriented DBMS. This kind of database is physically different from row oriented database.

Logical difference between data warehouse and plain database is the structure where in data warehouse you will find star or snow flake schema kind of structure to define relationship between Fact tables and their Dimensions.

Data warehouse is a place where you store your data for later analysis as oppose to storing transactional data. You can build it using any DBMS or you can use column oriented DBMS to get more performance.

I think what you want is Business Intelligence suite. They usually have ETL tools that can be used to Extract data from your production database, Transform them (e.g. to create facts and dimensions) and Load them to your data warehouse. Because you mainly work with database, you can use ETL tool with different systems (finance, CRM, etc.) as long as you understand the structure of their databases.

Once you built your data warehouse, you use other tools in the BI suite to do analysis, reporting, data mining, etc. You need to build each of them, but the tools basically only need to be able to get data from data warehouse, they don't need to be able to speak to those other systems (finance, CRM, etc.). You can even use ETL tool from one BI product and build report using another BI product.

Answer 2

Your assumption that a data warehouse can be a "single, off the shelf product" doesn't seem to fit in the context of what a data warehouse can be or is.

From the info provided here:

http://en.wikipedia.org/wiki/Data_warehouse

• The first thing you need to define is what "type" of data warehouse you want to build. The more generic the situation, the more likely you can use a product built by another company.

• You also need to be aware of the fact that if such generic data-warehouse software exists, it will most likely incorporate the ability to pull in data from a variety of sources (as the article mentions: "Integrating data from one or more disparate sources creates a central repository of data, a data warehouse (DW)")

• Lastly, it could also be that a data-warehouse piece of software is built similar to something like a Business Process Management(BPM) or Enterprise Resource Planning (ERP) software, where the end user is meant to build up the models using the software, thereby allowing the end-user to specify the customizations for integrating/pulling in the data from a variety of sources. This would mean that the end user would most likely need to know how to write programming scripts, or at the very minimum, know something like SQL.

If you feel my answer is lacking in any way, feel free to comment below and I will try to fix it.

Answer 3

It is true Data Warehouses typically pull data from many different sources/parts of the business. A Datamart typically pulls data from a lone source.

There are applications that cater to specific industries that offer the ability to integrate and pull data from different resources into a common data structure. I still wouldn't call this a data warehouse because much of the data is structured for transactions and other relational forms and not completely optimized for denormalized report pereformance or historical data capturing.

These types of systems are easier for the manufacturer to create a data warehouse. The core structure (Star Schema?) may be there along with some customizable data cubes. Many firms have inhouse expertise with something like Excel which has PowerPivot to do some customizable BI.

I've worked in the legal and financial industries and some of their popular enterprise apps offer this. Lengthy engagements, installations, customizations and testing are common, so I wouldn't exactly say this works "right out of the box". Then again, which enterprise apps do?