I am about to embark on a large analysis/data extraction project, which I intend to do with Python.

My data to analyse consists of tier 1 files that include some details. and each file points to several tier 2 files. Each of these tier 2 files includes more details and references to PDF files (tier 3) that might need to be analysed. And of course there are lots of cross-references, conflicting details, etc., that will need to be manually curated later in a trackable manner.

To add insult to injury, the extraction has to be repeated on multiple datasets that differ substantially (so most of the extraction code will need to be rewritten). The data is therefore destined to enter a suitable database where it should be structured across all dataset for easy presentation.

I am looking for some advice on how to structure and run all this. I was intending to have e.g. a script for each step (i.e. a script for extracting data from tier 1, populating the database. Script for tier 2 would use database as basis for which files to further analyse, etc.).

My thoughts on file structure are as follows:

/rawdata  # this is where all input data is located.
    /2009  # and the different datasets, collected by different teams
    /2010  # in different formats, in several different years. *sigh*
/docs  # probably needed for different documentations...
/XX  # for lack of a better name, where the extraction takes place.
    database.db  # common database for depositing extracted data.
/manual_curation  # something to track how manual curation was performed.
/output  # contains code for generating the end product in a presentable manner (e.g. pdf-files, web-site, etc.).

All of this is naturally contained within a git or SVN repository.

Regarding the database, it is my impression that there are many relations, which is why I am looking at an SQLite database. I have previously used SQLite in several projects. I am also experienced in programming in Python, however, structuring files and running the analyses in a consistent manner is not something I can boast about.

Any advice and experiences in this kind of setup would be very welcome!

(PS. This is a duplicate of my https://stackoverflow.com/questions/31427538/analysis-project-structure-in-python because I've just found this community and thought it might be a more suited place to ask.)

  • 3
    Posting the same question on multiple sites is not welcome (but I don't see that in our Help page). It's best to either reword each question to better fit each site or delete one. In your case you may wish to delete the SO question. Jul 15, 2015 at 16:17

1 Answer 1


I see this kind of project as a data pipeline. I usually tend to write individual scripts to handle single steps (e.g. normalize 2009 format, normalize 2010 format, process normalized data, etc ...), and then coordinate these steps with a build system.
I have used GNU Make in the past, but I recently have been using Doit and really like it.
If data is really large, you might want to look into processing it on AWS, possibly with the datapipeline product

  • Thanks. I had missed the data pipeline keyword. Also, Doit looks intriguing - kinda a python based Make?
    – MrGumble
    Jul 15, 2015 at 20:15
  • In a nutshell, yes. It is a system that lets you declare steps and their inter-dependencies. Then it calculates for you which steps need to be re-run. It is less confusing, more flexible than Make. It is also more verbose at the superficial level, but because ultimately it operates on python dictionaries, you can generate them anyway you want and meta-program most of the boilerplate away
    – ppbitb
    Jul 15, 2015 at 20:57
  • I agree doit is a good fit (disclaimer I am doit author). For the database I would definetely suggest you take a look at pandas instead of SQL. pandas not only much easier to do data analysis but also much easier to integrate with a pipeline model where you can save intermediate files in cvs, json, hfd5... Jul 16, 2015 at 4:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.