I am about to embark on a large analysis/data extraction project, which I intend to do with Python.
My data to analyse consists of tier 1 files that include some details. and each file points to several tier 2 files. Each of these tier 2 files includes more details and references to PDF files (tier 3) that might need to be analysed. And of course there are lots of cross-references, conflicting details, etc., that will need to be manually curated later in a trackable manner.
To add insult to injury, the extraction has to be repeated on multiple datasets that differ substantially (so most of the extraction code will need to be rewritten). The data is therefore destined to enter a suitable database where it should be structured across all dataset for easy presentation.
I am looking for some advice on how to structure and run all this. I was intending to have e.g. a script for each step (i.e. a script for extracting data from tier 1, populating the database. Script for tier 2 would use database as basis for which files to further analyse, etc.).
My thoughts on file structure are as follows:
/rawdata # this is where all input data is located. /2009 # and the different datasets, collected by different teams /2010 # in different formats, in several different years. *sigh* /docs # probably needed for different documentations... /XX # for lack of a better name, where the extraction takes place. database.db # common database for depositing extracted data. /2009 step1.py step2.py /manual_curation # something to track how manual curation was performed. /output # contains code for generating the end product in a presentable manner (e.g. pdf-files, web-site, etc.).
All of this is naturally contained within a git or SVN repository.
Regarding the database, it is my impression that there are many relations, which is why I am looking at an SQLite database. I have previously used SQLite in several projects. I am also experienced in programming in Python, however, structuring files and running the analyses in a consistent manner is not something I can boast about.
Any advice and experiences in this kind of setup would be very welcome!
(PS. This is a duplicate of my https://stackoverflow.com/questions/31427538/analysis-project-structure-in-python because I've just found this community and thought it might be a more suited place to ask.)