I've found several things about how to manage a data science projet with GIT but I didn't find something about how to manage a set of projects.

In 90% of the case I'm working alone and over the month a lot of people ask me to check:

  • the performance of our marketing operations
  • the impact on sales of special period like christmas.
  • clustering of our customers
  • simple predictive models (churn,...)

Here is my typical workflow for a single project:

  1. Prepare the data in SQL
  2. Make descriptive and predictive analysis in R/Python. I often use my own library of code which I update over the time
  3. Create output results in Markdown or powerpoint presentation.

Here is the folder organisation for each project:

  1. Data
    • base
    • processed
  2. R scripts

  3. Python scripts

  4. Outputs (figures, markdown, powerpoint,...)

And two libraries of code in R and Python that I use for all the analysis

Question: In this case what is the best strategy ?

  1. A single repository with all the analysis because the libraries are shared among several analysis ?

If yes, is it ok to have dozen of branches in the same repository like:

  • R_library_prod
  • R_library_dev
  • Python_library_prod
  • Python_library_dev
  • clustering_2015_prod
  • clustering_2015_dev
  • christmas_sales_analysis_prod
  • christmas_sales_analysis_dev
  • and so on

    1. A repository for each project ? (with potentially only 2 branches: prod and dev)

If yes, how to manage the updates of the R and Python libraries ? Should I have a distinct repo for them and updates the libraries manually in the analytics projects repositories ?

  • FYI, I initially post thie question there – Vadi Jun 11 '15 at 20:18
  • 2
    I would use 2; why don't you make the projects dependent on the libraries (you can specify specific versions, if that's what's needed)? – jonrsharpe Jun 11 '15 at 20:57
  • @jonrsharpe Could you please explain to me what do yoou mean by make the projects dependent on the libraries, I'm really new to this. – Vadi Jun 12 '15 at 10:59
  • That's way too long for an answer! I suggest you start learning about "dependency management" - in Python, for example, you can define a requirements.txt file listing the libraries required for your project to run, or include them in a setup.py. – jonrsharpe Jun 12 '15 at 11:01
  • Have you looked into git submodules for option #2? – Oxymoron Jun 12 '15 at 14:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.