This question continues on a previous question asked here about continuous integration for scientific software. Like the poster of that question, I am developing software for numerical simulations and I am in the process of applying continuous integration (CI).
My main problem is that I have tests that are used for verification that have to run for a long time (>10 hours). Also, those tests require the use high performance computing resources (an HPC cluster).
From what I have read so far, the idea behind CI is to make sure that a merge cannot be executed in the remote repository, if:
- the build has failed (this is not a problem)
- a verification case is broken by the pull request.
Testing for the build success is possible on the server that is hosting git (Bitbucket, Gitlab, etc), because the code can be compiled rather quickly (order of minutes).
Testing for the verification cases would require the remote git repository server to communicate with the HPC cluster and run simulations there until it is certain that no verification case is broken.
I am using Bitbucket as the remote git server, and I was reading about Bitbucket pipelines, Travis, Jenkins, etc.
The problem I see with the use of another server for running the tests is the authentication (security). The users of the HPC cluster are accessing this machine via SSH. The HPC cluster manages the execution of simulations with a help of the workload manager that schedules simulations as jobs that are described with a queue, priority and status.
If I use Jenkins to submit a job on the HPC cluster using the ssh plugin, the script will submit a simulation and exit with 0, if the submission was successful. This does not mean that the test has succeeded, because the simulation can take hours to complete.
Also, if the Jenkins server is to use SSH to connect to the HPC cluster, it needs the public SSH key. I haven't found the way for Bitbucket to communicate this information to Jenkins.
Has anyone tried to use continuous integration with tests that run for hours/days?
Edit : The responses to the question address the fact that one cannot wait for 10 hours for his/her commit to be accepted. This is not the plan: the idea is to run the whole test suite, when a pull request is submitted to the main upstream repository, to make sure that nothing submitted in the pull request breaks what has already been implemented. Same tests can (and should) be run manually by the devs on the HPC cluster before even submitting the pull request. In my field, the pull request means a numerical agorithm has been developed and tested, this happens maybe once in a month.