The specific setting in github is whether or not branches are required to be up to date with main in order to merge to main. You can turn that off. Your current build process presumably requires at least two things before merge: the branch must be up to date and the tests must pass. I don't believe it's uncommon for a repo to apply both these requirements: I work with many repos that have both, but their tests don't take hours to run.
If you change this to, "the rebase and merge is done automatically for you after CI passes", then you'll no longer be requiring that the up-to-date version of the branch passes the tests. You'll only be requiring that some version of your changes that has no conflicts with the main branch, passes the tests. Some people will consider that sufficient, and if you want to persuade your current team to change their rules then you need to convince them that it is.
Clearly it's a least a bit riskier than the stronger requirement, though. It's easy enough to come up with two changes that do not conflict in the git sense, since they don't change the same lines, and neither of which causes the tests to fail on its own, but which do cause the tests to fail once both of them are applied. The current process would catch this without merging to main: your proposed change to the process would not. But if it's the only way to get any work done then the team might accept the risk, especially if CI runs again on main after the merge. Then the risk is of a main branch clearly marked broken, not the risk of a main branch that's unknown to be broken until someone else rebases off it and tries to run the tests on their branch.
More generally the problem you're facing is that optimistic locking can cause extremely long delays with high contention. For that matter, pessimistic locking can also cause extremely long delays with high contention, but at least with the pessimistic locking you don't think you're making progress when you aren't!
Your optimistic lock looks like this:
- rebase onto main. At this point you acquire the read-only part of an imaginary reader-writer lock
- run tests (many hours)
- attempt to acquire writer lock in order to merge: fails if any write has occurred since you acquired the read-only lock. If it fails, restart the transaction from scratch.
A pessimistic lock for the same process would look like this:
- stand up in the middle of the room and loudly say, "I have a PR ready to go and I'm rebasing onto main. Nobody else merge to main until I'm done, please". Or maybe you can find some implementation other than shouting: old-timey VCS used to let you "lock" files in the repo pretty much for this reason (also to prevent conflicts even happening). But distributed version control doesn't really work like that.
- run tests (many hours)
- merge to main
- stand up un the middle of the room and loudly say, "OK, other people can merge now if they like" - they are now free to stand up, take the lock, and start their transaction.
A further consideration for your team is that regardless of the locking mechanism, there is a fundamental time constraint. Since each PR's "final" run of the tests cannot start until the previous PR's run has completed, necessarily you cannot merge PRs faster than the tests run. This is a real problem when the tests take many hours: how many PRs per day does your team want to be capable of merging? Put like that, probably more than two in most teams, and ideally without having to log back into work just before bed to check on your build. This argument might help persuade the team to relax the merge requirements, and/or to find ways to shorten the build process.
github does have an auto-merge feature, which you can use, to merge your PR as soon as all requirements are met. This needs to be enabled per-repo IIRC. Once enabled, on each PR that's still running the tests you get an option to "set auto-complete" in place of the usual option to merge. For that matter, you can set auto-complete while the PR is still waiting for a required review, and if that's the only requirement missing then the PR will merge as soon as it's approved. It would help you in the "happy" case where nobody else merges during your testing window. It won't help in the unhappy case where they do.
There may be another option, which is to use fast-forward only merges into the main branch. The reason this helps is that if someone is ahead of you in the queue, then instead of rebasing onto main, you can rebase your branch onto their branch. That way, once they merge to main, your branch is instantly up to date with main! And maybe you've already tested your branch by then, in which case you can instantly merge. Victory! However, this doesn't work if they do a regular merge into main, since your branch won't have their merge commit, so it won't be up to date even though it's base on an identical revision. I've never actually done this as a routine process, but I do occasionally base my feature branch off some other feature branch that I know will go in ahead of me.
I'm not aware of a github feature to do what you really want, which is "at the end of a successful test run, if the branch is not up to date with main but doesn't conflict, then rebase and rerun all the tests". Combined with auto-commit, this would solve your problem, although potentially it still would run the tests repeatedly for days before it can actually merge. The fact I haven't seen this feature, though, doesn't mean it doesn't exist and also doesn't mean you can't figure out a way to make your build script do that. I can't remember what the best way is to set up the permissions, because I create repos fairly rarely and they need this kind of shenanigans extremely rarely. But with the right token you basically could do "git pull; git rebase origin/main; git push" directly in your build script.
CI (which takes hours). Just curious: why?