0

CI/CD options such as bitbucket pipelines and github actions make use of virtual machines called runners. Changes in source code trigger the runners, which in turn trigger a set of commands. The commands may include instructions to run a test suite. The test suite checks whether source code changes break existing functionality.

Contributors often develop tests on their local machines. Tests that succeed on one platform do not necessarily succeed on another platform. This means that tests running on a remote machine can (and often do) behave unpredictably as compared with how they behave under local machine conditions. For example, a test that relies on Windows-style line returns may fail if it runs under a Linux operating system.

There are a few ways to circumvent this problem:

  • Make test assertions platform-independent. In the above example, this might mean replacing every newline character with an empty string.
  • Replicate the runner conditions on the local machine using containerization.
  • Accept failing tests in the local machine

Only the second option seems viable, however I have seen very little to no support for this in the documentation of most CI/CD tools. This leads me to believe that I might be missing something. For the professional software developers/testers out there, has anyone run into this problem? What is the correct way to address it?

2
  • 5
    Tests that fail on the CI/CD runners should be fixed. I would think tests could be made robust enough to detect the platform and adjust their assertions accordingly. Even better would be to avoid assertions on data which may be subject to platform variations. – Dan Wilson Feb 25 at 16:20
  • Or replace newlines with \r\n on Windows and \n on Linux... – user253751 Feb 25 at 16:44
6

If the test fails, the software is broken. So fix the software.

It sounds like you haven't even figured out which operating system your software runs on, yet. Figure that out. If you are deploying the software on Linux then you must test it on Linux. If you are deploying the software on Windows then you must test it on Windows.

Then, tests run on the other platform are simply not valid. If developers don't like this, they should either make the tests platform-independent1 or use the same operating system that the software is designed to run on. If someone complains that the Linux test doesn't pass on Windows... tough, this is Linux software, why would you expect the tests to pass on Windows?

1 NOT by ignoring differences, but by coding them in the tests! Tests should use \r\n on Windows and \n on Linux.

1
  • Your answer helps. The platform exists at a lower level than the app itself. Therefore it will have inevitable consequences on how the app will run. I must develop apps fully aware of the platform. I was sort of looking for a "generic" enough platform that would allow me to test on one platform and deploy for all platforms. I realize now that this does not exist. For instance, if I want to develop a Mac OS app on a Windows machine, I can only do this if I have access to a Mac OS somewhere from my Windows machine... – user32882 Mar 1 at 8:26
0

If your software runs on both Windows and Linux, then your test assertions need to work for both platforms. I would write a helper method for string comparisons that ignore line ending differences, and change your assertions to use that helper method.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.