Integration tests in OSS projects - how to handle 3rd parties with authentication?

Question

One of my (open source) hobby projects is a backup tool which makes offline backups of repositories from GitHub, Bitbucket etc.
It calls the hosters' API to get a list of repositories, and then it uses Git/Mercurial/whatever to clone/pull the repositories to the local computer.

So I have integration tests where I'm calling the GitHub API, with authentication.
(and when the cloning/pulling feature is finished, there probably will be tests which clone repositories from GitHub, and need to authenticate as well)

I created a user and an organization especially for use in these integration tests.

Problem: I can't just hard-code the passwords somewhere in the source code because it's open source, and the code is public on GitHub.

---

What I'm doing now

In the tests, I'm getting all user names, passwords and repository names from environment variables.
Here's an example:

config.Name = TestHelper.EnvVar("GithubApiTests_Name");
config.Password = TestHelper.EnvVar("GithubApiTests_PW");

(TestHelper.EnvVar is a helper method which gets the value of an environment variable and throws an exception when it doesn't exist)

Then, I have a batch file which sets those environment variables.
The real one (environment-variables.bat) is called in my build script and before executing the tests, but ignored in source control, so it's not actually in my repository.

What is in source control is environment-variables.bat.sample, which sets the same environment variables, but with fake passwords:

rem copy/rename this file to environment-variables.bat

echo Setting environment variables for integration tests...

set GithubApiTests_Name=scm-backup-testuser
set GithubApiTests_OrgName=scm-backup-testorg
set GithubApiTests_PW=not-the-real-password
set GithubApiTests_Repo=scm-backup

So I can clone the repository to my machine, rename this file to environment-variables.bat, replace the fake password by the real one, and all the integration tests will work.

This works with Continuous Integration as well - I'm using AppVeyor, and there I can set these environment variables in the web UI.

---

What I don't like about it

I think it's not a good solution for an OSS project, and especially not for this project:

In theory, a contributor to my project would be able to run the integration tests right now by:

• creating his own test user and test organization on GitHub
• creating some test repositories
• creating his own version of environment-variables.bat with different values

The problem is that my application will be able to backup multiple source code hosters.
Right now, it supports GitHub only, but it will be easy to add support for more hosters by adding a few classes which implement the right interfaces.

So when I implement support for more hosters later, the number of environment variables will grow.
To be able to execute all integration tests, a potential contributor would create his own users, organizations and test repositories at GitHub, Bitbucket, GitLab, .... and who knows how many more, and add all of them to his environment-variables.bat version.

Is there a better solution how to do this on a project where the code is public?

I know that other projects do something similar to what I'm currently doing.
Octokit.net, for example, has a script to setup environment variables for integration tests that are calling the GitHub API.
But they only need one user and one organization, and I will need a lot more.

Maybe I don't need a solution which enables a contributor to actually run all integration tests.
For example, if someone would want to contribute to my project's GitHub support, he would only need to be able to run the GitHub integration tests.
So maybe I just need a sane way to be able to divide my integration tests into an infinite number of "groups"(?) and then to say "and now execute all tests which belong to group 'Github'".

Answer 1

I think your current setup is just fine, but I would make a few adjustments.

To be able to execute all integration tests, a potential contributor would create his own users, organizations and test repositories at GitHub, Bitbucket, GitLab, .... and who knows how much more, and add all of them to his environment-variables.bat version.

Yes, that's true, but contributors don't necessarily have to run all integration tests before creating a PR on your project. Once a PR is created, the CI will run the full suite of tests.

It's common to have a test suite that is somehow not possible to run easily. For many organizations they maintain test suites that take days to run--therefore developers must selectively run tests that are easy to run and push the code forward for more rigorous testing. I'm suggesting the same approach.

For regular/trusted contributors, you can make them actual contributors on your project which should allow them to run CI on their branch before making a PR.

That said, you aren't preventing contributors from running the full suite of tests. They can provide their own GitHub credentials or create their own test accounts, and regular contributors will likely do this.

The adjustments I suggest are to:

First, make most of your test coverage unit tests. These should cover all the branches of your codebase.

Second, write integration tests where the endpoint is mocked. You can even mock the transport layer of these API and simulate HTTP request/response flows by starting a fake GitHub Rest service. For example (in pseudo code):

// Test failed authentication to GitHub
val server = new WebServer("localhost", 9453, { request =>
    return Response(401, "unauthenticated")
})
server.start()
val backupService = new GitHubBackupService("http://localhost:9453")
backupService.backup must throw UnauthenticatedException()
server.stop()

These tests will be more complex, but will allow you to

1. Test without creating fake accounts and repositories
2. Test failure conditions that would be hard to simulate with real GitHub. For example 502 responses, connection timeouts, unusual/unparseable response bodies.

Third, disable any tests that need special knowledge from running under a normal build. In most build management tools there is a way to separate integration tests, or tag tests and selectively run them. Contributors should be able to build and test the software without any prior configuration. The full test suite should run after each build in CI.

Finally, document the tests and how to run them in your test documentation so contributors can choose to run them.