0

In my projects I have several third-party Python packages which I need to build from source (because there are no .whl packages published for the versions+platforms I need). Currently these packages are built in the Dockerfiles which create my development containers and deployment containers. This has a few drawbacks:

  • the third-party packages are rebuilt too often (even if the package itself has not changed it might be rebuilt, depending on the Docker layer cache). This increases build times.
  • if the same package is used in multiple projects, the build instructions are sometimes duplicated (i.e. once in each project). Over time this can cause inconsistency between the builds, if changes are not applied in all places.
  • the Dockerfiles become more difficult to maintain: rather than just installing all packages from a single requirements.txt or Pipfile, there are multiple installation steps; and it is not clear afterwards from which source/repository a package was installed.

I already have a company-wide repository for Python packages; so I could build and upload .whl files for the third-party Python packages, and then install them as usual.

Also, the third-party packages change rarely (maybe once every few months); so they don't need to be rebuilt often. OTOH this also means that the builds should be automated as much as possible (because noone here will get much practice in building these packages if this is only done rarely).

And sure, I could try to solve some of the mentioned drawbacks by using Docker features (like including Docker files in each other, using multi-stage builds, etc.). But I wonder: what is the "usual" solution for reliably building and maintaining my own binary packages of third-party dependencies?

Improve this question
6
  • 2
    Is automating the builds worth it? Let's say the third-party package is on v1.2 now. There's nothing that says that the build process for 1.3 or 2.0 will be the same. If any scripts or manual steps change, your automation will also change. And if you're only building a new version of this packages every few months, is the time to build and maintain automation for someone else's project in your own private space worth it? If they are open-source projects, perhaps it's more valuable to contribute improved build scripts to the projects and get community help to make everyone's build easier.
    – Thomas Owens ♦
    Commented Oct 5, 2023 at 15:55
  • Author and add conda forge feedstock for the package. Encourage the upstream package author to maintain it.
    – J_H
    Commented Oct 5, 2023 at 18:32
  • 1
    "packages change rarely - [...] so they don't need to be rebuilt often[...]. OTOH this also means that the builds should be automated as much as possible" - Sorry, but that is a fallacy. It actually means first and foremost you need to document the build steps thoroughly, so a rebuild can be done manually (and can be adapted more easily when the build steps change, as Thomas Owens' scetched above. When you just automate, and then the scripts fails for a new version, someone will have a pretty hard time to fix the automation.
    – Doc Brown
    Commented Oct 5, 2023 at 20:28
  • ... that does not mean you cannot try to automate the process, too, as a bonus, but not "as much as possible", only "as much as sensible".
    – Doc Brown
    Commented Oct 5, 2023 at 20:30
  • 1
    @oliver: it depends how readable the build steps in some script are. Ideally, you have both, some script plus some documentation, probably at a slightly higher level of abstraction. Prose can help to understand what you have to do in a few months when your original buld script does not work any more because of a changed build process.
    – Doc Brown
    Commented Oct 6, 2023 at 14:04

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.