29

I am developing code mainly using Bash, C, Python and Fortran and recently also HTML/CSS+JavaScript. My OS is Ubuntu.

Maybe I am exaggerating, but I figured that I kind of spend more time getting software (Debian and Python packages mainly, sometimes also from source) to be installed properly than actually developing code. And I am not talking about coding vs. debugging, debugging is part of coding for me.

It happens so often to me that I update my Linux packages and then my Python packages and my software does not work anymore, because some .so files have another name now, and Python does not find them anymore. Or I setup a totally clean Ubuntu VM, install a package with pip and get two screens of error message, because some debian package was not installed. I am not a system administrator, I enjoy developing software. But this just annoys me. I do not want to inform myself on all the 157 Python packages and thousands of Debian packages I have on my system and know what their dependancies are. I want to write code and implement new functionality into my code.

What am I doing wrong?

7
  • 12
    Why are you so often updating core packages or setting up a new system? I don't mean that rhetorically, I think the solutions will look different for different root causes: Docker containers might be really helpful for recreating the same environment repeatedly, but will just give you even more things to configure if the required environment really is constantly changing or evolving.
    – IMSoP
    Jan 15 at 13:54
  • 2
    Do you use docker or other kind of virtual environments for developing? They might solve these kind of problems.
    – johnlinp
    Jan 15 at 13:55
  • No, I do not use Docker (I don't know it) but I will look into it. I have to setup these VMs for a project I am involved with. Once a VM is out, I am telling users not to update it and this works fine. On my own system, I am doing apt-get upgrade very regularly. But as I said, I am exaggerating a bit, but if installation problems occur, they are often REALLY difficult so solve. I guess I should open github issues sooner to see if the problem is on their side before spending hours searching the error in my own system. Jan 15 at 14:07
  • 1
    Install everything with apt and pip. Don't build things yourself when they're in apt or pip.
    – user253751
    Jan 15 at 14:10
  • 2
    Perhaps try Arch Linux, a rolling-release near-"bleeding-edge" distro where system maintenance is ironically far less of a time drain than Ubuntu. Installing most programming packages is usually a breeze... far quicker and easier than when I had to work with Ubuntu. The starkest difference for me was a single command pacman -S python-tensorflow-cuda versus having to copy paste two dozen commands which may or may not be outdated and broken, or deal with conda environments. Disclaimer: this was my experience back in the Tensorflow 1.x days; perhaps it's easier now. Perhaps not. Jan 16 at 8:02
43

What am I doing wrong?

You're trying to develop in an environment where you're also the sysadmin, devops and the local technical product owner for every pip package you use - and you're assuming that the sysadmin, devops and TPO roles should be no effort just because they're not what you're interested in.

Those are paid full-time jobs (ok, maybe not TPO) that people have because they are not trivial.

Maintaining up-to-date development environments can be a lot of work. The usual approaches are

  1. to work for a large enough organization that it's someone else's job, or
  2. to somehow automate it (which is why things like conda and docker exist - although this is still a non-trivial amount of work you'd prefer the person from #1 to do instead)
  3. to just update infrequently

Specifically, you have two different package managers (apt and pip) that don't know much about each other and aren't co-ordinated.

I'd recommend you:

  • get a working initial development environment
  • choose some way to be able to clone that environment when you want a new VM (or docker or other) container starting at a working baseline
  • don't update it at all unless there's a specific feature or security update you want
  • don't update it when you actually want to be developing, because you'll get frustrated whenever it doesn't work instantly
  • ideally perform updates in a clone, so you can give up and get back to developing in a working environment if it is more broken than you can face fixing right away
7
  • 12
    "Those are paid full-time jobs" - by all accounts, that's precisely the problem!
    – Steve
    Jan 15 at 16:13
  • 1
    There may be some element of the role expanding to fill the space available, but dev environments are genuinely more complicated than during, say, the thirteen-year interregnum between C++ standards when you might stick with GCC 2.95.x for years at a time, and Perl 5 was used for > 20 years.
    – Useless
    Jan 15 at 16:22
  • 8
    Good answer. Another tip is to seek ways to reduce en.wikipedia.org/wiki/Dependency_hell e.g. don't use pips that are particularly fragile. Add looser binding e.g. environment variables or symlinks that point to apt libs that move. Use a virtualenv per Python project so each one has a minimal set of pips and you can build a new virtualenv (from a requirements.txt file) before deleting the previous one. Try pipenv and conda.
    – Jerry101
    Jan 15 at 17:09
  • 2
    Also: Be ahead with compliance. If you use function or feature foobar and it already brings you a "deprecated" warning, be sure to replace it early and not only after some update gets rid of foobar altogether Jan 16 at 6:47
  • yep, that's why I am always doing updates early Jan 16 at 16:19
3

I avoid updating & upgrading my OS and IDE, unless I know there is something in new versions that will solve my problem.

1
  • 1
    I'd add to that, that I also don't do updates in the middle of a project
    – Peter M
    Aug 26 at 23:00
2

My approach is:

  • update the Linux distro often (daily) - most (even Debian testing) are very stable
  • update python rarely. In my experience it often breaks.
  • as a corollary, install any python packages available from your Ubuntu repos from there - not through pip.
1

In terms of practical solutions, I'd encourage you to have a look at Nix. It's a package manager which makes it possible to isolate the environment for each of your projects from each other and from the base system. In practice this means you can set up a small file like this in your project, listing the development dependencies, and simply run nix-shell in that directory to build and enable all those tools for the remainder of that session. There are also tools which wrap common project-level package managers to install those packages as part of your isolated environment. Basically, you can replace all of pip, npm, apt, docker, make etc. with a declarative setup.

As with all such things, there's an initial learning investment to get started, but after using frankly inferior solutions like Ansible and Puppet for years it's well worth it.

If that sounds interesting you might also be interested in NixOS, which takes it to the next level by configuring the entire system like this. I recently migrated to using Nix on NixOS instead of Puppet, ending up with this configuration. That might look like a lot, but it replaced over 130 files and nine third party modules with 350 lines which do a lot more than the original Puppet.

Not the answer you're looking for? Browse other questions tagged or ask your own question.