Debian - Backports, Other Repositories, and Pinning

The two Linux distributions I use the most are Debian and Fedora. I prefer Debian, but I've found there are machines on which it either won't install or doesn't detect some hardware (in my life, this has mostly been touchpads on laptops). That's where Fedora steps in. I've written previously about Package Management on both Fedora and Debian: today's post builds on that for Debian.

Understanding Debian Versions

At any given time, Debian has several versions of the OS on the go. The main one right now is "stretch," the current official release. The current development version of Debian is "buster," and the old stable version (still supported, still undoubtedly in use on many servers) is "jessie." For those not familiar, Debian names its releases entirely after characters in the Pixar "Toy Story" series of movies. But there's another naming scheme you have to get your head around: the current release is called "stable," the upcoming release that's currently in development is called "testing," and the previous release is called "oldstable." When "buster" (currently "testing") is officially released (probably in a couple years), things change: "buster" becomes "stable," "stretch" becomes "oldstable," and the new "testing" distribution will be called "bullseye." Unfortunately, I'm not done confusing you with names yet. What happens, you may wonder, to "jessie" when "stretch" is released? It becomes "oldoldstable." I haven't yet mentioned the development channels for really new and untested packages either. It used to be that there was only one: it was called "unstable" and the official release name was "sid" (if you remember the original "Toy Story" movie, "Sid" was the kid who destroyed toys). But this is where the naming gets weird: when all the other distro names move down the stack, "sid" is always "unstable." Packages are put in "sid" when they're too new and untested for the "testing" distro. And finally there's a channel for packages too unstable for "sid" - it's called "experimental."

I've learned to not use sid/unstable as a machine platform. The package thrash is absolutely astonishing: dozens a day. If you leave a sid machine disconnected for a couple months, you'll have to download a couple of Gigabytes of files if you run apt-get dist-upgrade - and the end result is still going to be a system of dubious reliability. It's a test bed for developers and not meant for users unless you're really eager to help with debugging the OS.

I usually run systems as "stable," but the longer it's been since the release, the older the software looks: once it becomes the official 'stable' distribution, no new software is added, and existing software versions aren't incremented. The only changes made are security patches.

Debian Backports

Where this lack of new software is most troublesome is for things like Firefox on a desktop, or certbot on a server when we were on jessie: certbot was never part of jessie, but I needed it on my stable servers and it was available in stretch (then testing). With Firefox, they have between four and six releases a year but Debian stable locks onto one version for the approximately 2.5 years of the release's lifespan. You can look at Debian Backports, but the packages there seem to be fairly system-oriented. It seems you can always get a new kernel, but newer packages for other purposes seem to be a matter of personal interest among developers. If a dev feels like supporting a testing version of some package on stable, it will show up here, but there are no guarantees. If you can find what you need in backports, follow the instructions Debian provides: it's fairly straight forward. That's how I got certbot.

I was looking for "synergy:" the Debian stable version is 1.4.16, whereas the current Fedora version is 1.8.8 and I suspect that's why I don't have a shared clipboard between two laptops I link together frequently. But synergy isn't in backports, so I have to resort to a trickier method of getting a newer package: actually adding testing repositories to my stable machine. This is a dangerous thing to do because if apt-get starts installing new packages, it can often open the flood gates. The problem is, a new build of a package often requires new builds of the shared libraries. And if you upgrade the libraries, other packages that rely on those libraries may be upgraded too. I'm ignoring the fact that you can have multiple copies of a given library installed, which mostly avoids this problem: but adding repos from different versions of Debian can have some weird and destabilizing effects.

Adding a Repository From Another Version

To get yourself fully up-to-date before messing with your package lists, you should run an apt-get dist-upgrade. This will get you a baseline. If you run it again, it should tell you there's nothing to upgrade.

I've updated my repository list on the stable machine I'm trying to get the newer synergy on:

# /etc/apt/sources.list
deb http://ftp.us.debian.org/debian/ stretch main contrib non-free
deb http://ftp.us.debian.org/debian/ buster main contrib non-free  # <- NEW LINE!
deb http://security.debian.org/ stretch/updates main contrib non-free
deb http://ftp.us.debian.org/debian/ stretch-updates main contrib non-free
deb http://www.deb-multimedia.org stretch main

If you have a lot of lines in your sources.list file, you should consider putting some of the non-standard ones under /etc/apt/sources.list.d/ in individual text files, but my list is short and it's easier to deal with and understand in one place.

The first thing you should do after updating /etc/apt/sources.list:

# apt-get update
...

But I find myself with a significant problem: if I try apt-get dist-upgrade I would hope for a zero return - but Apt wants to upgrade 753 packages and install 93 new ones. Essentially it wants to change the entire OS. Removing that one line and refreshing it gets me back to zero.

Apt-Pinning

Apt-pinning is the process of assigning priorities to different repositories so you get the right packages from the right place. But there's a really big problem: as already mentioned, Debian's primary documentation on Apt-pinning contains more disclaimers and updates by users in disagreement with the primary documentation than it does the original document. Which suggests that NOBODY actually understands this, and the documentation is just vague enough to make it really hard to figure out. So I did this by the empirical method. To overcome buster's eagerness, I added this file:

# /etc/apt/preferences.d/buster
Package: *
Pin: release o=Debian,n=buster
Pin-Priority: 300

This led to an interesting result:

Reading package lists...
Building dependency tree...
Reading state information...
Calculating upgrade...
The following packages will be upgraded:
  virtualbox-dkms virtualbox-guest-dkms virtualbox-guest-utils
3 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 1,727 kB of archives.
After this operation, 4,325 kB of additional disk space will be used.
Do you want to continue? [Y/n] n
Abort.

Why? I suspect this is because there are no install candidates in stretch for these packages (either they're no longer available or I installed them by hand from another source - very likely with VirtualBox) and there are candidates in buster that I've now added 300 points to - so it figures the new ones are higher priority than the installed ones. Try again:

# /etc/apt/preferences.d/buster
Package: *
Pin: release o=Debian,n=buster
Pin-Priority: -10

apt-get dist-upgrade now shows me what I want to see: nothing to be installed. A negative priority means "never install this stuff."

Here's a really useful command: use apt-cache policy <pkg-name> liberally.

# apt-cache policy synergy
synergy:
  Installed: 1.4.16-2
  Candidate: 1.4.16-2
  Version table:
     1.8.8-stable+dfsg.1-1 500
        -10 http://ftp.us.debian.org/debian buster/main amd64 Packages
 *** 1.4.16-2 500
        500 http://ftp.us.debian.org/debian stretch/main amd64 Packages
        100 /var/lib/dpkg/status

The important part here is the string of asterisks beside version 1.4.16-2: this is the version that will be installed (on this system it's already installed, but you get the idea). Notice though that version 1.8.8 is available. But the package from the current distribution is favoured.

If you read the Debian backports Instructions above, you've already seen one possible solution that you can use at this point:

# apt-get -t buster install synergy  # don't do this yet, keep reading

This says "use the version of synergy from the buster release." Let's use the '--dry-run' option to find out what would actually happen:

# apt-get --dry-run -t buster install synergy
Reading package lists... Done
Building dependency tree...
Reading state information... Done
The following package was automatically installed and is no longer required:
  libcrypto++6
Use 'apt autoremove' to remove it.
The following additional packages will be installed:
  libavahi-compat-libdnssd1
The following NEW packages will be installed:
  libavahi-compat-libdnssd1
The following packages will be upgraded:
  synergy
1 upgraded, 1 newly installed, 0 to remove and 754 not upgraded.
Inst libavahi-compat-libdnssd1 (0.7-3 Debian:testing [amd64])
Inst synergy [1.4.16-2] (1.8.8-stable+dfsg.1-1 Debian:testing [amd64])
Conf libavahi-compat-libdnssd1 (0.7-3 Debian:testing [amd64])
Conf synergy (1.8.8-stable+dfsg.1-1 Debian:testing [amd64])

There's another very similar-looking command that https://wiki.debian.org/AptPreferences tells me is significantly different:

# apt-get --dry-run install synergy/buster
Reading package lists... Done
Building dependency tree...
Reading state information... Done
Selected version '1.8.8-stable+dfsg.1-1' (Debian:testing [amd64]) for 'synergy'
The following package was automatically installed and is no longer required:
  libcrypto++6
Use 'apt autoremove' to remove it.
The following additional packages will be installed:
  libavahi-compat-libdnssd1
The following NEW packages will be installed:
  libavahi-compat-libdnssd1
The following packages will be upgraded:
  synergy
1 upgraded, 1 newly installed, 0 to remove and 754 not upgraded.
Inst libavahi-compat-libdnssd1 (0.7-3 Debian:testing [amd64])
Inst synergy [1.4.16-2] (1.8.8-stable+dfsg.1-1 Debian:testing [amd64])
Conf libavahi-compat-libdnssd1 (0.7-3 Debian:testing [amd64])
Conf synergy (1.8.8-stable+dfsg.1-1 Debian:testing [amd64])

Unfortunately, as an example, 'synergy' is apparently the wrong package to use here: the results are phrased slightly differently but the outcome is exactly the same. Or it could be that the information on that page is wrong: it's not exactly a masterpiece of technical documentation. Here's the claim: "['synergy/buster' without '-t'] will not attempt to upgrade any packages on your system, so if specific dependencies are not met, the install will fail. ['-t buster '] will attempt to install/upgrade any dependencies." What we see above is that there appear to be no libraries that need to be upgraded - only new packages to be installed. ('libcrypto++6' is a dependency of the old version of synergy, and it will be obsolete once this upgrade is done.)

This would probably achieve what I want, but the OS would in effect "fight it" in the future as the installed version would never be the prioritized version. This probably won't cause problems, but I'd like to see if I can handle this the way it's meant to be done.

# /etc/apt/preferences.d/synergy
Package: synergy
Pin: release o=Debian,n=buster
Pin-Priority: 650

This leads to the outcome I want. I chose 650 as a priority because we've already weighted buster as -10, and this needs to weigh in at more than the 500+100 that packages from stretch are assigned by default.

# apt-cache policy synergy
synergy:
  Installed: 1.4.16-2
  Candidate: 1.8.8-stable+dfsg.1-1
  Version table:
     1.8.8-stable+dfsg.1-1 650
        -10 http://ftp.us.debian.org/debian buster/main amd64 Packages
 *** 1.4.16-2 500
        500 http://ftp.us.debian.org/debian stretch/main amd64 Packages
        100 /var/lib/dpkg/status

And:

# apt-get --dry-run dist-upgrade
Reading package lists... Done
Building dependency tree...
Reading state information... Done
Calculating upgrade... Done
The following package was automatically installed and is no longer required:
  libcrypto++6
Use 'apt autoremove' to remove it.
The following NEW packages will be installed:
  libavahi-compat-libdnssd1
The following packages will be upgraded:
  synergy
1 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Inst libavahi-compat-libdnssd1 (0.6.32-2 Debian:9.2/stable [amd64])
Inst synergy [1.4.16-2] (1.8.8-stable+dfsg.1-1 Debian:testing [amd64])
Conf libavahi-compat-libdnssd1 (0.6.32-2 Debian:9.2/stable [amd64])
Conf synergy (1.8.8-stable+dfsg.1-1 Debian:testing [amd64])

So I seem to have my outcome (notice the nice touch that the library comes from stable).

If you're going to do anything more complex than this, Read that horrible web page ( https://wiki.debian.org/AptPreferences ) through a couple times and make sure you get some sense of it: as bad as it is, there's nothing better on the web that I could find.

P.S. After the upgrade, Synergy finally has bi-directional clipboard sharing.