If you subscribe to my newsletter (all 6 of you), you’ll know that I’ve been meaning to write a post with my experiences upgrading a lone Docker host from Debian 12 (Bookworm) to Debian 13 (Trixie). Normally I would say this sort of thing doesn’t warrant discussion, but I’ve discovered “One Weird Trick” that made my life so much easier than it used to be.
You see, I like to maintain a small fleet of servers in my free time. If I’m not purchasing new domain names, I’m spinning up new servers somewhere just to run my own simple services. I’ve been doing this for over 20 years across various on-prem and cloud platforms. That means I’ve been through a LOT of Linux distro upgrades.
My main method of service deployment has been installing services manually from the Ubuntu repositories directly into Ubuntu Linux servers. Before Ubuntu, it was Gentoo – when I had LOTS more free time. After server deployment, I maintain the /etc configuration files for all the services I install. This has made my life a living hell of complexity for those 20+ years. Every few years a distro upgrade in Ubuntu comes along and I have to inevitably:
- Backup all my
/etcfiles - Hand merge in all of the
/etcfile changes that came from upstream for the new versions. - Fix all of my services post-upgrade so they run on the new upstream versions.
- Find alternatives for all of the services that were deprecated upstream.
It’s not like this list of services deprecated was ever all that long AT ONE TIME, but the accumulation of 20 years of doing this has finally worn me down. NO MORE! I refuse to perform another Ubuntu dist-upgrade.
I took the path of Debian and Docker, and that has made all the difference. Perhaps I’ll win a prize for “Seven-millionth post where nerd discovers Docker is good.”
I’ll highlight the mostly simple path I now have, and share what I like about it at the end.
Pre-Upgrade Tasks
I had to start somewhere, so why not here.
Reading
First, Read The Friendly Manual. The entire upgrade guide found within the release notes is long, but helpful.
Even more importantly, read the issues to be aware of! This list was long, but because I am able to use a very plain installation of Debian, I didn’t have to really worry about anything here. This is the first sigh of relief.
Backing Up
I still do have some services and configuration in /etc. I could use a tool like etckeeper with a git repo for source control, but honestly, a simple tar -czpf "${DEST_FILE}" --selinux --acls "${BACKUP_SOURCE}" of the /etc directory to a separate mounted partition where my spinning disks live in a RAID gave me enough confidence that I could get back to a known good state were the worst to happen. I’ll reserve source control for the things I care more about.
Next, I backed up the list of packages that were installed on that Debian system. It was small, but I didn’t want to forget something.
dpkg --get-selections '*' > /dest/dir/package-selections-$(date +%Y%m%d).txt
Next, I needed to commit all Docker compose and config files to my local repo. This is the biggest difference between my old systems. Now, everything I CARE about as a running service has a configuration file in my ~/docker/<service> directory. That’s either in a compose.yml, a config file or directory, or a .env file. All of THESE files get committed to the ~/docker git repository. This is just one big repo, instead of a separate repo per service.
I use the exact same simple approach to backing up this git repo. I could send it to another server somewhere and setup a bare git repo, but just copying the whole damn directory to my spinning disk RAID is fine.
I run the following in a script twice a week.
rsync -alv "$SRCDIR" "$DSTDIR"
Now I have a version history of every important-to-my-services config file I’ve changed, and another copy on a pair of disks.
This next part is really where the rubber hits the road, and part of what has made all of my other upgrades in past years so terrible.
Section 4.2 of the release-notes says: “Start from ‘Pure’ Debian.” This means any packages you’ve installed from other repositories should probably be uninstalled or removed somehow before doing this system upgrade. Those packages in the past were often all the important dependencies that I needed to run my services!! What a pain.
How is this different with Debian and Docker? Well, the only packages I really have from another repository are:
- Docker
- Tailscale
- Influx (client)
I didn’t need the Influx client anymore, so that’s easy to remove.
Docker and Tailscale repositories are something I can comment out, without the package system automatically removing them.
This list of just a few small dependencies from other repositories is much easier to deal with than the bigger handful I’ve had in the past. With all of my services running in Docker, I just have to make sure Docker comes back post-upgrade and everything else is managed. Every single service dependency is now managed separately from my distro upgrade!
I took the easy path here and just commented out all the repository list files I didn’t want being managed in the upgrade. I opened these files and put # at the start of all the lines.
/etc/apt/sources.list.d/docker.list
/etc/apt/sources.list.d/tailscale.list
/etc/apt/sources.list.d/influxdata.list # this one I just deletedUpgrade
There were a few tricky upgrade steps, but they weren’t really too bad.
Change Sources
First, Debian recommends you convert to the new debian.sources file format specified in deb822. This was as simple as removing the entries in /etc/apt/sources.list and instead pasting the contents from these instructions into a new /etc/apt/sources.list.d/debian.sources file.
Types: deb
URIs: [https://deb.debian.org/debian](https://deb.debian.org/debian)
Suites: trixie trixie-updates
Components: main non-free-firmware
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg
Types: deb
URIs: [https://security.debian.org/debian-security](https://security.debian.org/debian-security)
Suites: trixie-security
Components: main non-free-firmware
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpgThe important part of this is that the apt lines changed from the keyword of ‘bookworm’ to the keyword of ‘trixie’. The next time you run an upgrade, you’ll reference the new version.
Note that the signing keys inside the pre-upgrade instructions are listed with a .gpg extension. That’ll come back around later.
Two Stage Upgrade
Now we’re ready to follow the two-stage upgrade process. I use tmux for session preservation, just in case my connection to this headless server is interrupted during the upgrade.
Refer to the instructions for the complete list of tasks, but it boils down to:
tmux new-session -A -s upgrade
sudo apt upgrade --without-new-pkgs
sudo apt upgrade full-upgradeThis will upgrade, install, and remove a bunch of packages to get your system from Debian 12 to Debian 13.
Along the way, any package that has an /etc config file change will need fixing. The system will prompt you whether to keep the old etc file and ignore the new upstream goodness, use the new etc file and lose all your configuration, or or attempt a merge in the least user-friendly tool I’ve ever seen for merging. I always choose “Keep existing configuration” and sort it out post-upgrade. I wish this was cleaner to do DURING the upgrade. Maybe this is some upgrade merge command I’m missing. Let me know in the comments.
Post-Upgrade Tasks
That’s mostly it. You can reboot the system and it will come up as a Debian 13 system. Now there is some cleanup to do.
Modernize Sources
For my Docker and Tailscale repositories, I wanted to modernize their sources to the new deb822 format AND upgrade them from bookworm to trixie. I uncommented the lines in the docker.list and tailscale.list files as well as changed bookwork to trixie.
╭─burns@brian /etc/apt/sources.list.d
╰─➤ cat docker.list
deb [arch=amd64 signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian trixie stable
╭─burns@brian /etc/apt/sources.list.d
╰─➤ cat tailscale.list
# Tailscale packages for debian trixie
deb [signed-by=/usr/share/keyrings/tailscale-archive-keyring.gpg] https://pkgs.tailscale.com/stable/debian trixie mainNext I ran the sudo apt modernize-sources command.
This moved each of those .list files to a .list.bak and made a new docker.sources and tailscale.sources file. Ignore the .pgp extensions. We’ll get to that in a bit.
╭─burns@brian /etc/apt/sources.list.d
╰─➤ cat tailscale.sources
Types: deb
URIs: https://pkgs.tailscale.com/stable/debian/
Suites: trixie
Components: main
Signed-By: /usr/share/keyrings/tailscale-archive-keyring.pgp
╭─burns@brian /etc/apt/sources.list.d
╰─➤ cat docker.sources
Types: deb
URIs: https://download.docker.com/linux/debian/
Suites: trixie
Components: stable
Signed-By: /usr/share/keyrings/docker.pgpNow I could run a sudo apt update and sudo apt upgrade, which did install some new Docker versions. All my containerized services came back though.
Merge Config Files
Remember all those config files we ignored during the upgrade? The upgrade helpfully keeps the newer version around for you to deal with later.
You can find them all by their extension *.ucf-dst inside the /etc/ directory with this command:
sudo find /etc -type f -name '*.ucf-dist'
I had the following to fix, merging in the upstream distro config with my customized config:
- sshd_config – I disable password login
- 50_unattended-upgrades – I customize the email notifications
- smb.conf – Lots of config for my home shares
SSH, Unattended Upgrades, and Samba are critical services for me, so it was important to merge any new config from the upgrade into my existing config CAREFULLY. If I deleted or altered config lines I could break my services.
I’ve grown way too accustomed to GUI merge tools, so I had to dust off my vim skills and look up vimdiff. Here’s what I learned, that I’m sure to forget again:
sudo vimdiff oldfile newfile
Use ctrl-w to move the cursor between left and right panes. Everything is relative, so know where you are.
Use ]c to jump to the next diff
Use [c to jump back to the previous diff
Use dp to put the change under the cursor into the OTHER pane.
Use do to get the change from the OTHER pane and pull it into this one.
This made a lot of sense once I did it a few times. You just have to know which window you’re in! In my case, it was helpful to keep my cursor inside the old config file and just use do to pull in ONLY the diffs I wanted from the *-ucf-dist file. These diffs didn’t really turn out to be very meaningful. Just some comments and instructions with no behavior changes.
Once you’re happy that your old config file looks the way you want it, you can restart or reload your service and once successful, delete the *-ucf-dist file.
Switch keys to .pgp format
After the upgrade, I read that the keys inside /usr/share/keyrings were moving from .gpg extensions to .pgp extensions. This was a great reminder that I had sources still referencing /etc/apt/keyrings (from the Docker instructions) and I had best consolidate everything to one directory. This conversion is not strictly necessary – but the warning text tells me .gpg will be deprecated, and I just felt like preparing early.
All of the Debian 13 key files are moved to .pgp extension after the upgrade, and there are symlinked .gpg files hanging around.
sudo mv /etc/apt/keyrings/docker.gpg /usr/share/keyrings/docker.pgp #rename docker key
sudo mv /usr/share/keyrings/tailscale-archive-keyring.gpg /usr/share/keyrings/tailscale-archive-keyring.pgp #rename tailscale key
sudo vim /etc/apt/sources.list.d/docker.sources # change ref to .pgp location
sudo vim /etc/apt/sources.list.d/debian.sources # change ref to .pgp
sudo vim /etc/apt/sources.list.d/tailscale.sources # change ref to .pgp<br>Faillog no longer needed
This one is pretty simple, the faillog is no longer needed, and there are other commands to look at failed logins. You can delete this file. This is described here in the removal of last, lastlog, and the Year 2038 problem these and similar packages will have.
sudo rm /var/log/faillog
This is a great opportunity to check out commands like: w, who, and lslogins.
These are files that I’ve seen on Linux systems for decades. There’s probably not some slew of other systems that have been written to depend on these, right?! 😉
Clean out old tmp
Since I am only human, I failed to read the instructions completely BEFORE rebooting. Now I have to take an extra step. Because of changes in Debian 13, the /tmp directory is now mounted as a tmpfs in memory instead of on disk. It’s also auto-deleted more frequently. This change is completed as soon as you reboot. You should delete the files on disk in /tmp before you reboot, because the tmpfs gets mounted over that location and you won’t be able to EASILY get at them to delete them after the reboot.
If you’re reading this, though, there is a solution clearly laid out in the release-notes.
I had to modify the instructions slightly, because they tell you to mount right over /mnt. That would be a disaster for me since I have other things there. I just made a temporary /mnt/cleanup for this purpose.
sudo mkdir /mnt/cleanup
sudo mount --bind / /mnt/cleanup
ls /mnt/cleanup/tmp
# two files listed
ls /tmp
# lists lots of files, but not the two above
sudo rm /mnt/cleanup/tmp/<file names>
sudo umount /mnt/cleanup
sudo rm -rf /mnt/cleanupNot fixed yet
I use Telegraf and it can no longer get the count of logged in users on this server, because Debian 13 removed /var/run/utmp. Careful readers will see that I foreshadowed this 2038 problem. Telegraf has an upstream dependency on gopsutils, which relies on the utmp file. I’m just tracking that and expecting some eventual Telegraf update. No big deal I suppose. I just can’t see a nice graph of when I logged in and with how many shells.
Parting Thoughts
Looking back at this, it doesn’t seem easy. However, I swear that this was a walk in the park compared to every pre-Docker Ubuntu distribution upgrade I’ve ever done. I’m MUCH happier with the above upgrade.
The secret sauce is that Docker has allowed me to separate the dependencies of the applications from the dependencies of the underlying host. During all of these host distro upgrades, my Docker services came right back up after reboot.
Previous distro upgrades would completely break my running applications, because they upgraded or removed critical dependencies that the apps relied on.
Second, I think that Debian does an awesome job writing their guides. The level of detail and technical instruction is exactly what I’m looking for. With that guide in hand I had everything I needed to plan and then run the upgrade. I felt like all of my past Ubuntu upgrades were weekends spent lost with Google and Stack Exchange. This is a breath of fresh air.
Leave a Reply