I neglected one of my Linux instances for too long. Life is busy, things get out of date, it happens. When new, this instance was installed with Fedora 24, and the OS was never upgraded again. I set up a website on it, set a script to automatically update most packages as they became available through the package management system, and mostly didn’t think about it.

Then one day I started getting notifications that my TLS cert couldn’t be renewed, because the software I was using to renew it was too outdated: the protocol had changed, and I hadn’t kept up. It wasn’t even easy to install the new versions of the script because the OS I was running was too stale.

Rawhide, Yum, and Node

“Gardens are not made by singing ‘Oh, how beautiful!’ and sitting in the shade.” –Rudyard Kipling

Time to upgrade.

I didn’t have much experience upgrading Fedora. I think I had done it once before, many years ago, and I was pretty unsure of the process. So I checked to see I had off-instance backups, found some instructions, and got going.

The first couple upgrades – Fedora 24 to 25, and then 25 to 26 – went fine. Then I got off track. I followed some instructions that switched me to Rawhide, thinking it was the route to Fedora 27 – a misunderstanding of what Rawhide was. I quickly realized my mistake, but didn’t immediately know how to get back. I managed to get off Rawhide quickly. But couldn’t figure out how to get on anything else. My attempts to upgrade now accomplished nothing: dnf kept telling me it had “nothing to do”, and my system-upgrade reboot calls didn’t get my anywhere.

Dependencies resolved.
Nothing to do.
Complete!

Even though dnf has replaced yum, some of the yum infrastructure remains in place. It’s not worth renaming/moving everything, right? grep enabled /etc/yum.repos.d/*.repo shows which repositories are enabled on Fedora: and I had none enabled.

Once I set enabled=1 in fedora and fedora-updates, I finally saw updates again. I was able to resume the good instructions published in the Fedora Magazine, which is the source I recommend users follow when upgrading their systems. For additional information when you encounter problems, see the DNF System Upgrade topic on the Fedora wiki.

One of those steps errored for me. yum-utils was blocking the installation of dnf-utils. Well that’s an easy decision: dnf remove yum-utils. Onward… It seemed like everything was fine, but system-upgrade reboot still accomplished nothing. It seemed like no matter what I tried, the number returned by cat /etc/fedora-release would never tick up.

So after a failed system-upgrade reboot, I poured through journalctl logs until I found the problem. Buried about 2,600 lines back was this gem:

systemd-journald[374]: Suppressed 1 messages from /system.slice/dnf-system-upgrade.service
  Running scriptlet: filesystem-3.3-3.fc27.x86_64                           1/1
  Running scriptlet: npm-1:6.4.1-1.8.12.0.1.fc27.x86_64                     1/1
  Running scriptlet: nodejs-inherits-2.0.3-2.fc27.noarch                    1/1
  Preparing        :                                                        1/1Transaction
    couldn't start:
file /usr/lib/node_modules/npm/doc from install of npm-1:6.4.1-1.8.12.0.1.fc27.x86_64 conflicts
  with file from package nodejs-2:8.2.1-1nodesource.fc24.x86_64
file /usr/lib/node_modules/npm/html from install of npm-1:6.4.1-1.8.12.0.1.fc27.x86_64 conflicts
  with file from package nodejs-2:8.2.1-1nodesource.fc24.x86_64
file /usr/lib/node_modules/npm/man from install of npm-1:6.4.1-1.8.12.0.1.fc27.x86_64 conflicts
  with file from package nodejs-2:8.2.1-1nodesource.fc24.x86_64
Error: Could not run transaction.
systemd[1]: dnf-system-upgrade.service: Main process exited, code=exited, status=1/FAILURE

Really? The website was down because I couldn’t finish upgrading the OS, and move on to resolving the issues caused by upgrading. I couldn’t finish upgrading because of some Nodejs crap? (No offense…) I didn’t even use Nodejs, I had just installed it some long time ago because I was considering writing a Zapier script in it – a direction I eventually abandoned in favor of a simple HTTP request from Zapier containing data I would process on another server, rather than in a Zapier Zap step.

In short: npm destroyed my Linux box. (Mostly joking here.)

I fixed this with dnf remove nodejs. Worked great. After that I was able to finish my upgrades smoothly, and get cat /etc/fedora-release to tick up to 29.

PostgreSQL Won’t Start

So that’s great. But why is the website still down? Ah: it can’t access the database. Postgres can’t start:

An old version ‘9.5’ of the database format was found. You need to dump and reload before using PostgreSQL 10.7.

No problem, I’ve got backups, right? Well about that…

I had relied on my automated backup script for my pre-upgrade backups. It creates a new backup every hour. It keeps a cap of how many backup copies to store at a time, and deletes some of the old ones when it goes beyond the cap. In effect, this means it’s deleting one backup every time it places a new backup.

Unfortunately the system was in mid-upgrade limbo for an extended period of time. The best backups had been deleted by the script: replaced by backups created from an inaccessible database: worthless, practically empty gzip files.

I had older backup files, but there were some data changes that had occurred since they were recorded.

So the best data available was in /var/lib/pgsql. Maybe postgres couldn’t access it, but I could. The first thing I did was rename it to pgsql9.5. Each time I wanted to try something with it, I would make a recursive copy to pgsql.

I did some research on my situation. I quickly concluded that I had to install the old version of PostgreSQL again. This wasn’t available on mirrors in my enabled repositories. I had to turn to Koji. I found the latest version of PostgreSQL released on Fedora 24. It was postgresql-9.5.7-1.fc24.

First dnf remove postgresql. Now I used wget to download the fc24 RPMs. I installed them with dnf install in the order postgresql-libs, postgresql, postgresql-server. But the service still failed to start. It said the data directory wasn’t found. How was that? I could see it…

Ah: when I had copied /var/lib/pgsql9.5 to /var/lib/pgsql I had neglected to su postgres: they were all user/group root. So chown -R postgres:postgres pgsql resolved the issue, and the postgres service started up without error.

pg_dumpall

Okay, now I had my old database running. But I still wanted to migrate to the much newer version of PostgreSQL in Fedora 29. Fortunately, exporting the entire database cluster was easy:

pg_dumpall -f 2019.04.06-01.sql

There are several posts online recommending -U postgres and other options. I didn’t need them.

That was it – now I could remove postgres again, upgrade to the latest version, initdb, and import my postgres cluster. Here it is in the brevity of linux commands:

su
systemctl stop postgres
dnf remove postgresql
dnf remove postgresql-libs
cd /var/lib
rm -rf pgsql
dnf install postgresql
dnf install postgresql-server
su postgres
initdb /var/lib/pgsql/data
exit
systemctl start postgresql
su postgres
psql -f 2019.04.06-01.sql postgres

The one exit in there got me back to root. That was all it took. With my database back online the site was back up, but the cert had expired. Time to fix that:

dnf install python3-certbot-apache
certbot renew --apache

And voila! The site was fixed!

Upgrading Go and Gogs

I was on a roll, and saw the the version of Go on this Linux node was 1.11.4. That wouldn’t do: 1.11.5 was a security patch. And while I’m at it, I might as well go to 1.12.2, which had been released the day before.

Go installs to /usr/local/go. I usually make that a symbolic link to some specific version, so I can easily keep multiple versions installed and switch between them. But I hadn’t done that on this Linux instance, so that was the first step.

cd /usr/local
mv go go1.11.4

Then I downloaded Go and installed per usual, and then finished my setup:

mv go go1.12.2
ln -s go1.12.2 go

Restarting Gogs at that point got the service working again – did the Go reinstall fix it, or had it just been the postgres issue? Probably the latter. I saw I was two Gogs versions behind there too, and one of them had security updates, so might as well upgrade that:

cd /home/gogs/installed
su gogs
mv gogs gogs_old
wget https://github.com/gogs/gogs/releases/download/v0.11.86/linux_amd64.tar.gz
tar -xzvf linux_amd64.tar.gz
cp -R gogs_old/{custom,data,log} gogs
systemctl gogs start
journalctl -u gogs
Apr 06 18:11:22 fedora gogs[3707]: 2019/04/06 18:11:22 [ INFO] Gogs 0.11.86.0130

The morals of the story?