A few days ago, I decided to update my Google Compute Engine instance which was running an older Debian version. This was the same instance that is hosting this WordPress blog, so I had to be careful not to break anything lest I lose all the data and/or the service can be down. This post is a recount of my journey through the update process because I found some interesting things about Google Compute Engine (GCE) and how things can break if you are not careful and how things can go very smoothly if you plan properly.


It all started when WordPress's Site Health recommended that I upgrade my php version to 7.x from 5.x. Given the opportunity, I thought I should also upgrade the Debian OS from 8 (Jesse) to 9 (Stretch) on my GCE instance. Before I do anything radical, I wanted to make backups of as many things as possible. I installed a backup plugin in WordPress a while ago which automatically makes a backup of my blog and uploads it to Google Drive. Next I made a snapshot of my GCE instance in Google Cloud Console. Then and only then did I proceed to update my instance.


Normally, to update I would SSH to a GCE instance and just do

sudo apt-get update
sudo apt-get upgrade
sudo apt-get dist-upgrade


But I started seeing errors right from the very beginning. The update was not working because a bunch of repositories were not available anymore. In particular the backports repositories. I was seeing errors like this

W: Failed to fetch http://deb.debian.org/debian/dists/jessie-updates/InRelease Unable to find expected entry 'main/source/Sources' in Release file (Wrong sources.list entry or malformed file)


After some Googling I figured out that they have been moved to archive urls. So made changes to /etc/apt/sources.list as suggested, basically replacing deb.debian.org to archive.debian.org and ran the update again with validity check disabled.

sudo apt-get -o Acquire::Check-Valid-Until=false update


The update was successful. Next I performed sudo apt-get upgrade which also went smoothly. However the sudo apt-get dist-upgrade did nothing. After some more searching I found this helpful article and the dist-upgrade started working. But another roadblock awaited me. For some reason after some packages were updated, I suddenly lost SSH connection to my instance. Not sure if it was after the SSH server was restarted (after an update). Anyway, I could not proceed beyond this point because I was unable to SSH to the instance. Perhaps SSH keys mismatch. The instance was still running and GCE was showing a 50% usage on the machine which led me to believe that something must have stuck hogging the CPU. The WordPress service was still running on the machine and my blog is working fine.


At this point I had no other choice but to bring up another instance. Which means it will cost me money because only one micro instance is allowed to run for free on GCE. I decided to start another f1-micro instance with the boot disk pointing to the snapshot I made earlier.



However, I made one mistake which I realized later. I chose the region as us-west1 and my previous instance was on us-central1, the implications of which, we will find later. I followed the same steps as before, replacing urls and updating. This time everything went quite smoothly and the instance was upgraded to Debian 9. Yay! Now all I need to do is switch traffic from the old instance to the new one. Well not so fast. Since the instances were now running on different locations, I could not port over the static IP of the old instance to the new one :( . I did not install any load balancers because I always had only one instance anyway and moreover the feature is not free. Another deadend.


I deleted the new instance that I created in us-west1 and created another one in us-central1 which was the same region as the old instance. Went through the whole process again and this time I was able to switch the external IP address of the old instance to the new one. Made sure that HTTPS certificates were working, and just like that, my blog was running on a new instance. But I am not done yet. Remember, the reason I started this work was to upgrade my php version to 7.x. Before I began upgrading php, I created another snapshot. Now I have 2 snapshots, and the total size exceeded the free 5 GB limit. Seems like there will be more expenses.


I created another instance from the newly created snapshot and followed this guide to upgrade php. The install went smoothly without any hiccups. After restarting the php service, I checked the site health in WordPress. It complained about some missing php plugins which I promptly installed.

sudo apt-get install php7.2-imagick
sudo apt-get install php7.2-bcmath
sudo apt-get install php7.2-gd


Everything was working fine and I switched the traffic one more time from the Debian 9 PHP 5.x instance to the new instance with Debian 9 and PHP 7.x instance. But soon after the traffic switch, as I was monitoring the health of the instance, I noticed that the instance was running very low on RAM. Well a f1-micro instance really only has 0.6 GB of RAM to start with, but even so, I felt that the resource usage was much higher than I remember with the Debian 8 and php 5.x instance. A cursory look at which process is using the most RAM, quickly point me in the direction of a process running MariaDB which seems to be the successor of MySQL. Some more Google searching later, I found out that I needed to change the performance scheme of MariaDB.


cat /etc/my.cnf
[mysqld]
performance_schema = off


Restarted MariaDB and the RAM usage was much better. Phew! I used to think that writing a blog is taking up a lot of my time. Seems like the maintenance could be also as much work. The journey ends here, but I kept some of the old instances and snapshots around in case I need to rollback. Hopefully, if everything looks good for a couple of weeks, I can delete the instances and snapshots. In the meantime, my billing report shows that my expenses might be higher this month.