How to solve MySQL problem "upgrade in progress"

How to solve MySQL update loop?

How to solve MySQL problem "upgrade in progress"
Photo by Caspar Camille Rubin / Unsplash

Short Answer

In my case, the solution is to update mysqld.

I got this same issue today and this is How I solved this.

First of all I stopped mysql: service mysql stop.

After stopping mysql I ran MYSQLD with upgrade: /usr/sbin/mysqld --upgrade=minimal  Make sure upgrade argument is minimal. [https://serverfault.com/questions/1130794/mysql-stopped-working-on-ubuntu-22-04-failed-to-upgrade-server-error]

Context

cup of coffe and laptop computer
Photo by Alessandro Bianchi / Unsplash

During my coffee break just a few minutes after lunch, I randomly check my blog post for inspiration.

However, I immediately see that my blog isn't operational; 502 Bad Gateway error.

My first reaction is that my blog is down because I may have missed a monthly payment. I promptly go to Digital Ocean and make a payment.

I restart my server - wait a few minutes - still the same error.

As I need to continue work, I postpone any research for later.

Problem statement

The following day, I glance at my blog again; the same error. A bit confused, I start to investigate further as I haven't updated anything.

I log into my server and use ghost cli utility to understand the issue. Ghost Doctor indicates that everything else is fine apart from my database connection.

Using the command sudo service mysql status, I see that indeed my mysql service is stuck with the "Upgrade in progress" message. At this point, I think that this is normal and that I should wait. Yet, after hours, it's still stuck.

What I attempted

I restart the mysql service using sudo service mysql restart. Nevertheless, this is not possible as there's an socket lock error. Looking online, I see that this is a common issue and there are many solutions. (way too many actually)

The first answer I read on StackExchange seems relevant to this particular issue  MySQL Update in Progress and looks promising. Nevertheless, in the end, the commenter describes the process of backing up the database and restoring it. For me, this looks like too much work and I skip it. (Plot twist: I end up doing this anyway and it works)

Whenever I use the mysql command, I still encounter the mysql socket lock error.  Therefore, I cannot backup the database.

Here's a list of things I try (using a brute force approach):

  • I restart the system; no progress (yes this is an Ubuntu server I know)
  • I try to kill the mysql process using sudo killall -9 mysql and then manually update mysql - still stuck with the lock error
  • I update the system with the commands sudo apt-get update and sudo apt-get upgrade - still stuck with the lock and upgrade in progress error

At this point, I don't dare try to directly update Ghost but rather take my time to understand and fix the root cause.

Quick fix solution

Meanwhile, I experiment with an alternative to showing the 502 error. I use Cloudflare to redirect my domain to a landing page I create on Vercel.

I use the following steps:

  • I connect my Github account to Vercel
  • Using React with ViteJS, I code a new landing page
  • I copy and paste links for my social media accounts including LinkedIn and Medium, which I cross-post my posts on
  • I deploy the landing page on Vercel for free (you can alternatively use Netlify or GitHub pages)
  • I add a new page Redirect rule on Cloudflare to redirect my domain to the new landing page

What I learned

Photo by Kelly Sikkema / Unsplash

During this process, I learned two things in particular:

  1. I have no idea how and why Ghost and MySQL decided to update automatically
  2. People are bad at estimation

What I've noticed throughout my career as a professional Software Developer, is that people are generally bad at estimation - including myself. I thought this update would take an incredible amount of time and therefore kept procrastinating.

Working on the landing page and setting up the redirect rule took me less than 20 minutes, which I did during a relaxing tea break.

In hindsight, if I attempted the first part of the solution as described above, this would have taken me less than 5 minutes. This is something JoKi mentionned in one of my LinkedIn posts.

I should have listened to him and addressed the real root cause.

What I will do next

Photo by Jess Bailey / Unsplash

Now my next move is to update my Blog from the current version, which I think is Ghost 4 to the latest one Ghost 5 (at the time of this writing).

My safe bet is to create a new droplet on Digital Ocean and then migrate my current blog to the new one.

Another possibility is to back up everything on my current droplet and then update Ghost and most probably the NodeJS version. Once done, I can then restore the backup.

I'm also tempted to just duplicate the droplet and then try updating Ghost and NodeJS on the new droplet. If everything works fine, I can then delete the old droplet and in Cloudflare, I can then redirect my domain to the new droplet. Otherwise, I just delete the new one and restart it again.


Resources

Mysql stopped working on Ubuntu 22.04. Failed to upgrade server. Error
I have a basic DigitalOcean droplet (1gb ram and 1 cpu) that has a few websites on it. There are a couple nodejs sites and a couple of Wordpress sites. All are pretty low traffic. It normally works...

Stackoverflow

Digital Ocean

Vercel

Cloudflare

Ghost

NodeJS

MySQL

React

ViteJS

How To Backup MySQL Databases on an Ubuntu VPS | DigitalOcean
MySQL is a popular database management system that is often used to manage the data from websites and applications. This guide will demonstrate how to backu…