ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/opt/bitnami/mariadb/tmp/mysql.sock' (2)

So yesterday I hosed my Redmine server. It was totally my fault, I stupidly messed up the ownership and permissions on the MariaDB database files.

When I tried to restart the docker stack the redmine could not connect to the database.

First, I went through the database files and fixed all the ownership and permission. This did not solve the problem.

Bring the stack up in the foreground using docker-compose up gave these messages:

rm_app | redmine 15:05:48.80 INFO ==> Restoring persisted Redmine installation
rm_app | redmine 15:05:48.87 INFO ==> Trying to connect to the database server
rm_app | redmine 15:06:49.23 ERROR ==> Could not connect to the database

Using docker exec I connected to the database container. ps aux revealed that the mysql daemon was not running. Trying to connect to the database using mysql gave the following message:

ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/opt/bitnami/mariadb/tmp/mysql.sock' (2)

I Googled this on the Internet, but none of the solutions I found worked.

I created a new redmine stack to see what/how a working system would look like. On the working system ps aux showed mysqld running with the following arguments:

/opt/bitnami/mariadb/sbin/mysqld --defaults-file=/opt/bitnami/mariadb/conf/my.cnf --basedir=/opt/bitnami/mariadb --datadir=/bitnami/mariadb/data --socket=/opt/bitnami/mariadb/tmp/mysql.sock --pid-file=/opt/bitnami/mariadb/tmp/mysqld.pid

So I restarted the broken system connected to the database container, copy and pasted this command to see what would happen, which revealed the following:

<<< A bunch of stuff deleted >>>
...
2024-04-18 15:24:04 0 [Note] Recovering after a crash using tc.log
2024-04-18 15:24:04 0 [ERROR] Bad magic header in tc log
2024-04-18 15:24:04 0 [ERROR] Crash recovery failed. Either correct the problem (if it's, for example, out of memory error) and restart, or delete tc log and start mysqld with --tc-heuristic-recover={commit|rollback}
2024-04-18 15:24:04 0 [ERROR] Can't init tc log
2024-04-18 15:24:04 0 [ERROR] Aborting

I deleted tc.log (located in /bitnami/mariadb/data). Kill the stack and restarted.

This worked!