Docker instance trashed after restart

Today I started to look at configuring backups for my instance’s database. My instance uses the Docker setup and I had my config in the database. During this process, I, in order, and while my instance was still live…

  • used docker exec -u akkoma akkoma-db-1 pg_dump -Fc akkoma > db.dump from the host - this encountered an error when reaching the config table
  • refreshed my instance page - no errors
  • went into the db container and used SELECT statements and \dt commands in psql, nothing else - SELECT * FROM users; returned ERROR: missing chunk number 0 for toast value 790543 in pg_toast_16391, some other tables threw similar errors, some other tables returned results just fine
  • shut down both the instance and database containers with docker compose down, in case something weird was going on with my database
  • noticed there was a stray database container - stopped it with docker stop
  • started the instance again with docker compose up -d

Now, I cannot make posts, notifications are completely broken, and I no longer have pinned posts on my profile.

docker compose exec db pg_dump -d akkoma --format=custom -f akkoma.pgdump throws a permission denied error; not sure what the path at the end is meant to be in the postgre container, docs just give a placeholder path without elaboration.

At this point, I’ve pretty much accepted the fact that I’ve lost my instance and am willing to restart, but I’d like to see if I can get it in a working and usable state again before I try setting up database backups again.

Overall though, I have a few questions:

  1. Why did just looking at my database, and performing a restart, trash my instance?
  2. Why did the backup command I used error when reaching the config table?
  3. Why did SELECT statements throw errors for some tables in psql?

I’d say your db was already broken when you started to interact with it, tried the export in line 1
All the following selects just made things worse.

I constantly access my db containers directly.
And the path for your db export is likely the home folder of root, given that you didn’t specify a user to connect to the db. Like -u postgres

Sure, but I don’t really understand how just trying to copy info from my database into a backup can break it? Or how just querying info can break it further?

What did I do wrong here that caused the database to break?

I am not a db expert but could see a situation where e.g. a filesystem problem could lead to this.
You run a select *, the db hit’s a bad block and marks it faulty, and as you progress you hit more and more bad blocks until the whole db starts to fall over…