Docker instance trashed after restart

Support
1

Today I started to look at configuring backups for my instance’s database. My instance uses the Docker setup and I had my config in the database. During this process, I, in order, and while my instance was still live…

  • used docker exec -u akkoma akkoma-db-1 pg_dump -Fc akkoma > db.dump from the host - this encountered an error when reaching the config table
  • refreshed my instance page - no errors
  • went into the db container and used SELECT statements and \dt commands in psql, nothing else - SELECT * FROM users; returned ERROR: missing chunk number 0 for toast value 790543 in pg_toast_16391, some other tables threw similar errors, some other tables returned results just fine
  • shut down both the instance and database containers with docker compose down, in case something weird was going on with my database
  • noticed there was a stray database container - stopped it with docker stop
  • started the instance again with docker compose up -d

Now, I cannot make posts, notifications are completely broken, and I no longer have pinned posts on my profile.

docker compose exec db pg_dump -d akkoma --format=custom -f akkoma.pgdump throws a permission denied error; not sure what the path at the end is meant to be in the postgre container, docs just give a placeholder path without elaboration.

At this point, I’ve pretty much accepted the fact that I’ve lost my instance and am willing to restart, but I’d like to see if I can get it in a working and usable state again before I try setting up database backups again.

Overall though, I have a few questions:

  1. Why did just looking at my database, and performing a restart, trash my instance?
  2. Why did the backup command I used error when reaching the config table?
  3. Why did SELECT statements throw errors for some tables in psql?
2

I’d say your db was already broken when you started to interact with it, tried the export in line 1
All the following selects just made things worse.

I constantly access my db containers directly.
And the path for your db export is likely the home folder of root, given that you didn’t specify a user to connect to the db. Like -u postgres

3

Sure, but I don’t really understand how just trying to copy info from my database into a backup can break it? Or how just querying info can break it further?

What did I do wrong here that caused the database to break?

4

I am not a db expert but could see a situation where e.g. a filesystem problem could lead to this.
You run a select *, the db hit’s a bad block and marks it faulty, and as you progress you hit more and more bad blocks until the whole db starts to fall over…