Updated to latest 2023.08 build but no web UI

Support
1

Hi all,

Recently updated my Akkoma instance to the latest build earlier today but after the upgrade was done, I keep getting Internal Server Error. Troubleshooting the issue, it appears that the updated version of Akkoma no longer listens on port 4000.

I checked using netstat and port 4000 is no longer bound.

At first I thought there was an issue with my Erlang & Elixir install so I updated it to this version:

Erlang/OTP 25 [erts-13.2.2.2] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:1] [jit]

Elixir 1.15.4 (compiled with Erlang/OTP 25)

That didn’t seem to fix the issue and I tried recompiling from the source but that didn’t seem to fix the issue.

I checked my /var/log/akkoma/akkoma.log & there doesn’t seem to be any indication of what’s wrong. Checking journalctl -xe didn’t seem to answer what’s going on either.

Instance specs:

2x Ampere Altra ARM64 cores
8 GB RAM
Ubuntu 22.04 LTS
Installed from source, upgraded via source using standard instructions
Web frontend via Varnish cache & Caddy is the webserver

I don’t think Varnish & Caddy are the issue as the issue seems to stem from Akkoma not listening to port 4000 as that port isn’t being bound anywhere.

Any pointers would be greatly appreciated, thanks!

2

given that the listening port has not changed in the 6ish years the application has existed, I think you may be looking in the wrong place

have you tried running the application in the foreground to see if it says anything of note?

MIX_ENV=prod mix phx.server

I’m personally placing bets on bad config permissions

3

Hi @FloatingGhost,

Thanks for the prompt response, much appreciated. When trying to start Akkoma manually, it seems to just completely crashed out whereas starting it from systemctl seems to run the software but without listening on port 4000.

Not sure why it’s crashing out when running it manually…

I’ve uploaded the console output here: https://pastes.io/nf7fumnkop

Could it be that the installation is located at /opt/akkoma & the home directory is /data/akkoma?

I also checked the mix.exs and the permissions are currently rw-r–r–.

May you please advise?

Thank you!

4

again you’re looking in totally the wrong place, this has nothing do to with home directories?

it sounds like you’ve got a nil somewhere in your config’s :logger, :backends

not sure how you’d get that

this may be able to shine more light on the situation

https://docs.akkoma.dev/stable/administration/CLI_tasks/config/#dump-the-saved-configuration-values-for-a-specific-group-or-key

you’ll want to dump, and potentially delete it the “logger” group

1 Like
5

I am also running into this same stacktrace as Deltatux after upgrading, and the new instance fails to start.

** (Protocol.UndefinedError) protocol Enumerable not implemented for nil of type Atom.
....
 (pleroma 3.10.2-0-gba1ed37) lib/pleroma/config/transfer_task.ex:113: Pleroma.Config.TransferTask.configure/1
 (elixir 1.15.4) lib/enum.ex:984: Enum."-each/2-lists^foreach/1-0-"/2
 (pleroma 3.10.2-0-gba1ed37) lib/pleroma/config/transfer_task.ex:51: Pleroma.Config.TransferTask.load_and_update_env/2
 (pleroma 3.10.2-0-gba1ed37) lib/pleroma/config/transfer_task.ex:34: Pleroma.Config.TransferTask.start_link/1

I have tried dumping the saved configuration to search for nils, but if I dump just the logger group I don’t see any. (nor does anything look off in the Admin-FE).

./bin/pleroma_ctl config dump logger
config :logger, :backends, [:console]

Lemme know if I can provide anything else.

If it helps, this was an instance that was upgraded from pleroma → akkoma (not sure if that matters). I will dig around a bit more. :mag:

6

there is a possibility that it could be something to do with the logger levels changing

if you’ve got anything set to :warn, you might need to set it to :warning

1 Like
7

Thanks! I’ll go looking in the DB manually I guess. According to the Admin-FE they’re all set to :info

8

for reference, it’s this exact call here failing

no idea why that get_env would ever return nil though

9

I find it strange to have a different problem between running manually and running with systemctl. Could it be that some actions (git or mix most probably) were done with another user than the akkoma user? If so, maybe some files have wrong ownership, and now maybe we’re seeing weird bugs due to that. (I think chown -R akkoma:akkoma /opt/akkoma/ should fix that then?)

Another very wild guess is that maybe the build got corrupted by switching between elixir/otp/erlang versions or smthng. If so, maybe recompiling can help. The way I do that is: stop Akkoma, then in the akkoma folder rm -rf ./_build && systemctl restart akkoma and wait for it to recompile. It’s just a wild guess, but I remember years ago I had to do that once because I also had all kind of unexplainable weirdness.

10

oh actually OTP changing is a good call, that could be it if people updated systems

probably the “correct” way to do that is
export MIX_ENV=prod
mix deps.clean --all
mix clean
mix deps.get

then reboot
maybe?

1 Like
11

Apologies, I have failed at including informative troubleshooting details. :man_facepalming:I have to step away from the computer for a bit, but I am happy to try and track this down.

I am using otp releases on debian and systemd.

The process I followed (which I’ve used for past Akkoma upgrades, sans the Debian version bump):

  • Build new server from base image
  • Upgrade bullseye → bookworm (including reboot)
  • Copy instance specific config and static directories
  • Run pleroma_ctl update tasks
  • Try to start the server with systemd

Initially it failed with the config.exs permissions issue, which I resolved (thanks for the helpful error!!).

I’ll try resetting permissions for everything, or maybe even starting from a new Bookworm base container instead of going down the upgrade path.

Thanks again for the additional details.

12

applied a patch - Ensure we can't crash out on unusual logger backend settings · 368b22fd2f - akkoma - Akkoma Development

whilst i am genuinely unsure as to why that call would ever return nil, i guess it’s possible, and hence i’ve a default value in the case that no backends are configured

CI build here
https://ci.akkoma.dev/repos/1/pipeline/2062

from-source can update immediately, OTP wait for your flavour to release above then you should be good to go

1 Like
13

Dang, with the quickness! :raised_hands:

I haven’t tried the fix yet, as I am terribly curious (and this isn’t causing downtime or anything, the old version is still running fine).

I attached to the currently running (old)instance and it gets even more confusing.

iex(pleroma@127.0.0.1)8> Application.get_env(:logger, :backends)
[:console]

iex(pleroma@127.0.0.1)9> Pleroma.Repo.get(Pleroma.ConfigDB, 25)
%Pleroma.ConfigDB{
  __meta__: #Ecto.Schema.Metadata<:loaded, "config">,
  id: 25,
  key: :backends,
  group: :logger,
  value: [:console],
  db: [],
  inserted_at: ~N[2022-11-27 00:54:08],
  updated_at: ~N[2022-11-27 00:54:08]
}

I went reading through the code a bit, and the default returned for Config.Holder.default_config(group, key) is :nil, but I would expect for can_be_merged/2 to return false in that case because nil fails the is_list guard clause, thus returning the value.

Is it possible this is getting executed before the Application env for logger is actually set? :thinking:

14

a quick grep through the codebase shows that the only time we actually touch :logger, :backends is during this transfer task, so… hm

some part of me is nagging that Changelog for Elixir v1.15 — Elixir v1.15.4 might have something to do with it

1 Like
15

ooooo, I think you may be right. It looks like the new logger is doing some “”“”“” shenanigans “”“”“”“” to be backwards compatible with :console :D.

Your patch did resolve the issue for me and I was able to finish upgrading / roll the new image. Thanks again!

16

Hi @FloatingGhost,

Thanks for looking into this, much appreciated. Can you please let me know how I can apply these patches to my existing install?

I also tried doing the cleaning the deps but that didn’t seem to fix it, during the mix.compile, it seems to suggest that there were some warn messages when compiling the pleroma app: https://pastes.io/t0mka2q0xw

Thanks!

17

those are normal, that’s just pheonix being slightly outdated

to update you do exactly the same procedure as normal, no special handling, just git pull or whatever you usually do

18

Hi @FloatingGhost,

So good news is that running it manually works but with tons of warnings: https://pastes.io/zl06bpjjpp

However, if I try starting it from systemctl, it doesn’t & I have no clue why. Is there something wrong with the akkoma.service file? Here’s a copy of my akkoma.service file: https://pastes.io/kxrrkwj3oi

Thanks so much!

19

refer to the above

make sure you’ve set your levels to warning

though this is not fatal

20

@deltatux are you able to systemctl status akkoma.service?

I have better luck with that output when troubleshooting systemd units behaving badly.