Akkoma Instance Suddenly Crashing, Won't Boot

Support
1

Running 24.04.1/3.13.2 on FreeBSD 13.4, my instance was running fine earlier today but has now crashed and won’t reboot.

I get this error multiple times when I try to start:

07:06:02.845 [error] GenServer Pleroma.Stats terminating
** (Postgrex.Error) ERROR XX000 (internal_error) failed to look up symbol "evalexpr_0_2": Failed to materialize symbols: { (main, { evalexpr_0_5, evalexpr_0_10, evalexpr_0_9, evalexpr_0_0, evalexpr_0_2, evalexpr_0_13, evalexpr_0_8, evalexpr_0_4 }) }
    (ecto_sql 3.10.2) lib/ecto/adapters/sql.ex:1047: Ecto.Adapters.SQL.raise_sql_call_error/1
    (ecto_sql 3.10.2) lib/ecto/adapters/sql.ex:945: Ecto.Adapters.SQL.execute/6
    (ecto 3.10.3) lib/ecto/repo/queryable.ex:229: Ecto.Repo.Queryable.execute/4
    (ecto 3.10.3) lib/ecto/repo/queryable.ex:19: Ecto.Repo.Queryable.all/3
    (pleroma 3.13.2) lib/pleroma/stats.ex:73: Pleroma.Stats.calculate_stat_data/0
    (pleroma 3.13.2) lib/pleroma/stats.ex:121: Pleroma.Stats.handle_continue/2
    (stdlib 4.3.1.3) gen_server.erl:1123: :gen_server.try_dispatch/4
    (stdlib 4.3.1.3) gen_server.erl:865: :gen_server.loop/7
Last message: {:continue, :calculate_stats}
State: nil

Then I get this error and Akkoma gives up:

07:06:02.254 [error] Error when calling MFA defined by measurement: Pleroma.Web.Telemetry :instance_stats []
Class=:exit
Reason={:noproc, {GenServer, :call, [Pleroma.Stats, :get_state, 5000]}}
Stacktrace=[
  {GenServer, :call, 3, [file: ~c"lib/gen_server.ex", line: 1063]},
  {Pleroma.Stats, :get_stats, 0, [file: ~c"lib/pleroma/stats.ex", line: 45]},
  {Pleroma.Web.Telemetry, :instance_stats, 0,
   [file: ~c"lib/pleroma/web/telemetry.ex", line: 232]},
  {:telemetry_poller, :make_measurement, 1,
   [
     file: ~c"/skel/home/akkoma/akkoma/deps/telemetry_poller/src/telemetry_poller.erl",
     line: 336
   ]},
  {:telemetry_poller, :"-make_measurements_and_filter_misbehaving/1-lc$^0/1-0-",
   1,
   [
     file: ~c"/skel/home/akkoma/akkoma/deps/telemetry_poller/src/telemetry_poller.erl",
     line: 332
   ]},
  {:telemetry_poller, :handle_info, 2,
   [
     file: ~c"/skel/home/akkoma/akkoma/deps/telemetry_poller/src/telemetry_poller.erl",
     line: 354
   ]},
  {:gen_server, :try_dispatch, 4, [file: ~c"gen_server.erl", line: 1123]},
  {:gen_server, :handle_msg, 6, [file: ~c"gen_server.erl", line: 1200]}
]

I also get logs about excessive memory use, but when I check top the size doesn’t get larger than my memory, not to mention my swap.

Dump file available on request

2

I still don’t know why this happened but I did some deferred maintenance on my server and updated Akkoma in the process, which seems to have solved the problem.