Akkoma stable 2024.03 - Securer? I barely know her!

Hey there, quite an important one for you here today. We’ve got ourself a vulnerability and it’s time to make like a tree and patch. Not sure what that means but that’s neither here nor there.

I won’t drone on, let’s get to the interesting stuff

if you don’t care about details and just want to cover yourself, skip down to “Updating”

New stuff

Security Fixes

The big things:

  • Content-Type headers are sanitised for emoji, proxied files and user uploads from LocalUploader, in particular it’s no longer possible to make Akkoma serve such files as ActivityPub objects
  • fetched ActivityPub object now need strict id/url matches

These two enabled impersonation other users and likely affect every AP-capable version of *oma ever, so do upgrade now!

But there’s more:

  • new CLI tasks to best-effort check if impersonation was done or attempted (this won’t get 100% of things, but will look for what we expect to see if anyone actually found this before us)
  • when fetching AP objects cross-domain redirects are now disallowed; it is no longer possible for the final destination domain to pose as the initial fetch domain (e.g. media.akkoma.dev pretending to be otp.akkoma.dev)
  • some proactive hardening against any more path traversal attacks in emoji handling (no known exploit)
  • tighter checks for user objects
  • URLs of stolen emoji from StealEmoji are no longer predictable
  • refetched objects can no longer attribute themselves to foreign actor

Other

  • dropped the same-domain default for media proxy and local uploader (see upgrade notes)
  • StealEmoji
    • will avoid downloading too large files if the remote sends a Content-Length header
    • will by default not download files whose size can’t be queried ahead of time (see cheatsheet if you want to re-enable this)
    • now uses pack.json format (see upgrade notes)
    • as a result now works with all image types out of the box
    • only steals shortcodes we actually recognise as valid ourselves (this mostly means alphanumeric, the same restrictions you’re used to when uploading emoji)
  • the Dedupe upload filter is now always enabled
  • AnonymizeFilenames is again opt-in

Updating

First follow the usual steps; if upgrading from 3.11.0 there are no DB migrations or frontend changes.

https://docs.akkoma.dev/stable/administration/updating/

But before you startup the instance some manual changes may be needed depending on your (past) config.

Explicit upload and media proxy domains

If you haven’t explicitly configured the base_urls already and are using the features, you’ll need to do so now. E.g. if your main instance is on example.com the config needs to contain something like this:

Whilst it is still possible to keep media proxy/uploads on the same domain, we massively encourage you to change that ASAP!!!

config :pleroma, Pleroma.Web.Endpoint,
  url: [host: "example.com", scheme: "https", port: 443],
  ...

# TODO: migrate to a subdomain!
config :pleroma, Pleroma.Upload,
  base_url: "http://example.com/media/", #the /media/ here is important!
  ...

# TODO: migrate to a subdomain!
config :pleroma, :media_proxy,
  base_url: "https://example.com",
  ...

Converting stolen emoji to pack.json format

If you’re using or used StealEmoji in the past, some manual intervention is needed to carry existing emoji over to the new pack.json format. Create a new script file in e.g. the repo root and insert the following:

#!/bin/sh
# SPDX-License-Identifier: CC0-1.0 or 0BSD
set -eu

DIR="${DIR:-$(dirname "$0")/instance/static/emoji/stolen/}"
ALLOWED_EXT="${ALLOWED_EXT:-.png .gif}"

echo "Converting '$DIR' to pack.json..." >&2
cd "$DIR"

exec > pack.json

printf '{
  "pack": {
    "can-download": false,
    "share-files": false,
    "description": "Collection of emoji auto-stolen from other instances"
  },
  "files": {
'

count=0
for ext in $ALLOWED_EXT ; do
  for f in *"$ext" ; do
    if [ "${f%.*}" = "*" ] ; then
          continue
    fi
    f="$(echo "$f" | sed -e 's/"/\\"/g')"
    [ "$count" -gt 0 ] && printf ',\n'
    printf '    "%s": "%s"' "${f%.*}" "$f"
    count="$((count + 1))"
  done
done

printf '
  },
  "files_count": %d
}
' "$count"

echo "Done." >&2

Now check the setting of :pleroma, :emoji, :pack_extensions and where stolen emoji are stored. E.g. if :pack_extensions is set to ["*.png", "*.gif", "*.jpg"] and your stolen emoji are at /var/lib/akkoma/instance/emoji/stolen run the script as the akkoma user like:

ALLOWED_EXT=".png .gif .jpg" DIR="/var/lib/akkoma/instance/emoji/stolen" sh ./convert_pack.sh

If you didn’t set :emoji, :pack_extensions yourself, just drop the ALLOWED_EXT="..." prefix.

Starting up

You’re now ready to start up the instance again.

Post-Upgrade Steps

checking for impersonation (attempts)

Two new CLI tasks got added to check if your instance was ever used for impersonation or fell for any counterfeit messages. You should probably run those and let others know if any real abuse was found. Detection (esp of inserted remote counterfeits) is not perfect, read the notes in the task output and reexamine possible matches!

See their docs for how to run them.

media subdomain

Going forward same-domain setups are still supported but strongly discouraged. For now continuing to use the same domain is ok since all known exploits are fixed, but you probably should migrate to a subdomain when you can.

The old guide from Mitigating the recent Pleroma issues mostly still applies, but:

  • don’t redirect /proxy/ from the main domain, return 404 instead
  • there’s no need to manually set response headers

You can also take a look at the newly updated example nginx config for reference and if you use Caddy see also this thread. We might write a self-contained migration guide in the future.

Was my instance vulnerable?

Almost certainly, yes you were.

Who could place payloads for impersonating users from your instance depends on config:

  • always: authors of emoji packs added to your instance (if you didn’t check pack contents ahead of time)
    Note: the payloads need special adjustment for each instance though
  • using local uploader?: any other local user if uploads are on the same domain or used to be on the same domain and a redirect was left in place
  • using StealEmoji?: anyone with emoji management rights on an explicitly allowed host
  • using media proxy?: anyone who can send a post to your instance and then retrieve the proxied link from your instance (e.g. via scrolling through the global timeline)

Once a payload was placed, counterfeit insertion into any *oma instance (or others with lax id checks) can be triggered by either anyone with access to search for remote content or anyone allowed to send AP objects to your instance (which can reference the local payload e.g. in inReplyTo).

There is no workaround for this and the only mitigation is updating asap.

One massive thanky

All of my concievable thanks, from the percievable to the theoretical, go to Oneric on akko.wtf

This was discovered by them and mitigated by them, and they even handled synchronisation with some other projects. I cannot emphasize enough how helpful they have been. Thank you.

7 Likes

i’m running the mix pleroma.security spoof-inserted and get DBconnection error. mix pleroma.security spoof-uploaded did not give any problems. As i understand the issue, I should be safe, though. It’s asingle user instance and already serving media from a subdomain.

/opt/akkoma # rc-service akkoma stop
 * Stopping akkoma ...                                                                                                                                           [ ok ]
/opt/akkoma # su -s "$SHELL" akkoma
~ $ export MIX_ENV=prod
~ $ mix pleroma.security spoof-inserted
+----------------------+
|  SPOOF SEARCH NOTES  |
+----------------------+
Starting fuzzy search for counterfeit activities.
NOTE this can not guarantee detecting all counterfeits
     and may yield a small percentage of false positives.

Searching for local posts whose Create activity has no ActivityPub id...
  This is a pretty good indicator, but only for spoofs of local actors
  and only if the spoofing happened after around late 2021.

** (DBConnection.ConnectionError) tcp recv: closed (the connection was closed by the pool, possibly due to a timeout or because the pool has been terminated)
    (ecto_sql 3.10.2) lib/ecto/adapters/sql.ex:1047: Ecto.Adapters.SQL.raise_sql_call_error/1
    (ecto_sql 3.10.2) lib/ecto/adapters/sql.ex:945: Ecto.Adapters.SQL.execute/6
    (ecto 3.10.3) lib/ecto/repo/queryable.ex:229: Ecto.Repo.Queryable.execute/4
    (ecto 3.10.3) lib/ecto/repo/queryable.ex:19: Ecto.Repo.Queryable.all/3
    (pleroma 3.12.0) lib/mix/tasks/pleroma/security.ex:194: Mix.Tasks.Pleroma.Security.do_spoof_inserted/0
    (mix 1.14.4) lib/mix/task.ex:421: anonymous fn/3 in Mix.Task.run_task/4
    (mix 1.14.4) lib/mix/cli.ex:84: Mix.CLI.run_task/2
~ $ exit

ah i bet you this needs an :infinity timeout in there

but yes you’re probably ok

i’ll fix this

1 Like

^ fixed, OTPs building with that fix if you do run into it

no code changes beyond that

thk you. It first didn’t work with the “check out the latest tag” line, but switching to stable worked perfectly! <3

Also thank you Oneric for al the work you’ve been doing lately <3

I see output[1], but it looks like a false positive (the output also says it’s possible it will find false positives). Looking at the code, it’s matching on /emoji/[2], but in this case it’s an actual user, so OK.

[1]

Maybe Spoofed Posts
===================
  {AYhBOIt669RmafIg9g, https://don.nzws.me/users/emoji/statuses/107909887298227557}
  {AYhBORQ0NgWX32l4Yi, https://don.nzws.me/users/emoji/statuses/110032897910487696}
  {AYhBP01lhmvuO0gi8m, https://don.nzws.me/users/emoji/statuses/109788765398757903}
  {AYhBPB78UAlKlqHeXA, https://don.nzws.me/users/emoji/statuses/109788768833932593}
  {AYhBPBpno6UX0NBK9g, https://don.nzws.me/users/emoji/statuses/108639694072858294}
  {AYhBPI3qenIAJfncsi, https://don.nzws.me/users/emoji/statuses/110026372636600880}
  {AYhBPMDDDv9vBsDA24, https://don.nzws.me/users/emoji/statuses/109866428267159848}
  {AYhBPbixYASzIgBE6C, https://don.nzws.me/users/emoji/statuses/110053430674904036}

[2]

# https://akkoma.dev/AkkomaGang/akkoma/src/branch/stable/lib/mix/tasks/pleroma/security.ex#L262
        fragment("?->>'id' LIKE ?", o.data, "%/emoji/%") or

oh yeah I didn’t tag the .1 because I’m stupid

edit: now i did

1 Like

Hello, thank you for update. I get this error when running mix compile. It looks scary.

ERROR! the application :mime has a different value set for key :extensions during runtime compared to compile time. Since this application environment entry was marked as compile time, this difference can lead to different behaviour than expected:

  * Compile time value was set to: %{"activity+json" => "application/activity+json"}
  * Runtime value was set to: %{"activity+json" => "text/plain", "jrd+json" => "text/plain", "xrd+xml" => "text/plain"}

To fix this error, you might:

  * Make the runtime value match the compile time one

  * Recompile your project. If the misconfigured application is a dependency, you may need to run "mix deps.compile mime --force"

  * Alternatively, you can disable this check. If you are using releases, you can set :validate_compile_env to false in your release configuration. If you are using Mix to start your system, you can pass the --no-validate-compile-env flag

FWIW I ran this update on an OTP install with no issues encountered so far.

1 Like

if this actually causes any issues, just run a mix clean to get a fresh build going

1 Like

Upgrade worked fine there, also no security issue found with the two commands. Thanks for the fix!

At the risk of showing my complete ignorance, I’d like to ask a few questions regarding the settings changes.
First of all, my instances are small, 1 and 2 users only.

  • I have my base urls set, but that’s it.
  • I don’t have the upload setting in my config/prod.secrets.exs,
  • My media_proxy section is disabled:
config :pleroma, :media_proxy,
  enabled: false,
  redirect_on_failure: true
  #base_url: "https://cache.pleroma.social"

If I understand your advice correctly I now have to add the Upload section and need to enable the media_proxy?

Your Upload example shows the “/media” subdomain added to the url, so this already shows the desired state, correct?

Your media_proxy example has not got a subdomain, this probably means I need to come up with one? Is any subdomain ok?
Considering the description for the media proxy The base URL to access a user-uploaded file. Useful when you want to proxy the media files via another host/CDN fronts.I don’t have another host or CDN to handle such requests, do I need to setup a new host, and if so, what needs to run on that host?
Will I also need a new DNS entry for the example.com/media subdomain or will Akkoma handle the redirection?

Is there a way to test that things are working as desired?

released v3.12.2 which will blow up the process in the case that a user has not set their base_url in the uploader because we keep getting messaged about it

1 Like

you are confusing two different systems

the uploader is configured by Pleroma.Upload - this is when users upload media to your instance. Your base URL here should be a subdomain, and probably suffixed by /media/ for routing purposes.

media proxy is for external media, things other people post. If you do not use media proxy, you do not need to configure it.

I realized those two were two different systems, but what confused me was that the media_proxy shown in your example has got the same TODO label. So I thought I need to touch both. I’ll keep the proxy disabled for my small user count use case.

Means all I need to change is Pleroma.Upload section, which in my case means create a new Nginx entry which points https://social.myDomain.com/media to a different folder?

@FloatingGhost I migrated from pleroma a few weeks ago and ran into a similar problem when trying to update:

root@debian:/opt/pleroma# MIX_ENV=prod mix clean
root@debian:/opt/pleroma# MIX_ENV=prod mix compile
ERROR! the application :mime has a different value set for key :extensions during runtime compared to compile time. Since this application environment entry was marked as compile time, this difference can lead to different behaviour than expected:

  * Compile time value was set to: %{"activity+json" => "application/activity+json"}
  * Runtime value was set to: %{"activity+json" => "text/plain", "jrd+json" => "text/plain", "xrd+xml" => "text/plain"}

To fix this error, you might:

  * Make the runtime value match the compile time one

  * Recompile your project. If the misconfigured application is a dependency, you may need to run "mix deps.compile mime --force"

  * Alternatively, you can disable this check. If you are using releases, you can set :validate_compile_env to false in your release configuration. If you are using Mix to start your system, you can pass the --no-validate-compile-env flag



14:01:35.189 [error] Task #PID<0.583.0> started from #PID<0.96.0> terminating
** (stop) "aborting boot"
    (elixir 1.14.0) Config.Provider.boot/2
Function: &:erlang.apply/2
    Args: [#Function<1.127481437/1 in Mix.Tasks.Compile.All.load_apps/3>, [mime: "/opt/pleroma/_build/prod/lib"]]
** (EXIT from #PID<0.96.0>) an exception was raised:
    ** (ErlangError) Erlang error: "aborting boot"
        (elixir 1.14.0) Config.Provider.boot/2

mix clean did not help in my case. Any idea what to do about it?

I did the second suggestion and yolo’d mix deps.compile mime --force and it went through. But don’t take my word for it I don’t know how anything works.

1 Like

thank you. unfortunately it does not work for me. everything seems fine, but afterwards the server is down with 502.

502 means it isn’t running

please do check your logs

after running

mix deps.compile mime --force
systemctl stop pleroma
mix ecto.migrate
systemctl start pleroma

and looking for the logs

Apr 09 11:34:34 debian-4gb-fsn1-2 mix[19479]: Crash dump is being written to: erl_crash.dump...done
Apr 09 11:34:34 debian-4gb-fsn1-2 systemd[1]: pleroma.service: Main process exited, code=exited, status=1/FAILURE
Apr 09 11:34:34 debian-4gb-fsn1-2 systemd[1]: pleroma.service: Failed with result 'exit-code'.
Apr 09 11:34:34 debian-4gb-fsn1-2 systemd[1]: pleroma.service: Consumed 3.864s CPU time.
Apr 09 11:34:34 debian-4gb-fsn1-2 systemd[1]: pleroma.service: Scheduled restart job, restart counter is at 49.
Apr 09 11:34:34 debian-4gb-fsn1-2 systemd[1]: Stopped pleroma.service - Pleroma social network.
Apr 09 11:34:34 debian-4gb-fsn1-2 systemd[1]: pleroma.service: Consumed 3.864s CPU time.
Apr 09 11:34:34 debian-4gb-fsn1-2 systemd[1]: Started pleroma.service - Pleroma social network.
Apr 09 11:34:36 debian-4gb-fsn1-2 mix[19547]: 11:34:36.708 [notice]     :alarm_handler: {:set, {:system_memory_high_watermark, []}}
Apr 09 11:34:37 debian-4gb-fsn1-2 mix[19547]: 11:34:37.323 [error] !!!WARNING!!!
Apr 09 11:34:37 debian-4gb-fsn1-2 mix[19547]: Your config does not specify a base_url for uploads!
Apr 09 11:34:37 debian-4gb-fsn1-2 mix[19547]: Please make the following change:
Apr 09 11:34:37 debian-4gb-fsn1-2 mix[19547]: * `config :pleroma, Pleroma.Upload, base_url: "https://example.com/media/`
Apr 09 11:34:37 debian-4gb-fsn1-2 mix[19547]: Please note that it is HEAVILY recommended to use a subdomain to host user-uploaded med>
Apr 09 11:34:37 debian-4gb-fsn1-2 mix[19547]: 11:34:37.328 [notice] Application pleroma exited: exited in: Pleroma.Application.start(>
Apr 09 11:34:37 debian-4gb-fsn1-2 mix[19547]:     ** (EXIT) an exception was raised:
Apr 09 11:34:37 debian-4gb-fsn1-2 mix[19547]:         ** (ArgumentError) No base_url set for uploads - please set one in your config!
Apr 09 11:34:37 debian-4gb-fsn1-2 mix[19547]:             (pleroma 3.12.2) lib/pleroma/config/deprecation_warnings.ex:360: Pleroma.Co>
Apr 09 11:34:37 debian-4gb-fsn1-2 mix[19547]:             (pleroma 3.12.2) lib/pleroma/config/deprecation_warnings.ex:186: Pleroma.Co>
Apr 09 11:34:37 debian-4gb-fsn1-2 mix[19547]:             (pleroma 3.12.2) lib/pleroma/application.ex:53: Pleroma.Application.start/2
Apr 09 11:34:37 debian-4gb-fsn1-2 mix[19547]:             (kernel 8.5.3) application_master.erl:293: :application_master.start_it_old>
Apr 09 11:34:37 debian-4gb-fsn1-2 mix[19612]: [os_mon] memory supervisor port (memsup): Erlang has closed
Apr 09 11:34:37 debian-4gb-fsn1-2 mix[19547]: 11:34:37.361 [notice]     :alarm_handler: {:clear, :system_memory_high_watermark}
Apr 09 11:34:37 debian-4gb-fsn1-2 mix[19613]: [os_mon] cpu supervisor port (cpu_sup): Erlang has closed
Apr 09 11:34:38 debian-4gb-fsn1-2 mix[19547]: {"Kernel pid terminated",application_controller,"{application_start_failure,pleroma,{ba>
Apr 09 11:34:38 debian-4gb-fsn1-2 mix[19547]: Kernel pid terminated (application_controller) ({application_start_failure,pleroma,{bad>
Apr 09 11:34:38 debian-4gb-fsn1-2 mix[19547]: 
Apr 09 11:34:39 debian-4gb-fsn1-2 mix[19547]: Crash dump is being written to: erl_crash.dump...done
Apr 09 11:34:39 debian-4gb-fsn1-2 systemd[1]: pleroma.service: Main process exited, code=exited, status=1/FAILURE
Apr 09 11:34:39 debian-4gb-fsn1-2 systemd[1]: pleroma.service: Failed with result 'exit-code'.
Apr 09 11:34:39 debian-4gb-fsn1-2 systemd[1]: pleroma.service: Consumed 3.863s CPU time.
Apr 09 11:34:39 debian-4gb-fsn1-2 systemd[1]: pleroma.service: Scheduled restart job, restart counter is at 50.
Apr 09 11:34:39 debian-4gb-fsn1-2 systemd[1]: Stopped pleroma.service - Pleroma social network.
Apr 09 11:34:39 debian-4gb-fsn1-2 systemd[1]: pleroma.service: Consumed 3.863s CPU time.
Apr 09 11:34:39 debian-4gb-fsn1-2 systemd[1]: Started pleroma.service - Pleroma social network.
Apr 09 11:34:41 debian-4gb-fsn1-2 mix[19619]: 11:34:41.248 [notice]     :alarm_handler: {:set, {:system_memory_high_watermark, []}}

I can see the following entries

the log tells you exactly how to fix this
you need to set the media base URL