Meilisearch indexing can't keep up

ninja · 26 January 2023 23:20

Have been running Meilisearch for a while. I noticed the search results are no longer fresh. E.g. when I search for some common terms, the top results are >5 days old.

Also, I noticed the Meilisearch daemon is constantly reading from disk at full speed.

My gut feeling is there’re too many incoming federated posts to index, and the indexer has gradually fallen behind.

Does anyone else have this issue?

chainofflowers · 11 March 2023 15:38

I think I am experiencing the same. I can see in the journal logs that new messages are indeed sent to meilisearch, but whenever I search for anything I only get results for 1 month ago.

I am not sure what’s going on there… and if something broke, I can’t say since when it’s like that…

chainofflowers · 11 March 2023 17:25

So, I have scratched the meilisearch index storage and recreated everything following the guide.

When I try to start the index I get this:

# sudo -Hu akkoma MIX_ENV=prod mix pleroma.search.meilisearch index
** (FunctionClauseError) no function clause matching in Version.Parser.parse_version/2    
    
    The following arguments were given to Version.Parser.parse_version/2:
    
        # 1
        nil
    
        # 2
        false
    
    Attempted function clauses (showing 1 out of 1):
    
        def parse_version(string, approximate?) when is_binary(string)
    
    (elixir 1.14.0) lib/version.ex:580: Version.Parser.parse_version/2
    (elixir 1.14.0) lib/version.ex:458: Version.to_matchable/2
    (elixir 1.14.0) lib/version.ex:292: Version.match?/3
    (pleroma 3.6.0-9-g36cb19db) lib/mix/tasks/pleroma/search/meilisearch.ex:25: Mix.Tasks.Pleroma.Search.Meilisearch.run/1
    (mix 1.14.0) lib/mix/task.ex:421: anonymous fn/3 in Mix.Task.run_task/4
    (mix 1.14.0) lib/mix/cli.ex:84: Mix.CLI.run_task/2

In the system logs I can see:

[...]  mix[9498]: 18:47:30.660 [error] Failed to add activity ATVt88hwypSZuR1Axc to index: {:ok, %{"code" => "invalid_api_key", "link" => "https://docs.meilisearch.com/errors#invalid_api_key", "message" => "The provided API key is invalid.", "type" => "auth"}}

(Of course I checked 100 times that the private_key I put into config/prod.secrets.exs is the one returned by pleroma.search.meilisearch show-keys <your master key here>

Any help?

chainofflowers · 12 March 2023 11:05

I verified manually that the key I put in the akkoma’s config is correct. I can use it from curl and it works.
When akkoma talks to meilisearch, instead, it seems to pass an invalid key. This is the TRACE-level log of meilisearch in response to akkoma’s index endpoint:

[2023-03-12T11:04:22Z TRACE mio::poll] registering event source with poller: token=Token(1), interests=READABLE | WRITABLE
[2023-03-12T11:04:22Z TRACE actix_http::h1::dispatcher] start flags: (empty)
[2023-03-12T11:04:22Z TRACE actix_http::h1::dispatcher] start timers:
[2023-03-12T11:04:22Z TRACE actix_http::h1::dispatcher]   head timer is inactive
[2023-03-12T11:04:22Z TRACE actix_http::h1::dispatcher]   keep-alive timer is inactive
[2023-03-12T11:04:22Z TRACE actix_http::h1::dispatcher]   shutdown timer is inactive
[2023-03-12T11:04:22Z TRACE actix_http::h1::dispatcher] end timers:
[2023-03-12T11:04:22Z TRACE actix_http::h1::dispatcher]   head timer is active and due to expire in 4587.6147 milliseconds
[2023-03-12T11:04:22Z TRACE actix_http::h1::dispatcher]   keep-alive timer is inactive
[2023-03-12T11:04:22Z TRACE actix_http::h1::dispatcher]   shutdown timer is inactive
[2023-03-12T11:04:22Z TRACE actix_http::h1::dispatcher] end flags: STARTED
[2023-03-12T11:04:22Z TRACE actix_http::h1::dispatcher] start flags: STARTED
[2023-03-12T11:04:22Z TRACE actix_http::h1::dispatcher] start timers:
[2023-03-12T11:04:22Z TRACE actix_http::h1::dispatcher]   head timer is active and due to expire in 4572.349 milliseconds
[2023-03-12T11:04:22Z TRACE actix_http::h1::dispatcher]   keep-alive timer is inactive
[2023-03-12T11:04:22Z TRACE actix_http::h1::dispatcher]   shutdown timer is inactive
[2023-03-12T11:04:22Z TRACE actix_http::h1::dispatcher] end timers:
[2023-03-12T11:04:22Z TRACE actix_http::h1::dispatcher]   head timer is inactive
[2023-03-12T11:04:22Z TRACE actix_http::h1::dispatcher]   keep-alive timer is inactive
[2023-03-12T11:04:22Z TRACE actix_http::h1::dispatcher]   shutdown timer is inactive
[2023-03-12T11:04:22Z TRACE actix_http::h1::dispatcher] end flags: STARTED
[2023-03-12T11:04:22Z TRACE actix_http::h1::dispatcher] start flags: STARTED
[2023-03-12T11:04:22Z TRACE actix_http::h1::dispatcher] start timers:
[2023-03-12T11:04:22Z TRACE actix_http::h1::dispatcher]   head timer is inactive
[2023-03-12T11:04:22Z TRACE actix_http::h1::dispatcher]   keep-alive timer is inactive
[2023-03-12T11:04:22Z TRACE actix_http::h1::dispatcher]   shutdown timer is inactive
[2023-03-12T11:04:22Z DEBUG actix_web::middleware::logger] Error in response: ResponseError { code: 403, message: "The provided API key is invalid.", error_code: "invalid_api_key", error_type: "auth", error_link: "https://docs.meilisearch.com/errors#invalid_api_key" }
[2023-03-12T11:04:22Z INFO  actix_web::middleware::logger] 127.0.0.1 "GET /version HTTP/1.1" 403 146 "-" "Akkoma 3.6.0-9-g36cb19db; https://pleroma.neuromante.net <chainofflowers@neuromante.net>" 0.000939
[2023-03-12T11:04:22Z TRACE actix_http::h1::dispatcher] end timers:
[2023-03-12T11:04:22Z TRACE actix_http::h1::dispatcher]   head timer is inactive
[2023-03-12T11:04:22Z TRACE actix_http::h1::dispatcher]   keep-alive timer is inactive
[2023-03-12T11:04:22Z TRACE actix_http::h1::dispatcher]   shutdown timer is inactive
[2023-03-12T11:04:22Z TRACE actix_http::h1::dispatcher] end flags: STARTED | FINISHED | KEEP_ALIVE
[2023-03-12T11:04:22Z TRACE actix_http::h1::dispatcher] start flags: STARTED | FINISHED | KEEP_ALIVE
[2023-03-12T11:04:22Z TRACE actix_http::h1::dispatcher] start timers:
[2023-03-12T11:04:22Z TRACE actix_http::h1::dispatcher]   head timer is inactive
[2023-03-12T11:04:22Z TRACE actix_http::h1::dispatcher]   keep-alive timer is inactive
[2023-03-12T11:04:22Z TRACE actix_http::h1::dispatcher]   shutdown timer is inactive
[2023-03-12T11:04:22Z TRACE actix_http::h1::dispatcher] read half closed; start shutdown
[2023-03-12T11:04:22Z TRACE actix_http::h1::dispatcher] start flags: STARTED | KEEP_ALIVE | SHUTDOWN | READ_DISCONNECT
[2023-03-12T11:04:22Z TRACE actix_http::h1::dispatcher] start timers:
[2023-03-12T11:04:22Z TRACE actix_http::h1::dispatcher]   head timer is inactive
[2023-03-12T11:04:22Z TRACE actix_http::h1::dispatcher]   keep-alive timer is inactive
[2023-03-12T11:04:22Z TRACE actix_http::h1::dispatcher]   shutdown timer is inactive
[2023-03-12T11:04:22Z TRACE actix_http::h1::dispatcher] end flags: STARTED | KEEP_ALIVE | SHUTDOWN | READ_DISCONNECT
[2023-03-12T11:04:22Z TRACE mio::poll] deregistering event source from poller

it is not even able to get the version number… (which is 0.29.1)

ninja · 13 March 2023 06:29

Suggest to try 2 things:

Run meilisearch in dev mode, which doesn’t require API key
Upgrade meilisearch to latest. I’ve upgraded to 1.0.2 since the original post - Built from src

Note that meilisearch index is version-dependent, so re-index is required for #2. The official doc has a method of export/import the index but seems very cumbersome to run. Fortunately re-indexing is very fast.