Ch.Connection (#PID<0.3395.0>) failed to connect, connection refused crash loop

[Nikola-Milovic]

Hello everyone, I am trying to host on AWS ECS, and I am encountering a crash loop, but I have no idea what exactly is causing it. What does it fail to connect to? Is it the Postgres instance? Or the clickhouse? Or something else?

November 03, 2023 at 20:12 (UTC+1:00) | Crash dump is being written to: erl_crash.dump... | Plausible
-- | -- | --
November 03, 2023 at 20:12 (UTC+1:00) | Kernel pid terminated (application_controller) ({application_start_failure,plausible,{{shutdown,{failed_to_start_child,'Elixir.Plausible.Session.Salts',{#{'__exception__' => true,'__struct__' => 'Elixir.RuntimeError',message => <<"could not lookup Ecto repo Plausible.Repo because it was not started or it does not exist">>},[{'Elixir.Ecto.Repo.Registry',lookup,1,[{file,"lib/ecto/repo/registry.ex"},{line,22},{error_info,#{module => 'Elixir.Exception'}}]},{'Elixir.Ecto.Repo.Supervisor',tuplet,2,[{file,"lib/ecto/repo/supervisor.ex"},{line,160}]},{'Elixir.Plausible.Repo',delete_all,2,[{file,"lib/plausible/repo.ex"},{line,2}]},{'Elixir.Plausible.Session.Salts','-start_link/1-fun-0-',0,[{file,"lib/plausible/session/salts.ex"},{line,8}]},{'Elixir.Agent.Server',init,1,[{file,"lib/agent/server.ex"},{line,8}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,851}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,814}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]}}},{'Elixir.Plausible.Appli | Plausible
November 03, 2023 at 20:12 (UTC+1:00) |   | Plausible
November 03, 2023 at 20:12 (UTC+1:00) | {"Kernel pid terminated",application_controller,"{application_start_failure,plausible,{{shutdown,{failed_to_start_child,'Elixir.Plausible.Session.Salts',{#{'__exception__' => true,'__struct__' => 'Elixir.RuntimeError',message => <<\"could not lookup Ecto repo Plausible.Repo because it was not started or it does not exist\">>},[{'Elixir.Ecto.Repo.Registry',lookup,1,[{file,\"lib/ecto/repo/registry.ex\"},{line,22},{error_info,#{module => 'Elixir.Exception'}}]},{'Elixir.Ecto.Repo.Supervisor',tuplet,2,[{file,\"lib/ecto/repo/supervisor.ex\"},{line,160}]},{'Elixir.Plausible.Repo',delete_all,2,[{file,\"lib/plausible/repo.ex\"},{line,2}]},{'Elixir.Plausible.Session.Salts','-start_link/1-fun-0-',0,[{file,\"lib/plausible/session/salts.ex\"},{line,8}]},{'Elixir.Agent.Server',init,1,[{file,\"lib/agent/server.ex\"},{line,8}]},{gen_server,init_it,2,[{file,\"gen_server.erl\"},{line,851}]},{gen_server,init_it,6,[{file,\"gen_server.erl\"},{line,814}]},{proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,240}]}]}}},{'Elixir.Plausible.Application',start,[normal,[]]}}}"} | Plausible
November 03, 2023 at 20:12 (UTC+1:00) | 19:12:56.209 [error] Ch.Connection (#PID<0.3394.0>) failed to connect: ** (Mint.TransportError) connection refused | Plausible
November 03, 2023 at 20:12 (UTC+1:00) | 19:12:56.210 [error] Ch.Connection (#PID<0.3393.0>) failed to connect: ** (Mint.TransportError) connection refused | Plausible
November 03, 2023 at 20:12 (UTC+1:00) | 19:12:56.210 [error] Ch.Connection (#PID<0.3395.0>) failed to connect: ** (Mint.TransportError) connection refused | Plausible
November 03, 2023 at 20:12 (UTC+1:00) | 19:12:56.210 [error] Ch.Connection (#PID<0.3405.0>) failed to connect: ** (Mint.TransportError) connection refused | Plausible
November 03, 2023 at 20:12 (UTC+1:00) | 19:12:56.210 [error] Ch.Connection (#PID<0.3401.0>) failed to connect: ** (Mint.TransportError) connection refused/

Any advice on what direction to look into is greatly appreciated!

[ruslandoga]

👋 @Nikola-Milovic

Is it the Postgres instance? Or the clickhouse?

According to these errors, it's both :) Please check your DATABASE_URL and CLICKHOUSE_DATABASE_URL env vars + whether the ECS instance is in the same VPC as the databases, and if the security groups are setup correctly.

[ruslandoga]

Might be just blips in the network. It'd be interesting to see the logs though.

[CodeWithKyrian]

I get this same error randomly and the logs are similar to this

[ruslandoga]

👋 @CodeWithKyrian

Do you co-locate the databases and Plausible on the same VM or use external services for the databases?

[CodeWithKyrian]

They're in the same VM

[ruslandoga]

Would you be able to share the logs and your configuration files? Also docker compose ps and docker stats --no-stream.