Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Crash trying to shut down the ProcessSupervisor #154

Open
mgwidmann opened this issue Aug 14, 2019 · 2 comments
Open

Crash trying to shut down the ProcessSupervisor #154

mgwidmann opened this issue Aug 14, 2019 · 2 comments
Labels
bug Something isn't working

Comments

@mgwidmann
Copy link

mgwidmann commented Aug 14, 2019 •
edited
Loading

Perhaps I'm doing something wrong so please correct me if I have.

I have 5 different registries/supervisors to split the load of my application. One for "everything else" and 4 for one particular entity which I've split up by hashing the name of the worker and starting the workers under that supervisor (and registering the name in that registry).

I'm using local-cluster with dynamic supervision (and the example nodewatching genserver) for testing where I see this the most, but it also happens when two machines connect together.

It doesn't say why the ProcessSupervisor shut down before this error occurred, so I assume it was a normal shutdown. Any ideas?

2019-08-14T05:04:31.573000 [info] [libcluster:gossip] connected to :"[email protected]"

2019-08-14T04:51:03.580000 [error] GenServer MyApplication.DistributedSupervisor terminating
** (stop) exited in: GenServer.stop(MyApplication.DistributedSupervisor.ProcessesSupervisor, :normal, :infinity)
    ** (EXIT) no process: the process is not alive or there's no process currently associated with the given name, possibly because its application isn't started
    (elixir) lib/gen_server.ex:938: GenServer.stop/3
    (horde) lib/horde/supervisor_impl.ex:570: Horde.SupervisorImpl.shut_down_all_processes/1
    (horde) lib/horde/supervisor_impl.ex:347: Horde.SupervisorImpl.handle_info/2
    (stdlib) gen_server.erl:637: :gen_server.try_dispatch/4
    (stdlib) gen_server.erl:711: :gen_server.handle_msg/6
    (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
Last message: {:crdt_update, [{:add, {:member_node_info, {MyApplication.DistributedSupervisor, :"[email protected]"}}, %Horde.Supervisor.Member{name: {MyApplication.DistributedSupervisor, :"[email protected]"}, status: :shutting_down}}]}
State: %Horde.SupervisorImpl{distribution_strategy: Horde.UniformQuorumDistribution, members: %{{MyApplication.DistributedSupervisor, :"[email protected]"} => 1}, members_info: %{{MyApplication.DistributedSupervisor, :"[email protected]"} => %Horde.Supervisor.Member{name: {MyApplication.DistributedSupervisor, :"[email protected]"}, status: :alive}}, name: MyApplication.DistributedSupervisor, name_to_supervisor_ref: %{{MyApplication.DistributedSupervisor, :"[email protected]"} => #Reference<0.3617219455.1984430081.38639>}, process_pid_to_id: %{#PID<20100.778.0> => 126195642247525413471363530776329509736, #PID<0.955.0> => 126195642247525413471363530776329509736}, processes_by_id: %{126195642247525413471363530776329509736 => {{MyApplication.DistributedSupervisor, :"[email protected]"}, %{id: 126195642247525413471363530776329509736, restart: :permanent, shutdown: 10000, start: {MyApplication.ResultWriter, :start_link, []}}, #PID<0.955.0>}}, processes_updated_at: 0, processes_updated_counter: 0, shutting_down: true, supervisor_options: [members: [{MyApplication.DistributedSupervisor, :"[email protected]"}, {MyApplication.DistributedSupervisor, :"[email protected]"}], name: MyApplication.DistributedSupervisor, root_name: MyApplication.DistributedSupervisor, init_module: MyApplication.DistributedSupervisor, id: MyApplication.DistributedSupervisor, strategy: :one_for_one, distribution_strategy: Horde.UniformQuorumDistribution, max_restarts: 1000, max_seconds: 1], supervisor_ref_to_name: %{#Reference<0.3617219455.1984430081.38639> => {MyApplication.DistributedSupervisor, :"[email protected]"}}, waiting_for_quorum: []}
@derekkraan
Copy link
Owner

Hi Matt,

Thank for the bug report. There is something about the shutdown process that doesn't make 100% sense. I believe this is during a regular shutdown.

It would be very helpful if you could reproduce this in a test in Horde. Can I ask you to make a PR with that? Eventually I want to have some good tests using local_cluster (and later schism) to test these scenarios.

@mgwidmann
Copy link
Author

So local_cluster starts new nodes for the test and then shuts them down which would make sense if this occurred during the shutdown phase. After the process crashes it is restarted and everything is better.

I can try to reproduce the issue in the example application though I'm not sure what I'm doing differently to make this happen. Running mix test is the most reliable way to reproduce this error for me.

Here is the entire output from mix test. As you can see the crash occurs several times, when the machines are connecting and after connection. The sequence events as I follow it go like this:

  • Main instance boots up supervision tree
  • Test setup beings
  • Local cluster boots up 2 additional nodes (crashes occur, maybe on main instance?)
  • Additional nodes boot up supervision trees (more crashes)
  • Local cluster call returns (marked by "Test nodes that are now online:" line) for first test
  • Test passes
  • Repeat above for second test
✗ mix test

2019-08-14T13:52:27.834000 [info] Starting Horde.RegistryImpl with name MyApplication.Registry

2019-08-14T13:52:27.898000 [info] Starting Horde.RegistryImpl with name MyApplication.EntityRegistry1

2019-08-14T13:52:27.898000 [info] Starting Horde.RegistryImpl with name MyApplication.EntityRegistry2

2019-08-14T13:52:27.899000 [info] Starting Horde.RegistryImpl with name MyApplication.EntityRegistry3

2019-08-14T13:52:27.899000 [info] Starting Horde.RegistryImpl with name MyApplication.EntityRegistry4

2019-08-14T13:52:27.910000 [info] Starting Horde.SupervisorImpl with name MyApplication.DistributedSupervisor

2019-08-14T13:52:27.910000 [info] Starting Horde.SupervisorImpl with name MyApplication.EntitySupervisor1

2019-08-14T13:52:27.911000 [info] Starting Horde.SupervisorImpl with name MyApplication.EntitySupervisor2

2019-08-14T13:52:27.911000 [info] Starting Horde.SupervisorImpl with name MyApplication.EntitySupervisor3

2019-08-14T13:52:27.911000 [info] Starting Horde.SupervisorImpl with name MyApplication.EntitySupervisor4

2019-08-14T13:52:27.915000 [info] Starting up Elixir.MyApplication.Worker with 4 schedulers online

2019-08-14T13:52:28.790000 [error] GenServer MyApplication.DistributedSupervisor terminating
** (stop) exited in: GenServer.stop(MyApplication.DistributedSupervisor.ProcessesSupervisor, :normal, :infinity)
    ** (EXIT) no process: the process is not alive or there's no process currently associated with the given name, possibly because its application isn't started
    (elixir) lib/gen_server.ex:938: GenServer.stop/3
    (horde) lib/horde/supervisor_impl.ex:571: Horde.SupervisorImpl.shut_down_all_processes/1
    (horde) lib/horde/supervisor_impl.ex:347: Horde.SupervisorImpl.handle_info/2
    (stdlib) gen_server.erl:637: :gen_server.try_dispatch/4
    (stdlib) gen_server.erl:711: :gen_server.handle_msg/6
    (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
Last message: {:crdt_update, [{:add, {:member, {MyApplication.DistributedSupervisor, :"[email protected]"}}, 1}]}
State: %Horde.SupervisorImpl{distribution_strategy: Horde.UniformQuorumDistribution, members: %{{MyApplication.DistributedSupervisor, :"[email protected]"} => 1, {MyApplication.DistributedSupervisor, :"[email protected]"} => 1}, members_info: %{{MyApplication.DistributedSupervisor, :"[email protected]"} => %Horde.Supervisor.Member{name: {MyApplication.DistributedSupervisor, :"[email protected]"}, status: :alive}, {MyApplication.DistributedSupervisor, :"[email protected]"} => %Horde.Supervisor.Member{name: {MyApplication.DistributedSupervisor, :"[email protected]"}, status: :uninitialized}}, name: MyApplication.DistributedSupervisor, name_to_supervisor_ref: %{{MyApplication.DistributedSupervisor, :"[email protected]"} => #Reference<0.3609365875.1531183106.30378>}, process_pid_to_id: %{#PID<0.370.0> => 259476575823192166570427768043200980535}, processes_by_id: %{259476575823192166570427768043200980535 => {{MyApplication.DistributedSupervisor, :"[email protected]"}, %{id: 259476575823192166570427768043200980535, restart: :permanent, shutdown: 10000, start: {MyApplication.ScannerResultWriter, :start_link, []}}, #PID<0.370.0>}}, processes_updated_at: 0, processes_updated_counter: 0, shutting_down: false, supervisor_options: [members: [{MyApplication.DistributedSupervisor, :"[email protected]"}], name: MyApplication.DistributedSupervisor, root_name: MyApplication.DistributedSupervisor, init_module: MyApplication.DistributedSupervisor, id: MyApplication.DistributedSupervisor, strategy: :one_for_one, distribution_strategy: Horde.UniformQuorumDistribution, max_restarts: 1000, max_seconds: 1], supervisor_ref_to_name: %{#Reference<0.3609365875.1531183106.30378> => {MyApplication.DistributedSupervisor, :"[email protected]"}}, waiting_for_quorum: []}

2019-08-14T13:52:28.790000 [error] GenServer #PID<0.339.0> terminating
** (stop) exited in: GenServer.call(MyApplication.DistributedSupervisor, :horde_shutting_down, 5000)
    ** (EXIT) exited in: GenServer.stop(MyApplication.DistributedSupervisor.ProcessesSupervisor, :normal, :infinity)
        ** (EXIT) no process: the process is not alive or there's no process currently associated with the given name, possibly because its application isn't started
    (elixir) lib/gen_server.ex:989: GenServer.call/3
    (horde) lib/horde/signal_shutdown.ex:21: anonymous fn/1 in Horde.SignalShutdown.terminate/2
    (elixir) lib/enum.ex:769: Enum."-each/2-lists^foreach/1-0-"/2
    (elixir) lib/enum.ex:769: Enum.each/2
    (stdlib) gen_server.erl:673: :gen_server.try_terminate/3
    (stdlib) gen_server.erl:858: :gen_server.terminate/10
    (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
Last message: {:EXIT, #PID<0.334.0>, :shutdown}
State: [MyApplication.DistributedSupervisor.GracefulShutdownManager, MyApplication.DistributedSupervisor]

2019-08-14T13:52:28.836000 [info] Starting Horde.SupervisorImpl with name MyApplication.DistributedSupervisor

09:52:31.472 [info]  Starting Horde.RegistryImpl with name MyApplication.Registry

09:52:31.472 [info]  Starting Horde.RegistryImpl with name MyApplication.Registry

09:52:31.474 [info]  Starting Horde.RegistryImpl with name MyApplication.EntityRegistry1

09:52:31.474 [info]  Starting Horde.RegistryImpl with name MyApplication.EntityRegistry1

09:52:31.474 [info]  Starting Horde.RegistryImpl with name MyApplication.EntityRegistry2

09:52:31.474 [info]  Starting Horde.RegistryImpl with name MyApplication.EntityRegistry2

09:52:31.475 [info]  Starting Horde.RegistryImpl with name MyApplication.EntityRegistry3

09:52:31.475 [info]  Starting Horde.RegistryImpl with name MyApplication.EntityRegistry3

09:52:31.475 [info]  Starting Horde.RegistryImpl with name MyApplication.EntityRegistry4

09:52:31.475 [info]  Starting Horde.RegistryImpl with name MyApplication.EntityRegistry4

09:52:31.538 [info]  Starting Horde.SupervisorImpl with name MyApplication.DistributedSupervisor

09:52:31.538 [info]  Starting Horde.SupervisorImpl with name MyApplication.EntitySupervisor1

09:52:31.537 [info]  Starting Horde.SupervisorImpl with name MyApplication.DistributedSupervisor

09:52:31.538 [info]  Starting Horde.SupervisorImpl with name MyApplication.EntitySupervisor1

09:52:31.539 [info]  Starting Horde.SupervisorImpl with name MyApplication.EntitySupervisor2

09:52:31.539 [info]  Starting Horde.SupervisorImpl with name MyApplication.EntitySupervisor2

09:52:31.539 [info]  Starting Horde.SupervisorImpl with name MyApplication.EntitySupervisor3

09:52:31.540 [info]  Starting Horde.SupervisorImpl with name MyApplication.EntitySupervisor3

09:52:31.540 [info]  Starting Horde.SupervisorImpl with name MyApplication.EntitySupervisor4

09:52:31.541 [info]  Starting Horde.SupervisorImpl with name MyApplication.EntitySupervisor4

09:52:31.591 [info]  Starting up Elixir.MyApplication.Worker with 4 schedulers online

09:52:31.591 [info]  Starting up Elixir.MyApplication.Worker with 4 schedulers online
warning: redefining module MyApplication.DistributedTest (current version defined in memory)
  test/my_application/distributed_test.exs:1

Test nodes that are now online: [:"[email protected]", :"[email protected]"]
.
2019-08-14T13:52:31.997000 [error] GenServer MyApplication.EntitySupervisor1 terminating
** (stop) exited in: GenServer.stop(MyApplication.EntitySupervisor1.ProcessesSupervisor, :normal, :infinity)
    ** (EXIT) no process: the process is not alive or there's no process currently associated with the given name, possibly because its application isn't started
    (elixir) lib/gen_server.ex:938: GenServer.stop/3
    (horde) lib/horde/supervisor_impl.ex:571: Horde.SupervisorImpl.shut_down_all_processes/1
    (horde) lib/horde/supervisor_impl.ex:347: Horde.SupervisorImpl.handle_info/2
    (stdlib) gen_server.erl:637: :gen_server.try_dispatch/4
    (stdlib) gen_server.erl:711: :gen_server.handle_msg/6
    (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
Last message: {:crdt_update, [{:add, {:member_node_info, {MyApplication.EntitySupervisor1, :"[email protected]"}}, %Horde.Supervisor.Member{name: {MyApplication.EntitySupervisor1, :"[email protected]"}, status: :shutting_down}}]}
State: %Horde.SupervisorImpl{distribution_strategy: Horde.UniformQuorumDistribution, members: %{{MyApplication.EntitySupervisor1, :"[email protected]"} => 1, {MyApplication.EntitySupervisor1, :"[email protected]"} => 1}, members_info: %{{MyApplication.EntitySupervisor1, :"[email protected]"} => %Horde.Supervisor.Member{name: {MyApplication.EntitySupervisor1, :"[email protected]"}, status: :alive}, {MyApplication.EntitySupervisor1, :"[email protected]"} => %Horde.Supervisor.Member{name: {MyApplication.EntitySupervisor1, :"[email protected]"}, status: :dead}}, name: MyApplication.EntitySupervisor1, name_to_supervisor_ref: %{{MyApplication.EntitySupervisor1, :"[email protected]"} => #Reference<0.3609365875.1531183106.30390>}, process_pid_to_id: %{#PID<0.628.0> => 314042693166365608091164111263343981781, #PID<20246.738.0> => 221419068907050945833165595071189253006, #PID<20100.828.0> => 314042693166365608091164111263343981781}, processes_by_id: %{221419068907050945833165595071189253006 => {{MyApplication.EntitySupervisor1, :"[email protected]"}, %{id: 221419068907050945833165595071189253006, restart: :transient, shutdown: 10000, start: {MyApplication.IpScanner, :start_link, ["5.6.7.8"]}}, #PID<20246.738.0>}, 314042693166365608091164111263343981781 => {{MyApplication.EntitySupervisor1, :"[email protected]"}, %{id: 314042693166365608091164111263343981781, restart: :transient, shutdown: 10000, start: {MyApplication.IpScanner, :start_link, ["1.2.3.4"]}}, #PID<0.628.0>}}, processes_updated_at: 0, processes_updated_counter: 0, shutting_down: true, supervisor_options: [members: [{MyApplication.EntitySupervisor1, :"[email protected]"}], name: MyApplication.EntitySupervisor1, root_name: MyApplication.EntitySupervisor1, init_module: MyApplication.DistributedSupervisor, id: :ip_supervisor_1, strategy: :one_for_one, distribution_strategy: Horde.UniformQuorumDistribution, max_restarts: 1000, max_seconds: 1], supervisor_ref_to_name: %{#Reference<0.3609365875.1531183106.30390> => {MyApplication.EntitySupervisor1, :"[email protected]"}}, waiting_for_quorum: []}

2019-08-14T13:52:31.997000 [info] Starting Horde.SupervisorImpl with name MyApplication.EntitySupervisor1

2019-08-14T13:52:31.998000 [error] GenServer MyApplication.Cluster.NodeListener terminating
** (stop) exited in: GenServer.call(MyApplication.EntitySupervisor1, {:set_members, [{MyApplication.EntitySupervisor1, :"[email protected]"}]}, 5000)
    ** (EXIT) exited in: GenServer.stop(MyApplication.EntitySupervisor1.ProcessesSupervisor, :normal, :infinity)
        ** (EXIT) no process: the process is not alive or there's no process currently associated with the given name, possibly because its application isn't started
    (elixir) lib/gen_server.ex:989: GenServer.call/3
    (my_application) lib/my_application/cluster/node_listener.ex:30: anonymous fn/2 in MyApplication.Cluster.NodeListener.set_members/1
    (elixir) lib/enum.ex:1940: Enum."-reduce/3-lists^foldl/2-0-"/3
    (my_application) lib/my_application/cluster/node_listener.ex:24: MyApplication.Cluster.NodeListener.set_members/1
    (my_application) lib/my_application/cluster/node_listener.ex:19: MyApplication.Cluster.NodeListener.handle_info/2
    (stdlib) gen_server.erl:637: :gen_server.try_dispatch/4
    (stdlib) gen_server.erl:711: :gen_server.handle_msg/6
    (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
Last message: {:nodedown, :"[email protected]", [node_type: :visible]}
State: [MyApplication.DistributedSupervisor, MyApplication.Registry, MyApplication.EntitySupervisor1, MyApplication.EntitySupervisor2, MyApplication.EntitySupervisor3, MyApplication.EntitySupervisor4, MyApplication.EntityRegistry1, MyApplication.EntityRegistry2, MyApplication.EntityRegistry3, MyApplication.EntityRegistry4]

2019-08-14T13:52:32.659000 [error] GenServer MyApplication.DistributedSupervisor terminating
** (stop) exited in: GenServer.stop(MyApplication.DistributedSupervisor.ProcessesSupervisor, :normal, :infinity)
    ** (EXIT) no process: the process is not alive or there's no process currently associated with the given name, possibly because its application isn't started
    (elixir) lib/gen_server.ex:938: GenServer.stop/3
    (horde) lib/horde/supervisor_impl.ex:571: Horde.SupervisorImpl.shut_down_all_processes/1
    (horde) lib/horde/supervisor_impl.ex:347: Horde.SupervisorImpl.handle_info/2
    (stdlib) gen_server.erl:637: :gen_server.try_dispatch/4
    (stdlib) gen_server.erl:711: :gen_server.handle_msg/6
    (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
Last message: {:crdt_update, [{:add, {:member, {MyApplication.DistributedSupervisor, :"[email protected]"}}, 1}]}
State: %Horde.SupervisorImpl{distribution_strategy: Horde.UniformQuorumDistribution, members: %{{MyApplication.DistributedSupervisor, :"[email protected]"} => 1, {MyApplication.DistributedSupervisor, :"[email protected]"} => 1}, members_info: %{{MyApplication.DistributedSupervisor, :"[email protected]"} => %Horde.Supervisor.Member{name: {MyApplication.DistributedSupervisor, :"[email protected]"}, status: :alive}, {MyApplication.DistributedSupervisor, :"[email protected]"} => %Horde.Supervisor.Member{name: {MyApplication.DistributedSupervisor, :"[email protected]"}, status: :uninitialized}}, name: MyApplication.DistributedSupervisor, name_to_supervisor_ref: %{{MyApplication.DistributedSupervisor, :"[email protected]"} => #Reference<0.3609365875.1531183105.27253>}, process_pid_to_id: %{#PID<0.638.0> => 224928341895112729021255116702455286098, #PID<20100.747.0> => 224928341895112729021255116702455286098}, processes_by_id: %{224928341895112729021255116702455286098 => {{MyApplication.DistributedSupervisor, :"[email protected]"}, %{id: 224928341895112729021255116702455286098, restart: :permanent, shutdown: 10000, start: {MyApplication.ScannerResultWriter, :start_link, []}}, #PID<0.638.0>}}, processes_updated_at: 0, processes_updated_counter: 0, shutting_down: false, supervisor_options: [members: [{MyApplication.DistributedSupervisor, :"[email protected]"}, {MyApplication.DistributedSupervisor, :"[email protected]"}], name: MyApplication.DistributedSupervisor, root_name: MyApplication.DistributedSupervisor, init_module: MyApplication.DistributedSupervisor, id: MyApplication.DistributedSupervisor, strategy: :one_for_one, distribution_strategy: Horde.UniformQuorumDistribution, max_restarts: 1000, max_seconds: 1], supervisor_ref_to_name: %{#Reference<0.3609365875.1531183105.27253> => {MyApplication.DistributedSupervisor, :"[email protected]"}}, waiting_for_quorum: []}

2019-08-14T13:52:32.660000 [error] GenServer #PID<0.429.0> terminating
** (stop) exited in: GenServer.call(MyApplication.DistributedSupervisor, :horde_shutting_down, 5000)
    ** (EXIT) exited in: GenServer.stop(MyApplication.DistributedSupervisor.ProcessesSupervisor, :normal, :infinity)
        ** (EXIT) no process: the process is not alive or there's no process currently associated with the given name, possibly because its application isn't started
    (elixir) lib/gen_server.ex:989: GenServer.call/3
    (horde) lib/horde/signal_shutdown.ex:21: anonymous fn/1 in Horde.SignalShutdown.terminate/2
    (elixir) lib/enum.ex:769: Enum."-each/2-lists^foreach/1-0-"/2
    (elixir) lib/enum.ex:769: Enum.each/2
    (stdlib) gen_server.erl:673: :gen_server.try_terminate/3
    (stdlib) gen_server.erl:858: :gen_server.terminate/10
    (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
Last message: {:EXIT, #PID<0.334.0>, :shutdown}
State: [MyApplication.DistributedSupervisor.GracefulShutdownManager, MyApplication.DistributedSupervisor]

2019-08-14T13:52:32.709000 [info] Starting Horde.SupervisorImpl with name MyApplication.DistributedSupervisor

09:52:35.323 [info]  Starting Horde.RegistryImpl with name MyApplication.Registry

09:52:35.323 [info]  Starting Horde.RegistryImpl with name MyApplication.Registry

09:52:35.325 [info]  Starting Horde.RegistryImpl with name MyApplication.EntityRegistry1

09:52:35.325 [info]  Starting Horde.RegistryImpl with name MyApplication.EntityRegistry1

09:52:35.326 [info]  Starting Horde.RegistryImpl with name MyApplication.EntityRegistry2

09:52:35.326 [info]  Starting Horde.RegistryImpl with name MyApplication.EntityRegistry2

09:52:35.327 [info]  Starting Horde.RegistryImpl with name MyApplication.EntityRegistry3

09:52:35.327 [info]  Starting Horde.RegistryImpl with name MyApplication.EntityRegistry4

09:52:35.327 [info]  Starting Horde.RegistryImpl with name MyApplication.EntityRegistry3

09:52:35.327 [info]  Starting Horde.RegistryImpl with name MyApplication.EntityRegistry4

09:52:35.386 [info]  Starting Horde.SupervisorImpl with name MyApplication.DistributedSupervisor

09:52:35.386 [info]  Starting Horde.SupervisorImpl with name MyApplication.DistributedSupervisor

09:52:35.386 [info]  Starting Horde.SupervisorImpl with name MyApplication.EntitySupervisor1

09:52:35.387 [info]  Starting Horde.SupervisorImpl with name MyApplication.EntitySupervisor2

09:52:35.387 [info]  Starting Horde.SupervisorImpl with name MyApplication.EntitySupervisor3

09:52:35.386 [info]  Starting Horde.SupervisorImpl with name MyApplication.EntitySupervisor1

09:52:35.388 [info]  Starting Horde.SupervisorImpl with name MyApplication.EntitySupervisor4

09:52:35.387 [info]  Starting Horde.SupervisorImpl with name MyApplication.EntitySupervisor2

09:52:35.388 [info]  Starting Horde.SupervisorImpl with name MyApplication.EntitySupervisor3

09:52:35.389 [info]  Starting Horde.SupervisorImpl with name MyApplication.EntitySupervisor4

09:52:35.439 [info]  Starting up Elixir.MyApplication.Worker with 4 schedulers online

09:52:35.439 [info]  Starting up Elixir.MyApplication.Worker with 4 schedulers online
warning: redefining module MyApplication.DistributedTest (current version defined in memory)
  test/my_application/distributed_test.exs:1

Test nodes that are now online: [:"[email protected]", :"[email protected]"]
.

2019-08-14T13:52:35.918000 [error] GenServer MyApplication.EntitySupervisor2 terminating
** (stop) exited in: GenServer.stop(MyApplication.EntitySupervisor2.ProcessesSupervisor, :normal, :infinity)
    ** (EXIT) no process: the process is not alive or there's no process currently associated with the given name, possibly because its application isn't started
    (elixir) lib/gen_server.ex:938: GenServer.stop/3
    (horde) lib/horde/supervisor_impl.ex:571: Horde.SupervisorImpl.shut_down_all_processes/1
    (horde) lib/horde/supervisor_impl.ex:347: Horde.SupervisorImpl.handle_info/2
    (stdlib) gen_server.erl:637: :gen_server.try_dispatch/4
    (stdlib) gen_server.erl:711: :gen_server.handle_msg/6
    (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
Last message: {:crdt_update, [{:add, {:member_node_info, {MyApplication.EntitySupervisor2, :"[email protected]"}}, %Horde.Supervisor.Member{name: {MyApplication.EntitySupervisor2, :"[email protected]"}, status: :shutting_down}}]}
State: %Horde.SupervisorImpl{distribution_strategy: Horde.UniformQuorumDistribution, members: %{{MyApplication.EntitySupervisor2, :"[email protected]"} => 1, {MyApplication.EntitySupervisor2, :"[email protected]"} => 1, {MyApplication.EntitySupervisor2, :"[email protected]"} => 1}, members_info: %{{MyApplication.EntitySupervisor2, :"[email protected]"} => %Horde.Supervisor.Member{name: {MyApplication.EntitySupervisor2, :"[email protected]"}, status: :alive}, {MyApplication.EntitySupervisor2, :"[email protected]"} => %Horde.Supervisor.Member{name: {MyApplication.EntitySupervisor2, :"[email protected]"}, status: :dead}, {MyApplication.EntitySupervisor2, :"[email protected]"} => %Horde.Supervisor.Member{name: {MyApplication.EntitySupervisor2, :"[email protected]"}, status: :dead}}, name: MyApplication.EntitySupervisor2, name_to_supervisor_ref: %{{MyApplication.EntitySupervisor2, :"[email protected]"} => #Reference<0.3609365875.1531183106.30401>}, process_pid_to_id: %{#PID<20100.825.0> => 248425850269184959969685862485213231097, #PID<20100.829.0> => 128476685755369599155497243660738762471, #PID<0.866.0> => 248425850269184959969685862485213231097}, processes_by_id: %{128476685755369599155497243660738762471 => {{MyApplication.EntitySupervisor2, :"[email protected]"}, %{id: 128476685755369599155497243660738762471, restart: :transient, shutdown: 10000, start: {MyApplication.IpScanner, :start_link, ["2.3.4.5"]}}, #PID<20100.829.0>}, 248425850269184959969685862485213231097 => {{MyApplication.EntitySupervisor2, :"[email protected]"}, %{id: 248425850269184959969685862485213231097, restart: :transient, shutdown: 10000, start: {MyApplication.IpScanner, :start_link, ["2.3.4.5"]}}, #PID<0.866.0>}}, processes_updated_at: 0, processes_updated_counter: 0, shutting_down: true, supervisor_options: [members: [{MyApplication.EntitySupervisor2, :"[email protected]"}], name: MyApplication.EntitySupervisor2, root_name: MyApplication.EntitySupervisor2, init_module: MyApplication.DistributedSupervisor, id: :ip_supervisor_2, strategy: :one_for_one, distribution_strategy: Horde.UniformQuorumDistribution, max_restarts: 1000, max_seconds: 1], supervisor_ref_to_name: %{#Reference<0.3609365875.1531183106.30401> => {MyApplication.EntitySupervisor2, :"[email protected]"}}, waiting_for_quorum: []}

2019-08-14T13:52:35.918000 [info] Starting Horde.SupervisorImpl with name MyApplication.EntitySupervisor2
Finished in 7.9 seconds
2 tests, 0 failures

Randomized with seed 976897

@mgwidmann mgwidmann mentioned this issue Aug 15, 2019
Open
@derekkraan derekkraan added the bug Something isn't working label Dec 24, 2019
@brndnmtthws brndnmtthws mentioned this issue Feb 23, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@derekkraan @mgwidmann