Service dependency failed after the system reboot

[seberm]

Hello,
when I reboot the VM with nix-bitcoin, most of the services which depend on a bitcoind.service fail with the "Dependency failed for 'service_name'" error and they are not started/restarted at all.

It's a pain especially when the clightning.service is not started because most of other services depend on it:

$ systemctl status clightning.service
○ clightning.service
     Loaded: loaded (/etc/systemd/system/clightning.service; enabled; preset: enabled)
     Active: inactive (dead)

Mar 03 13:53:27 nix-bitcoin systemd[1]: Dependency failed for clightning.service.
Mar 03 13:53:27 nix-bitcoin systemd[1]: clightning.service: Job clightning.service/start failed with result 'dependency'.

$ systemctl status btcpayserver.service
○ btcpayserver.service
     Loaded: loaded (/etc/systemd/system/btcpayserver.service; enabled; preset: enabled)
     Active: inactive (dead)

Mar 03 13:53:27 nix-bitcoin systemd[1]: Dependency failed for btcpayserver.service.
Mar 03 13:53:27 nix-bitcoin systemd[1]: btcpayserver.service: Job btcpayserver.service/start failed with result 'dependency'.

$ systemctl status mempool.service
○ mempool.service
     Loaded: loaded (/etc/systemd/system/mempool.service; enabled; preset: enabled)
     Active: inactive (dead)

Mar 03 13:53:27 nix-bitcoin systemd[1]: Dependency failed for mempool.service.
Mar 03 13:53:27 nix-bitcoin systemd[1]: mempool.service: Job mempool.service/start failed with result 'dependency'.

$ systemctl status electrs.service
○ electrs.service
     Loaded: loaded (/etc/systemd/system/electrs.service; enabled; preset: enabled)
     Active: inactive (dead)

Mar 03 13:53:27 nix-bitcoin systemd[1]: Dependency failed for electrs.service.
Mar 03 13:53:27 nix-bitcoin systemd[1]: electrs.service: Job electrs.service/start failed with result 'dependency'

$ nodeinfo
{
  "bitcoind": {
    "local_address": "127.0.0.1:8333",
    "onion_address": "addr.onion:8333"
  },
  "btcpayserver": "'btcpayserver.service' is not running",
  "clightning": "'clightning.service' is not running",
  "clightning-rest": "'clightning-rest.service' is not running",
  "electrs": "'electrs.service' is not running",
  "mempool": "'mempool.service' is not running",
  "mempool-frontend": {
    "local_address": "127.0.0.1:60845"
  },
  "rtl": "'rtl.service' is not running",
  "sshd": {
    "onion_address": "addr.onion:22"
  }
}

This behavior is reproducible in 90% of reboots. I am running latest nix-bitcoin (v0.0.87) on:

$ cat /etc/os-release
BUILD_ID="22.11pre-git"
PRETTY_NAME="NixOS 22.11 (Raccoon)"
VERSION="22.11 (Raccoon)"
...

I think it could be possible that it just takes too long to start bitcoind unit and initialize the Tor network which results in dependent units timing out. What do you think?

My current workaround is just to manually restart of all the failed units.

[jonasnick]

Can you see in the bitcoind logs that it takes long to start? Are there any errors in bitcoind logs?

[seberm]

Hello Jonas,
I finally found the problem which is causing this issue.

The problem:

1. I run krops-deploy, which does rsync of my secrets to a remote node with my local permissions. These are:

$ ls -alh /var/src/secrets/
total 80K
drwx------ 2 root       root              4.0K Mar  3 19:38 .
drwxr-xr-x 6 root       root              4.0K Mar  3 13:48 ..
-rw------- 1 root       root                32 Mar  3 19:23 backup-encryption-env
-rw------- 1 root       root                21 Mar  3 19:23 backup-encryption-password
-rw-r--r-- 1 bitcoin    bitcoin             98 Aug 25  2022 bitcoin-HMAC-btcpayserver
-rw-r--r-- 1 bitcoin    bitcoin             98 Aug 25  2022 bitcoin-HMAC-privileged
-rw-r--r-- 1 bitcoin    bitcoin-exporter    98 Dec 30 14:32 bitcoin-HMAC-prometheus
-rw-r--r-- 1 bitcoin    bitcoin             98 Aug 25  2022 bitcoin-HMAC-public
-rw-r--r-- 1 bitcoin    nbxplorer           21 Aug 24  2022 bitcoin-rpcpassword-btcpayserver
-rw-r--r-- 1 bitcoin    bitcoin             21 Aug 24  2022 bitcoin-rpcpassword-privileged
-rw-r--r-- 1 bitcoin    bitcoin-exporter    21 Dec 30 12:38 bitcoin-rpcpassword-prometheus
-rw-r--r-- 1 bitcoin    bitcoinrpc-public   21 Aug 24  2022 bitcoin-rpcpassword-public
-rw------- 1 clightning clightning          21 Mar  3 19:38 clightning-replication-password
-rw------- 1 nginx      nginx              692 Nov 15 13:17 nginx-btcpayserver-cert
-rw-r--r-- 1 root       root                 1 Nov 15 13:17 nginx-btcpayserver-cert-alt-names
-rw------- 1 nginx      nginx              241 Nov 15 13:17 nginx-btcpayserver-key
-rw-r--r-- 1 root       root               662 Dec 26 22:49 README.md
-rw-r--r-- 1 root       root                26 Jan 31 23:04 root_password
-rw-r--r-- 1 rtl        rtl                 21 Aug 23  2022 rtl-password
-rw-r--r-- 1 root       root               241 Dec 25 20:18 tor-bridges

2. Because I do stop all of the services before the deployment using krops writeCommand, I need to reboot the remote machine.

krops.pkgs.krops.writeCommand "deploy-boot" {
  inherit target;
  source = import ./sources.nix { inherit extraSources krops; };

  command = targetPath: ''
    systemctl stop btcpayserver
    systemctl stop mempool
    systemctl stop electrs

    systemctl stop clightning
    systemctl stop bitcoind

    sync

    nixos-rebuild -I ${targetPath} boot
  '';
...
}

3. Machine is rebooted and you were correct, there are erros in journalctl -u bitcoind:

Mar 04 22:01:39 nix-bitcoin systemd[1]: Starting bitcoind.service...
Mar 04 22:01:39 nix-bitcoin bitcoind-pre-start[1003]: cat: /var/src/secrets/bitcoin-HMAC-btcpayserver: Permission denied
Mar 04 22:01:39 nix-bitcoin bitcoind-pre-start[1004]: cat: /var/src/secrets/bitcoin-HMAC-privileged: Permission denied
Mar 04 22:01:39 nix-bitcoin bitcoind-pre-start[1005]: cat: /var/src/secrets/bitcoin-HMAC-prometheus: Permission denied
Mar 04 22:01:39 nix-bitcoin bitcoind-pre-start[1006]: cat: /var/src/secrets/bitcoin-HMAC-public: Permission denied
Mar 04 22:01:39 nix-bitcoin bitcoind[1010]: 2023-03-04T21:01:39Z Ignoring unknown configuration value rpcconnect
Mar 04 22:01:39 nix-bitcoin bitcoind[1010]: Bitcoin Core version v24.0.1 (release build)
...
Mar 04 22:01:39 nix-bitcoin bitcoind[1010]: Using rpcauth authentication.
Mar 04 22:01:39 nix-bitcoin bitcoind[1010]: Invalid -rpcauth argument.
Mar 04 22:01:39 nix-bitcoin bitcoind[1010]: Error: Unable to start HTTP server. See debug log for details.
Mar 04 22:01:39 nix-bitcoin bitcoind[1010]: Error: Unable to start HTTP server. See debug log for details.
Mar 04 22:01:39 nix-bitcoin bitcoind[1010]: Shutdown: In progress...
Mar 04 22:01:39 nix-bitcoin bitcoind[1010]: scheduler thread exit
Mar 04 22:01:39 nix-bitcoin bitcoind[1010]: Shutdown: done
Mar 04 22:01:39 nix-bitcoin systemd[1]: bitcoind.service: Main process exited, code=exited, status=1/FAILURE
Mar 04 22:01:39 nix-bitcoin systemd[1]: bitcoind.service: Failed with result 'exit-code'.
Mar 04 22:01:39 nix-bitcoin systemd[1]: Failed to start bitcoind.service.

That is why all the dependent units fail. These errors are caused because the setup-secrets.service probably hasn't finished before the bitcoind was started.

The reason I have overlooked the error is because the bitcoind service is automatically restarted and then it's running without any errors.

It's running without any issues, because the setup-secrets.service unit has finished meanwhile.

Then the permissions are set to:

$ ls -alh /var/src/secrets/
total 80K
drwxr-x--x 2 root       root              4.0K Mar  3 19:38 .
drwxr-xr-x 6 root       root              4.0K Mar  3 13:48 ..
-r--r----- 1 root       root                32 Mar  3 19:23 backup-encryption-env
-r--r----- 1 root       root                21 Mar  3 19:23 backup-encryption-password
-r--r----- 1 bitcoin    bitcoin             98 Aug 25  2022 bitcoin-HMAC-btcpayserver
-r--r----- 1 bitcoin    bitcoin             98 Aug 25  2022 bitcoin-HMAC-privileged
-r--r----- 1 bitcoin    bitcoin-exporter    98 Dec 30 14:32 bitcoin-HMAC-prometheus
-r--r----- 1 bitcoin    bitcoin             98 Aug 25  2022 bitcoin-HMAC-public
-r--r----- 1 bitcoin    nbxplorer           21 Aug 24  2022 bitcoin-rpcpassword-btcpayserver
-r--r----- 1 bitcoin    bitcoin             21 Aug 24  2022 bitcoin-rpcpassword-privileged
-r--r----- 1 bitcoin    bitcoin-exporter    21 Dec 30 12:38 bitcoin-rpcpassword-prometheus
-r--r----- 1 bitcoin    bitcoinrpc-public   21 Aug 24  2022 bitcoin-rpcpassword-public
-r--r----- 1 clightning clightning          21 Mar  3 19:38 clightning-replication-password
-r--r----- 1 nginx      nginx              692 Nov 15 13:17 nginx-btcpayserver-cert
-r--r----- 1 root       root                 1 Nov 15 13:17 nginx-btcpayserver-cert-alt-names
-r--r----- 1 nginx      nginx              241 Nov 15 13:17 nginx-btcpayserver-key
-r--r----- 1 root       root               662 Dec 26 22:49 README.md
-r--r----- 1 root       root                26 Jan 31 23:04 root_password
-r--r----- 1 rtl        rtl                 21 Aug 23  2022 rtl-password
-r--r----- 1 root       root               241 Dec 25 20:18 tor-bridges

Workaround

Is it possible that bitcoind is started before the setup-secrets.service/nix-bitcoin-secrets.target hasn't finished?

My current workaround is just to restart the setup-secrets.service right after the deployment in the krops writeCommand, so the correct permissions are set before the machine reboots.

Thanks!

[elsirion]

I have a similar problem: bitcoind crashes from time to time (unreliable fs cache, need to fix) and clightning stops but starts again, but clightning-rest, rtl and spark-wallet stay in the "dependency failed" state.

Systemd output

bitcoind journal around the crash time:

May 10 11:45:43 btc-2 bitcoind[1555235]: ERROR: ReadRawBlockFromDisk: Block magic mismatch for FlatFilePos(nFile=3380, nPos=53835290): 00000000 versus expected f9beb4d9
May 10 11:45:43 btc-2 bitcoind[1555235]: bitcoind: net_processing.cpp:2149: void {anonymous}::PeerManagerImpl::ProcessGetBlockData(CNode&, {anonymous}::Peer&, const CInv&): Assertion `!"cannot load block from disk"' failed.
May 10 11:45:51 btc-2 systemd[1]: bitcoind.service: Main process exited, code=killed, status=6/ABRT
May 10 11:45:51 btc-2 systemd[1]: bitcoind.service: Failed with result 'signal'.
May 10 11:45:51 btc-2 systemd[1]: bitcoind.service: Consumed 5min 8.276s CPU time, received 104.5M IP traffic, sent 2.9G IP traffic.
May 10 11:45:51 btc-2 systemd[1]: bitcoind.service: Scheduled restart job, restart counter is at 2.
May 10 11:45:51 btc-2 systemd[1]: Stopped bitcoind.service.
May 10 11:45:51 btc-2 systemd[1]: bitcoind.service: Consumed 5min 8.276s CPU time, received 104.5M IP traffic, sent 2.9G IP traffic.
May 10 11:45:51 btc-2 systemd[1]: Starting bitcoind.service...
May 10 11:45:52 btc-2 bitcoind[1557879]: 2023-05-10T11:45:52Z Ignoring unknown configuration value rpcconnect
May 10 11:45:52 btc-2 bitcoind[1557879]: Bitcoin Core version v24.0.1 (release build)
May 10 11:45:52 btc-2 bitcoind[1557879]: Using the 'x86_shani(1way,2way)' SHA256 implementation
May 10 11:45:52 btc-2 bitcoind[1557879]: Using RdSeed as an additional entropy source
…
May 10 11:46:05 btc-2 bitcoind[1557879]: net thread start
May 10 11:46:05 btc-2 bitcoind[1557879]: addcon thread start
May 10 11:46:05 btc-2 bitcoind[1557879]: msghand thread start
May 10 11:46:05 btc-2 bitcoind[1557879]: opencon thread start
May 10 11:46:05 btc-2 bitcoind[1557879]: dnsseed thread start
May 10 11:46:05 btc-2 bitcoind[1557879]: Waiting 300 seconds before querying DNS seeds.
May 10 11:46:05 btc-2 bitcoind[1557879]: UpdateTip: new best=000000000000000000022f5c52126c931c9d504574be31236d57293ea8567783 height=788975 version=0x2fffe000 log2_work=94.169366 tx=834958440 date='2023-05-09T20:04:39Z' progress=0.999802 cache=1.8MiB(12777txo)
May 10 11:46:05 btc-2 bitcoind[1557879]: init message: Done loading
May 10 11:46:05 btc-2 systemd[1]: Started bitcoind.service.
May 10 11:46:07 btc-2 bitcoind[1557879]: UpdateTip: new best=00000000000000000005b938bff437192403c98b4910aebd18950441313617df height=788976 version=0x20002000 log2_work=94.169379 tx=834961732 date='2023-05-09T20:07:20Z' progress=0.999803 cache=3.4MiB(23093txo)
…
May 10 11:46:57 btc-2 bitcoind[1557879]: Leaving InitialBlockDownload (latching to false)
May 10 11:47:03 btc-2 bitcoind[1557879]: UpdateTip: new best=000000000000000000023ac6d1e121cdedd5fb54fb7a69b43511f0150466ab86 height=789066 version=0x3fff0000 log2_work=94.170581 tx=835327909 date='2023-05-10T11:21:14Z' progress=0.999995 cache=116.4MiB(765810txo)
May 10 11:47:03 btc-2 bitcoind[1557879]: UpdateTip: new best=00000000000000000002fe2e6894f341e36856b3c01ade1a467d43682493a0ab height=789067 version=0x20c00000 log2_work=94.170594 tx=835332576 date='2023-05-10T11:21:43Z' progress=0.999995 cache=117.3MiB(772839txo)
May 10 11:47:04 btc-2 bitcoind[1557879]: UpdateTip: new best=00000000000000000002c6be865bd1f67df8935bff5e6c325a8dd7d1ead0665c height=789068 version=0x26bfe000 log2_work=94.170607 tx=835336991 date='2023-05-10T11:22:16Z' progress=0.999995 cache=118.5MiB(781703txo)
May 10 11:47:04 btc-2 bitcoind[1557879]: UpdateTip: new best=000000000000000000055c85d5f78ad3e08a5bf825a56d0a756a476b475e1fa4 height=789069 version=0x25ce4000 log2_work=94.170621 tx=835341002 date='2023-05-10T11:23:34Z' progress=0.999995 cache=119.9MiB(792108txo)
May 10 11:47:10 btc-2 bitcoind[1557879]: Imported mempool transactions from disk: 8584 succeeded, 10732 failed, 0 expired, 0 already there, 0 waiting for initial broadcast
May 10 11:47:10 btc-2 bitcoind[1557879]: loadblk thread exit

clightning journal around the crash time:

May 10 11:45:51 btc-2 lightningd[1556215]: UNUSUAL plugin-bcli: bitcoin-cli -datadir=/bitcoin -rpcconnect=127.0.0.1 -rpcport=8332 -rpcuser=... -stdinrpcpass getblockhash 789070 exited with status 1
May 10 11:45:51 btc-2 systemd[1]: Stopping clightning.service...
May 10 11:45:51 btc-2 systemd[1]: clightning.service: Deactivated successfully.
May 10 11:45:51 btc-2 systemd[1]: Stopped clightning.service.
May 10 11:45:51 btc-2 systemd[1]: clightning.service: Consumed 1min 52.006s CPU time, received 170.3M IP traffic, sent 17.0M IP traffic.
May 10 11:46:05 btc-2 systemd[1]: Starting clightning.service...
May 10 11:46:12 btc-2 lightningd[1557919]: INFO    plugin-bcli: bitcoin-cli initialized and connected to bitcoind.
May 10 11:46:13 btc-2 lightningd[1557919]: bitcoind has gone backwards from 789054 to 788984 blocks!
May 10 11:46:13 btc-2 systemd[1]: clightning.service: Main process exited, code=exited, status=1/FAILURE
May 10 11:47:36 btc-2 systemd[1]: clightning.service: start-post operation timed out. Terminating.
May 10 11:47:36 btc-2 systemd[1]: clightning.service: Control process exited, code=killed, status=15/TERM
May 10 11:47:36 btc-2 systemd[1]: clightning.service: Failed with result 'exit-code'.
May 10 11:47:36 btc-2 systemd[1]: Failed to start clightning.service.
May 10 11:47:36 btc-2 systemd[1]: clightning.service: Consumed 7.904s CPU time, received 2.8K IP traffic, sent 2.0K IP traffic.
May 10 11:47:46 btc-2 systemd[1]: clightning.service: Scheduled restart job, restart counter is at 1.
May 10 11:47:46 btc-2 systemd[1]: Stopped clightning.service.
May 10 11:47:46 btc-2 systemd[1]: clightning.service: Consumed 7.904s CPU time, received 2.8K IP traffic, sent 2.0K IP traffic.
May 10 11:47:46 btc-2 systemd[1]: Starting clightning.service...
May 10 11:47:50 btc-2 lightningd[1558825]: INFO    plugin-bcli: bitcoin-cli initialized and connected to bitcoind.
May 10 11:47:54 btc-2 lightningd[1558825]: INFO    plugin-monitor.py: Plugin monitor.py initialized
May 10 11:47:54 btc-2 lightningd[1558825]: INFO    plugin-cln-zapper: 1
May 10 11:47:54 btc-2 lightningd[1558825]: INFO    plugin-cln-zapper: Waiting for index: 1
May 10 11:47:54 btc-2 systemd[1]: Started clightning.service.

systemd output:

○ rtl.service
     Loaded: loaded (/etc/systemd/system/rtl.service; enabled; preset: enabled)
     Active: inactive (dead) since Wed 2023-05-10 10:24:23 UTC; 9h ago
   Duration: 1h 50min 17.037s
    Process: 1552092 ExecStartPre=/nix/store/ym21fvcrrga8026wf4d1kd34gfjjvqik-rtl-setup-config (code=exited, status=0/SUCCESS)
    Process: 1552096 ExecStart=/nix/store/f64civv948i5igz7hag91wm67pm7apfc-rtl-0.13.6/bin/rtl (code=killed, signal=TERM)
   Main PID: 1552096 (code=killed, signal=TERM)
         IP: 4.7M in, 76.4K out
        CPU: 2.015s

May 10 08:38:50 btc-2 rtl[1552096]: [5/10/2023, 8:38:50 AM] INFO: Channels => Channels List Received.
May 10 08:38:50 btc-2 rtl[1552096]: [5/10/2023, 8:38:50 AM] INFO: Peers => Peers with Alias Received.
May 10 09:23:03 btc-2 rtl[1552096]: [5/10/2023, 9:23:03 AM] INFO: Channels => Getting Channel List Forwards...
May 10 10:23:23 btc-2 rtl[1552096]: [5/10/2023, 10:23:23 AM] INFO: Authenticate => Logged out.
May 10 10:24:23 btc-2 systemd[1]: Stopping rtl.service...
May 10 10:24:23 btc-2 systemd[1]: rtl.service: Deactivated successfully.
May 10 10:24:23 btc-2 systemd[1]: Stopped rtl.service.
May 10 10:24:23 btc-2 systemd[1]: rtl.service: Consumed 2.015s CPU time, received 4.7M IP traffic, sent 76.4K IP traffic.
May 10 10:26:13 btc-2 systemd[1]: Dependency failed for rtl.service.
May 10 10:26:13 btc-2 systemd[1]: rtl.service: Job rtl.service/start failed with result 'dependency'.

○ spark-wallet.service
     Loaded: loaded (/etc/systemd/system/spark-wallet.service; enabled; preset: enabled)
     Active: inactive (dead) since Wed 2023-05-10 10:24:23 UTC; 9h ago
   Duration: 1h 50min 17.270s
    Process: 1552078 ExecStart=/nix/store/l4pcffnr3kgi4m084jm1sdkyhb8g9zyc-unit-script-spark-wallet-start/bin/spark-wallet-start (code=exited, status=0/SUCCESS)
   Main PID: 1552078 (code=exited, status=0/SUCCESS)
         IP: 2.9K in, 1.9K out
        CPU: 1.408s

May 10 10:20:36 btc-2 spark-wallet-start[1552078]:     at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1159:16)
May 10 10:23:45 btc-2 spark-wallet-start[1552078]: Error: connect ETIMEDOUT 107.154.133.13:443
May 10 10:23:45 btc-2 spark-wallet-start[1552078]:     at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1159:16)
May 10 10:24:23 btc-2 systemd[1]: Stopping spark-wallet.service...
May 10 10:24:23 btc-2 spark-wallet-start[1552078]: Caught SIGTERM, shutting down
May 10 10:24:23 btc-2 systemd[1]: spark-wallet.service: Deactivated successfully.
May 10 10:24:23 btc-2 systemd[1]: Stopped spark-wallet.service.
May 10 10:24:23 btc-2 systemd[1]: spark-wallet.service: Consumed 1.408s CPU time, received 2.9K IP traffic, sent 1.9K IP traffic.
May 10 10:26:13 btc-2 systemd[1]: Dependency failed for spark-wallet.service.
May 10 10:26:13 btc-2 systemd[1]: spark-wallet.service: Job spark-wallet.service/start failed with result 'dependency'.

○ clightning-rest.service
     Loaded: loaded (/etc/systemd/system/clightning-rest.service; enabled; preset: enabled)
     Active: inactive (dead) since Wed 2023-05-10 10:24:23 UTC; 9h ago
   Duration: 1h 50min 17.148s
    Process: 1552077 ExecStartPre=/nix/store/hicrgp5c1ndbfwh2698bcwb5ny4k1h66-unit-script-clightning-rest-pre-start/bin/clightning-rest-pre-start (code=exited, status=0/SUCCESS)
    Process: 1552082 ExecStart=/nix/store/0rk0ldjxxf0xnb1pz3nfsh0ssghbg2jn-clightning-rest-0.9.0/bin/cl-rest (code=exited, status=0/SUCCESS)
    Process: 1552083 ExecStartPost=/nix/store/2pj40qdlkp7wj09apmlp5qqkr1dfak37-unit-script-clightning-rest-post-start/bin/clightning-rest-post-start (code=exited, status=0/SUCCESS)
   Main PID: 1552082 (code=exited, status=0/SUCCESS)
         IP: 19.2K in, 4.6M out
        CPU: 2.228s

May 10 08:34:08 btc-2 clightning-rest[1552082]: warn: --- Starting the cl-rest server ---
May 10 08:34:08 btc-2 clightning-rest[1552082]: warn: --- cl-rest api server is ready and listening on port: 3001 ---
May 10 08:34:08 btc-2 clightning-rest[1552082]: warn: --- cl-rest doc server is ready and listening on port: 4001 ---
May 10 10:24:23 btc-2 clightning-rest[1552082]: warn: Client disconnected, total clients: 0
May 10 10:24:23 btc-2 systemd[1]: Stopping clightning-rest.service...
May 10 10:24:23 btc-2 systemd[1]: clightning-rest.service: Deactivated successfully.
May 10 10:24:23 btc-2 systemd[1]: Stopped clightning-rest.service.
May 10 10:24:23 btc-2 systemd[1]: clightning-rest.service: Consumed 2.228s CPU time, received 19.2K IP traffic, sent 4.6M IP traffic.
May 10 10:26:13 btc-2 systemd[1]: Dependency failed for clightning-rest.service.
May 10 10:26:13 btc-2 systemd[1]: clightning-rest.service: Job clightning-rest.service/start failed with result 'dependency'.

● bitcoind.service
     Loaded: loaded (/etc/systemd/system/bitcoind.service; enabled; preset: enabled)
     Active: active (running) since Wed 2023-05-10 11:46:05 UTC; 8h ago
    Process: 1557872 ExecStartPre=/nix/store/ns2xw9hd3j3qlv91qr4yzz057xc7j2id-unit-script-bitcoind-pre-start/bin/bitcoind-pre-start (code=exited, status=0/SUCCESS)
    Process: 1557910 ExecStartPost=/nix/store/s9j6ndhq1kpca6qyjgnssg2p12610wiv-unit-script-bitcoind-post-start/bin/bitcoind-post-start (code=exited, status=0/SUCCESS)
   Main PID: 1557879 (bitcoind)
         IP: 1.0G in, 38.2G out
         IO: 296.3G read, 1.0G written
      Tasks: 26 (limit: 9527)
     Memory: 6.1G
        CPU: 2h 13min 431ms
     CGroup: /system.slice/bitcoind.service
             └─1557879 /nix/store/a36chbn6hz23968jvh555c11xg28djy5-bitcoind-24.0.1/bin/bitcoind -datadir=/bitcoin

May 10 19:37:20 btc-2 bitcoind[1557879]: New outbound peer connected: version: 70016, blocks=789133, peer=3711 (block-relay-only)
May 10 19:46:56 btc-2 bitcoind[1557879]: Timeout downloading block 00000000000000000003bf94447ad5adc5859cd78d600000525851b7784a5599 from peer=3514, disconnecting
May 10 19:47:03 btc-2 bitcoind[1557879]: UpdateTip: new best=00000000000000000003bf94447ad5adc5859cd78d600000525851b7784a5599 height=789133 version=0x3fff0000 log2_work=94.171474 tx=835591984 date='2023-05-10T19:36:48Z' progress=0.999998 cache=235.0MiB(1585031txo)
May 10 19:48:42 btc-2 bitcoind[1557879]: UpdateTip: new best=000000000000000000002e547daae0c3421698dcdb55bc4af10debb08fd6530f height=789134 version=0x2d054000 log2_work=94.171488 tx=835595967 date='2023-05-10T19:48:28Z' progress=1.000000 cache=236.0MiB(1592518txo)
May 10 19:51:14 btc-2 bitcoind[1557879]: New outbound peer connected: version: 70016, blocks=789134, peer=3824 (block-relay-only)
May 10 19:52:57 btc-2 bitcoind[1557879]: ERROR: AcceptBlock: bad-witness-nonce-size, ContextualCheckBlock : invalid witness reserved value size
May 10 19:52:57 btc-2 bitcoind[1557879]: ERROR: ProcessNewBlock: AcceptBlock FAILED (bad-witness-nonce-size, ContextualCheckBlock : invalid witness reserved value size)
May 10 19:53:01 btc-2 bitcoind[1557879]: UpdateTip: new best=00000000000000000004a2b02d0d5d22631724ec0438118c54d10d8ba00124c5 height=789135 version=0x20a00000 log2_work=94.171501 tx=835599718 date='2023-05-10T19:52:32Z' progress=1.000000 cache=236.8MiB(1599138txo)
May 10 19:53:59 btc-2 bitcoind[1557879]: UpdateTip: new best=00000000000000000003ec91f2476f9098129b1d9d68159842615807c4941baf height=789136 version=0x2001e000 log2_work=94.171514 tx=835603278 date='2023-05-10T19:53:45Z' progress=1.000000 cache=237.6MiB(1604592txo)
May 10 19:54:34 btc-2 bitcoind[1557879]: New outbound peer connected: version: 70016, blocks=789136, peer=3847 (block-relay-only)

● clightning.service
     Loaded: loaded (/etc/systemd/system/clightning.service; enabled; preset: enabled)
     Active: active (running) since Wed 2023-05-10 11:47:54 UTC; 8h ago
    Process: 1558819 ExecStartPre=/nix/store/zm567pfnggl5nslxm1j0vls5636fzvn9-unit-script-clightning-pre-start/bin/clightning-pre-start (code=exited, status=0/SUCCESS)
    Process: 1558826 ExecStartPost=/nix/store/3lv8jihx427g5v4x2f5xzyl84jdx6khv-unit-script-clightning-post-start/bin/clightning-post-start (code=exited, status=0/SUCCESS)
   Main PID: 1558825 (lightningd)
         IP: 392.9M in, 73.9M out
         IO: 744.1M read, 793.5M written
      Tasks: 55 (limit: 9527)
     Memory: 465.4M
        CPU: 6min 9.548s
     CGroup: /system.slice/clightning.service
             ├─1558825 /nix/store/kw76ywkn387rz9izpnvcc7l5l4yyyrm3-clightning-23.02.2/bin/lightningd --lightning-dir=/var/lib/clightning