docs: update troubleshooting & faq

docs/deploy-hyperlane-troubleshooting.mdx

@@ -1,6 +1,8 @@
-# How to Troubleshoot Your Hyperlane Deployment
+# Hyperlane Troubleshooting Guide
 
-## Logging
+## General Troubleshooting
+
+### Logging
 
 Logging levels and formats can be configured for Hyperlane tooling using the following two environment variables:
 
@@ -9,14 +11,17 @@ Logging levels and formats can be configured for Hyperlane tooling using the fol
 
 The [Hyperlane CLI](/docs/reference/cli.mdx) also allows configuring logging via the `--log` and `--verbosity` flags.
 
-## Chain configuration
+### Chain Configuration
 
 Within your working directory, you may find a `chains/` yaml files organized by chain name. These `metadata.yaml` files describe the information needed to use the chain in Hyperlane deployments and apps.
 
 You can define a full configuration for any new chain in this file. The metadata that can be configured is defined in this [example configuration](https://github.com/hyperlane-xyz/hyperlane-monorepo/blob/main/typescript/cli/examples/chain-config.yaml). You can also find the chain metadata schema at [chainMetadataTypes.ts](https://github.com/hyperlane-xyz/hyperlane-monorepo/blob/main/typescript/sdk/src/metadata/chainMetadataTypes.ts).
 
 Here's an example configuration for two local anvil chains:
 
+<details>
+<summary>Example: Two Local Anvil Chains</summary>
+
 ```yaml
 anvil1:
   chainId: 31337
@@ -38,6 +43,8 @@ anvil2:
     - http: http://localhost:8555
 ```
 
+</details>
+
 You can also extend a core chain config by providing the fields to be overridden:
 
 ```yaml
@@ -46,11 +53,12 @@ sepolia:
     confirmations: 2
 ```
 
-### Override RPC URLs
+#### Override RPC URLs
 
 You can override the RPC urls by extending the core chain config.
 
-In the example below, you can see how to define an array of RPCs and also adjust the retry parameters for any of them.
+<details>
+<summary>Example: Define RPC URLs array and adjust retry settings</summary>
 
 ```yaml
 demochain:
@@ -68,7 +76,9 @@ sepolia:
         maxRequests: 10
 ```
 
-### Override transaction settings
+</details>
+
+#### Override transaction settings
 
 Transaction overrides are any properties to include when forming transaction requests. For example:
 
@@ -96,7 +106,75 @@ If you are overriding the nonce in the chain configuration, ensure you are updat
 
 :::
 
-## `eth_getStorageAt()` Compatibility
+### Message Delivery Issues
+
+#### Long Message Delivery Times or Timeouts
+
+Delays or timeouts in message delivery often result from RPC issues, such as overloaded endpoints, or from chains that produce blocks only on demand.
+
+To identify the issue, check for logs indicating RPC-related problems such as: `Deprioritizing an inner provider in FallbackProvider`
+
+To address these delays:
+
+- **Check RPC Health:** Ensure RPC endpoints are responsive.
+- **Configure Chains with On-Demand Blocks:** For these chains, set the reorgPeriod to 0. This ensures that agents always look at the tip of the chain, avoiding delays caused by block finalization processes.
+
+## Agent Troubleshooting
+
+### Validator Troubleshooting
+
+- **`Validator has not announced any storage locations` Warning:**
+
+  - **Reason**: This occurs when the relayer cannot retrieve validator signatures required to process a message.
+  - **Solution**:
+    - Ensure the validators have announced their storage locations. Check validator logs for a message such as:
+      `Validator has announced signature storage location, locations: ["file:///tmp/hyperlane-validator-signatures-local"]`
+    - Verify that each validator has a unique signature storage path (`--checkpointSyncer.path`) to prevent overwriting.
+    - Confirm that the relayer has read access to the storage paths.
+
+- **`No ISM Found for Origin` Error**
+
+  - **Reason**: This occurs when the Interchain Security Module (ISM) does not recognize the origin chain.
+  - **Solution**:
+    - Ensure the ISM configuration includes the Validators for the origin chain.
+    - If it doesn't, add validators for the origin chain to your ISM.
+    - Restart the relayer after updating the ISM configuration.
+
+### Relayer Troubleshooting
+
+#### Checking Pending Messages in the Queue
+
+If messages are stuck in the relayer queue or not being processed, use the `list_operations` endpoint to inspect the relayer's queue:
+
+```shell
+curl http://0.0.0.0:9090/list_operations?destination_domain=<destination_domain_id>
+```
+
+This endpoint provides the status of messages in the queue and can help identify why they are not being delivered.
+
+#### Debugging with Logs
+
+For detailed insights, enable debug logging and monitor the relayer's activity: `HYP_LOG_LEVEL=debug`. Once logs are captured, you can use them to locate issues with specific message IDs.
+
+#### Retrying Messages
+
+After reviewing the logs, you can trigger an immediate retry of all pending messages using the `message_retry` endpoint:
+
+```shell
+curl --location 'http://127.0.0.1:9090/message_retry' \
+--header 'Content-Type: application/json' \
+--data '[{"messageid": "*", "origindomain": "*", "senderaddress": "*", "destinationdomain": "*", "recipientaddress": "*"}]'
+```
+
+Make sure that you use the most recent version of the relayer and capture all logs for debugging.
+
+#### Relayer Selecting Incorrect ISM
+
+Ensure the Interchain Security Module (ISM) on your chain is configured correctly. An incorrect ISM can result in messages not being processed. Verify the ISM address in your configuration and ensure it matches your deployment setup.
+
+## Advanced Troubleshooting
+
+### `eth_getStorageAt()` Compatibility
 
 Some chains do not natively support the [`eth_getStorageAt()`](https://ethereum.org/en/developers/docs/apis/json-rpc/#eth_getstorageat) API. If you're deploying on one of these chains and encounter issues, review the changes made to the Hyperlane codebase in this [commit](https://github.com/hyperlane-xyz/hyperlane-monorepo/commit/871df7a4dce203ff5cf23ae654d03743dc00ea61).
 

docs/faq.mdx

@@ -28,16 +28,7 @@ If you’re reading this FAQ, you’ve found the docs which is a great place to
 
 ## Technical Questions
 
-**How is Hyperlane secured?**  
-Hyperlane is secured by its modular security stack featuring Interchain Security Modules (ISMs). Developers can configure various pre-built ISMs, compose them with each other, or even create custom ISMs based on their application's needs.
-
-A modular approach to security ensures that Hyperlane will continue to stay up to the latest industry advances in security models.
-
-**Who are the Hyperlane validators?**  
-Hyperlane is secured with a modular security stack featuring ISMs. There is no protocol-enshrined security model, let alone validator set. That said, most Hyperlane deployments are configured with a Default ISM, which specifies the security model to use if the message recipient has not specified an ISM override.
-
-**Where is Hyperlane deployed?**  
-A list of known deployments can be found on the [contract addresses](./reference/addresses/mailbox-addresses.mdx) page.
+### Messaging
 
 **What happens when I send a message on Hyperlane?**  
 See the [send](./reference/messaging/send.mdx) and [receive](./reference/messaging/receive.mdx) pages for more details. In summary:
@@ -61,6 +52,32 @@ For chat use cases, consider [XMTP](https://xmtp.org/), [Push](https://push.org/
 **Is Hyperlane a token bridge?**  
 Not exactly. Hyperlane is a general message passing (GMP) protocol that allows communication between blockchains. Token bridges are just one of many types of applications that can be built on top of Hyperlane!
 
+### Deployment
+
+**Where is Hyperlane deployed?**  
+A list of known deployments can be found on the [contract addresses](./reference/addresses/mailbox-addresses.mdx) page.
+
+### Advanced Topics
+
+**How is Hyperlane secured?**  
+Hyperlane is secured by its modular security stack featuring Interchain Security Modules (ISMs). Developers can configure various pre-built ISMs, compose them with each other, or even create custom ISMs based on their application's needs.
+A modular approach to security ensures that Hyperlane will continue to stay up to the latest industry advances in security models.
+
+**How many validators does Hyperlane have, and who operates them?**  
+The number of validators depend on the Interchain Security Module (ISM) configuration. The Default ISM uses a specific validator set, and you can view the details such as threshold, operator and address [here](./reference/default-ism-validators).
+
+**How can we ensure message integrity?**  
+Hyperlane's message security is driven by the Interchain Security Module (ISM) configuration. By default, messages rely on the Default ISM's validator set, which requires a quorum of validators to sign for security. In the unlikely event of a validator compromise, the security of messages could be impacted. However, developers have the flexibility to configure their own ISM, enabling them to customize and strengthen the security model to suit their application's needs.
+
+**Does the source chain receive confirmation whether the transaction was executed on the destination chain?**  
+Currently, the source chain does not automatically receive a confirmation of execution from the destination chain. However, this functionality can be added by customizing the `handle` function on the destination chain. For more information, refer to the [handle function](./reference/messaging/receive#handle).
+
+**Why are outbound message IDs stored in the Merkle tree?**  
+Outbound message IDs are stored in the Merkle tree to efficiently track and verify messages. This mechanism ensures validators can accurately detect and sign new messages as they are created.
+
+**Why do validators call `MerkleTreeHook.latestCheckpoint()`?**  
+Validators use the [`MerkleTreeHook.latestCheckpoint()`](https://github.com/hyperlane-xyz/hyperlane-monorepo/blob/5e9b8ba7ab0f54e92b3cf32806fae95eb23ad0f8/solidity/contracts/hooks/MerkleTreeHook.sol#L49) function to determine when new transactions need to be indexed. This polling mechanism ensures validators can immediately start signing new messages without the need to backfill the entire tree.
+
 ## Community & Contributions
 
 **How can I join the Hyperlane community?**  
@@ -71,3 +88,6 @@ Check out our open roles [here](https://jobs.lever.co/Hyperlane).
 
 **How can I contribute to improve this documentation?**  
 You can make a PR to edit this documentation directly via the [docs repo](https://github.com/hyperlane-xyz/v3-docs).
+
+**Where can I get support or ask questions about Hyperlane?**  
+You can get support and ask questions on our [Hyperlane Discord](https://discord.com/invite/hyperlane), where the community and team are active and ready to assist.

docs/protocol/agents/validators.mdx

@@ -22,6 +22,8 @@ The Validator agent uses two key types: one for signing transactions on the bloc
 
 For messages using a [Multisig ISM](../../reference/ISM/multisig-ISM-interface.mdx), validators read the current merkle root by calling `MerkleTreeHook.latestCheckpoint()`.
 
+Validators use the `MerkleTreeHook.latestCheckpoint() function to determine when new transactions need to be indexed. This polling mechanism ensures validators can start signing new messages without the need to backfill the entire tree.
+
 ### Multisig ISM Prerequisites
 
 To validate messages from an origin chain, the Validator's checkpoint signing key must be registered in the [Multisig ISM](../../reference/ISM/multisig-ISM-interface.mdx) specified by the destination chain recipient. This can only be done when the ISM contract is deployed, but this is easily done by sending a `deploy(validatorAddresses, threshold)` transaction on any implementation of [StaticThresholdAddressSetFactory](https://github.com/hyperlane-xyz/hyperlane-monorepo/blob/be4617b18ba638e0ef5a5326614bc4f8c58d25f9/solidity/contracts/libs/StaticAddressSetFactory.sol#L10).
@@ -60,3 +62,7 @@ The only checkpoint storage that is currently supported by the agent implementat
 ### Liveness implications
 
 Depending on the threshold configured in the Multisig ISM, Validator downtime can halt liveness of the protocol. The submitter prioritizes new checkpoints over old ones, so that the Relayer can fetch metadata for newer messages first, which are more likely to not have been delivered yet.
+
+### Default ISM Validators
+
+The number of validators depend on the Interchain Security Module (ISM) configuration. The Default ISM uses a specific validator set, you can view the Validator details such as threshold, operator and addresses [here](../../reference/default-ism-validators).

sidebars.js

@@ -25,6 +25,8 @@ const sidebars = {
     {
       type: "category",
       label: "Quickstart",
+      collapsed: false,
+      collapsible: true,
       items: [
         {
           type: "doc",
@@ -39,7 +41,7 @@ const sidebars = {
         {
           type: "doc",
           id: "deploy-hyperlane-troubleshooting",
-          label: "Troubleshoot",
+          label: "Troubleshooting",
         },
       ],
     },