Skip to content

Logbook 2023 H2

Jump to bottom
Jean-Philippe Raynaud edited this page Jun 28, 2024 · 5 revisions

Newer Entries

Older Entries

December 2023

2023-12-21

2023-12-20

2023-12-19

2023-12-18

2023-12-15

2023-12-14

2023-12-13

2023-12-12

2023-12-11

2023-12-08

2023-12-07

2023-12-06

2023-12-05

2023-12-04

  • Today, we have worked on the following issues:

  • During our team session, we discussed about:

    • Threat modeling for P2P network
    • How to handle smoothly breaking changes with users of our distributions (Mithril client):
      • We could have 2 different OpenAPI files (one for signer and another for aggregator)
      • We will try to implement Consumer-driven contract testing strategy in a PoC with Hydra team
      • Each of our consumer will advertise the versions that they are using in (pre)release
      • During deployment steps of our CI/CD, we will check if there is any of our consumers that has a breaking change
      • This will help us handle deployment of breaking changes with less friction (especially as the number of projects using the client is growing)
    • Implementation of the Sanchonet test network
    • Implementation of the Light Client PoC with the Merkle tree we will likely use and which data we will sign (UTxO or more likely Transactions)
    • Size/performance concerns for storing transaction/UTxO sets locally on signer/aggregator nodes

2023-12-01

November 2023

2023-11-30

  • Today, we have worked on the issues:

  • We have also prepared the sprint planning for the next iteration

  • Finally, we have worked on the demo path:

    1. Update on Mithril Mainnet Beta
    2. What's new in distribution 2347
    3. Focus on newly released Mithril client library
    4. Focus on improvements of Mithril/Cardano nodes communication by TxPipe
    5. Showcase live coding of a CLI with the Mithril client library
    6. Showcase of Mithril aggregator performance improvements
    7. Showcase of Mithril relay P2P network

2023-11-29

2023-11-28

2023-11-27

2023-11-24

2023-11-23

2023-11-22

2023-11-21

2023-11-20

2023-11-17

2023-11-16

  • Today we have worked on the issues:

  • We have also worked on the sprint planning for the next iteration

  • Additionally, we have made some preparatory work for the creation of the next distribution: the PR Upgrade dependencies #1351 that will be merged shortly

  • During our team session, we have reviewed the work on the P2P network and we have decided to:

    • To merge the prototype as soon as it is running properly in the CI and without deploying the relay yet on any network
    • Create a separate PR for the activation of the relays in the infrastructure and the deployment pipeline
    • Release the P2P relay on the to be activated testing-sanchonet network
    • Work on a threat model and security risk assertion for the P2P networking layer

2023-11-15

2023-11-14

2023-11-13

2023-11-10

2023-11-09

2023-11-08

2023-11-07

2023-11-06

  • We have worked on the following issues:

  • We have also fixed a crash that was occurring at startup on the devnet, and the PR Fix devnet crashing at startup #1329 has been created, reviewed and merged.

  • Finally, during our team session, we have discussed about the P2P network PoC:

    • The goal is to have 2 aggregators producing certificates based on the signatures sent by the signers (via the relay)
    • We could try to implement also the signer registrations on a different topic of the pubsub
    • This could lead to complications, especially as the signer retrieves the list of other signers from the network
    • We have also talked about the protocol parameters which we consider to be network parameters (and that we will probably hard code or put in configuration files)
    • In terms of peer discovery, we will probably rely on the list of relays (available via cardano-cli and/or ledger state query), and target a specific port number on the host
    • We will keep working on it in a new team session this week

2023-11-02

  • Today, we have been working on the preparation of the demo with the following path:
    1. Presentation of the mainnet beta summary of signers registered into the protocol
    2. Presentation of the new 2342.0 distribution
    3. Presentation of the Mithril relay security advisory
    4. Presentation of the improvements on Docker images size
    5. Showcase of the UI improvements of the Mithril explorer
    6. Presentation and showcase of the Mithril relay P2P network PoC
# Demo: P2P network relay

---

# Setup demo

## Download source (if needed)
git clone https://github.com/input-output-hk/mithril.git
#or
git clone [email protected]:input-output-hk/mithril.git

## Checkout correct branch
cd mithril/
git switch ensemble/1300-mithril-relay-p2p-poc
git checkout 11a85d7ceb
cargo build --release -p mithril-relay
cp target/release/mithril-relay ../
cd ..
./mithril-relay

---
# Demo (port are dynamically allocated and must be updated during the demo)

## Create a first relay
./mithril-relay --node-type=relay

## Create 2 clients which connect in P2P to the first relay
./mithril-relay --node-type=client --dial-to="/ip4/127.0.0.1/tcp/44719"
./mithril-relay --node-type=client --dial-to="/ip4/127.0.0.1/tcp/44719"

## Send a dignature to the first relay (all clients should receive the message via P2P pubsub)
curl --data @signature-1.json -H "Content-type: application/json" -vvv http://127.0.0.1:43259/register-signatures

## Create a second relay which connect to a peer of the P2P network
./mithril-relay --node-type=relay --dial-to="/ip4/127.0.0.1/tcp/44719"

## Create 2 clients which connect in P2P to the second relay
./mithril-relay --node-type=client --dial-to="/ip4/127.0.0.1/tcp/46663"
./mithril-relay --node-type=client --dial-to="/ip4/127.0.0.1/tcp/46663"

## Send a signature to the first relay
curl --data @signature-2.json -H "Content-type: application/json" -vvv http://127.0.0.1:43259/register-signatures

## Send a signature to the second relay
curl --data @signature-3.json -H "Content-type: application/json" -vvv http://127.0.0.1:35849/register-signatures

## Kill some peers on the network
### Kill 2 client processes

## Send a signature to the first relay
curl --data @signature-4.json -H "Content-type: application/json" -vvv http://127.0.0.1:43259/register-signatures

  • We have also prepared the next iteration

  • Additionally, we have verified the KES keys of the test SPOs, and none of them needed to be rotated

October 2023

2023-10-31

2023-10-30

2023-10-27

2023-10-26

2023-10-25

2023-10-21

2023-10-20

2023-10-20

2023-10-19

2023-10-18

2023-10-17

2023-10-16

2023-10-13

2023-10-12

  • Today we have worked on the following issues:

  • We have also merged the PR Fix mithril-stm WASM build #1290 that fixes the compilation problem of the STM library in WASM

  • Finally, during our team session, we have mainly discussed about the P2P network for Mithril. We have investigated a draft of a possible design for introducing P2P networking and the possible technologies that we could use to support this design. We have decided to start by implementing a PoC for a Mithril relay:

    • Time box the implementation of a first prototype in Rust with libp2p
    • The idea is to get acquainted with the library and the constraints of operating a P2P network for Mithril
    • We intend to get a working demo, and also to point out the important questions/difficulties we will need to sort out to release it on the mainnet
    • Some of these questions are:
      • Do we need to authenticate messages sent on the network (with KES signatures e.g.)?
      • How do we have security and reliability for the network?
      • How do we handle the peers discovery (self discovery or by using a symmetric approach to Cardano relays)?
    • Here is a draft of the plan we have in mind:
      1. Setup a Mithril relay in Rust that acts as a proxy (replaces the current Squid implementation)
      2. Make the Mithril relay catch signatures sent from the signers and publish them on a P2P network
      3. Add a consumer of the signatures in the aggregator
      4. Assess a problems/difficulties (identified during the implementation phase)
      5. Conduct a risk analysis and get feedback from networking experts
      6. Prepare a testnet deployment with testing SPOs and with multiple aggregators producing certificates
      7. Work on other features: client and aggregator use peer discovery to select an aggregator, ...
    • We will create an issue and start working on it next week

2023-10-11

2023-10-10

2023-10-09

2023-10-05

2023-10-04

2023-10-03

2023-10-02

  • Today, we have worked on the following issues:

    • Add SPO tickers in explorer #1185: Changes have been published to the branch ensemble/1185/add-spo-tickers-in-explorer, and a new PR will be created shortly
    • Spike: Run client in browser WASM PoC #1254: A new branch has been created that cleans up the previous work ensemble/1254-poc-wasm-client, it adds display in the browser of certificate chain verification steps. We will keep working on it tomorrow
    • [Mithril Light Wallet Feature Draft]: the redaction of the issue is in progress, and will be completed shortly. Once completed, a discussion will be created.
    • [Benchmark aggregator performances]: some benchmarks have been run following the implementation of a temporary fix for ProtocolKeyRegistration creates performance bottleneck in aggregator#1187. The ingestion throughput of single signatures is much better and comparable to the ingestion throughput of signer registrations, However, the aggregator stress tests fail at the same client concurrency levels, which seem to indicate a problem with locks on the database connection. We will investigate further this bottleneck and create a new issue if needed

  • During our team session, we have talked about the foloowing subjects:

    • How to compute UTxO set for mainnet and verify it when the --whole-utxo option of the query utxo command fails on the mainnet: we can either work with the dbsync version to retrieve the full set, or work with some sampling and verification based on the query utxo on specific address which seems to work
    • Mithril relay default configuration: we will create a security advisory following the fix that updates the recommended listening port of the relay
    • Shallow node: we have discussed about implementing a shallow version of the Cardano node. The principle is that it would be bootstrapped by a mithril client. It would be called (via socket) by the Cardano node (forked version) when missing blocks are detected and would provide these certified information back to the Cardano node
    • UTxO certification for light wallet use case: we have talked about the current design proposition that will be released shortly. Also we have discussed about the current difficulty of signing data that is not finalized on the Cardano chain. An idea that we had (and that we will attempt to investigate further with researchers) is that the data signed by a Mithril quorum under full security protocol parameters should be sufficient to reduce the number of blocks needed to make sure the chain is final.

September 2023

2023-09-29

  • Today we have paired on the following issues:

    • Spike: Run client in browser WASM PoC #1254: we have recreated a PoC in a different branch in order to use a more recent template for the WASM client (we were having compilation and run issues with previous one). A new branch ensemble/1254/poc_wasm_client has been created in that manner. We have achieved the validation of a full certificate chain from one certificate up to the genesis certificate 🎉 (relying on the browser to retrieve the certificates form the aggregator, and with validation of the cryptographic part only of the certificate). We will polish the PoC next week in order to make it usable in a demo
    • Add SPO tickers in explorer #1185: Development are in progress, and a branch has been created ensemble/1185/add-spo-tickers-in-explorer. A PR will be created next week

  • Also, we have checked if some KES keys needed to be rotated, but this was not the case on any Mithril network

2023-09-28

Image

2023-09-27

2023-09-26

2023-09-25

2023-09-22

2023-09-21

2023-09-20

2023-09-19

2023-09-18

2023-09-11

2023-09-08

2023-09-07

# Demo: Mithril Clients and Snapshots Enhancements

---

# Setup demo

## Download Previous Mithril Client
wget https://github.com/input-output-hk/mithril/releases/download/2331.1/mithril-2331.1-linux-x64.tar.gz
tar -xzf mithril-2331.1-linux-x64.tar.gz mithril-client
rm -f mithril-2331.1-linux-x64.tar.gz
chmod u+x mithril-client
mv mithril-client mithril-client-2331
./mithril-client-2331 --version

## Download source
git clone https://github.com/input-output-hk/mithril.git
#or
git clone [email protected]:input-output-hk/mithril.git

## Build new version from source
cd mithril/
cargo build --release -p mithril-client
cp target/release/mithril-client ../
cd ..
./mithril-client --version

---

# Demo

## Watch 'data' directory
watch -n 2 -c "ls -al ./data"

## Watch 'data/db' directory
watch -n 2 -c "du -h ./data/db"

### Connect to release-preprod
export NETWORK=preprod
export AGGREGATOR_ENDPOINT=https://aggregator.release-preprod.api.mithril.network/aggregator
export GENESIS_VERIFICATION_KEY=$(wget -q -O - https://raw.githubusercontent.com/input-output-hk/mithril/main/mithril-infra/configuration/release-preprod/genesis.vkey)
export SNAPSHOT_DIGEST=$(curl -sL $AGGREGATOR_ENDPOINT/artifact/snapshots | jq -r '.[0].digest')

## Previous version need to identify latest snapshot first, and then download it
./mithril-client-2331 snapshot list
echo $SNAPSHOT_DIGEST
rm -rf ./data && time ./mithril-client-2331 snapshot download $SNAPSHOT_DIGEST --download-dir=./data
rm -rf ./data && time ./mithril-client-2331 snapshot download latest --download-dir=./data

## New version has the 'latest' snapshot digest magic value
./mithril-client snapshot show latest
rm -rf ./data && time ./mithril-client snapshot download latest --download-dir=./data

## New version has machine readable progress information
rm -rf ./data && ./mithril-client snapshot download latest --json --download-dir=./data

## Previous version does not stream download/unpacking
rm -rf ./data && time ./mithril-client-2331 snapshot download $SNAPSHOT_DIGEST --download-dir=./data

## New version stream download/unpacking
rm -rf ./data && time ./mithril-client snapshot download latest --download-dir=./data

## Connect to pre-release-preview
export NETWORK=preview
export AGGREGATOR_ENDPOINT=https://aggregator.pre-release-preview.api.mithril.network/aggregator
export GENESIS_VERIFICATION_KEY=$(wget -q -O - https://raw.githubusercontent.com/input-output-hk/mithril/main/mithril-infra/configuration/pre-release-preview/genesis.vkey)

## Show latest snapshot information in the new version
./mithril-client snapshot show latest

## Connect to testing-preview
export NETWORK=preview
export AGGREGATOR_ENDPOINT=https://aggregator.testing-preview.api.mithril.network/aggregator
export GENESIS_VERIFICATION_KEY=$(wget -q -O - https://raw.githubusercontent.com/input-output-hk/mithril/main/mithril-infra/configuration/testing-preview/genesis.vkey)

## Show latest snapshot information in the new version, and unpack with zstandard
./mithril-client snapshot show latest
rm -rf ./data && time ./mithril-client snapshot download latest --download-dir=./data
  • Finally, we have worked on the sprint planning for the next iteration

2023-09-06

2023-09-05

2023-09-04

2023-09-01

August 2023

2023-08-31

2023-08-30

2023-08-29

2023-08-28

2023-08-25

  • Today we have worked on the following issues:
    • Use zstandard compression for snapshot archives#876: More work on the POC has been made, we explored ZStandard dictionary & multithreading (only the later seems to have value in our case) + we also added a "stream" scenario in order to keep the capacity to report the download/unpack progress.

The POC findings are summarized in the following file: https://github.com/dlachaumepalo/download-extract-poc/blob/main/perf.md

2023-08-24

2023-08-23

2023-08-22

Github actions are back to normal.

2023-08-21

Today was marked by massive slowdown of the upload-artifact and download-artifact that we use a lot in our CI pipelines, making our CI barely usable (ie: a workflow that usually take 5 minutes is at 7 hours of running time right as I'm writting those lines). The problem is github side, as other users have reported the same issue, so we can do nothing but wait.

  • Today we have worked on the following issues:

Also a PR [#1165] have been submitted by Yvan Sraka that should stabilize the hydra CI build (those have been flaky for some weeks). Many thanks to him for his time.

2023-08-18

  • Today we have worked on the following issues:
    • Refactor (de)serialization of crypto entities #668:
      • A PR for the verification key signature #1164 have been submitted, reviewed and merged.
      • A regression was identified on testing-preview after merging #1164, a fix have been made with #1166 but we need to monitor the environment to see if this indeed solve the issue.
      • Advancement has also be made for two other types (GenesisSecretKey and GenesisVerificationKey), a PR should be submitted next Monday. With those two types we will be at 8/13 types converted.
    • Simplify logs on the Signer/Aggregator #1106: The main work is finished, the related PR #1165 is awaiting review.

Also we merged a PR for the routine task of upgrading our dependencies (#1163).

2023-08-17

We also split the logbook, extracting the second half of 2022 into a new page, and extracting the first half of 2023 into a new page too.

2023-08-16

2023-08-11

2023-08-10

2023-08-09

2023-08-08

2023-08-07

2023-08-04

2023-08-03

2023-08-02

2023-08-01

July 2023

2023-07-31

2023-07-28

2023-07-27

2023-07-26

2023-07-25

2023-07-24

2023-07-21 🚀

  • Today, we have officially launched the Mithril Protocol’s Mainnet Beta 💪 🎉. The aggregator of the release-mainnet network is up and running and runs the newly releasedd 2329.0 distribution. The first step is to get signer registrations before we can genesis its certificate chain. This is likely to occur as early next week. We have defined an expected timeline for the network to start producing certificates (as explained in this dev blog post):

mainnet-launch-timeline

  • We have talked about some scenario that could happen on the mainnet, such as having signers registered but not signing 2 epochs later. Depending on the protocol parameters and their shares in the total stakes registered, this could prevent the network to sign properly (specially when the number of signers registered is low). We all agreed that this must be considered as an edge case that is part of the risk of running the protocol on the mainnet.However, if this occurs, we will probably have to re-genesis the chain and/or update protocol parameters. This lead us to decide that we will work on the redaction of specific maintenance procedure that will be compiled in a Runbook. Also, we will work on designing a mechanism that facilitates the retirement of a signer from the network in a fashion close to what is done on the Cardano network

  • Some SPO have reported a bug for which the issue Shared library error in CI binaries #1073 has been created. It appears that the problem can be easily fixed by embedding the openssl in the binary builds (with a configuration in the Cargo.toml). We will implement this fix and probably create a fix for the 2329 distribution

  • Additionally, we have worked on the following PRs:

  • Finally, we have verified if the KES keys of the SPOs of the Mithril networks should be rotated: this was not needed this week

2023-07-20

2023-07-19

  • Today, we have made two pairing sessions on the issue Refactor (de)serialization of crypto entities #668. We have tried 2 different approache:

    • Using a custom serde attribute with to use a specific for serializing a ProtocolVerificationKey as verification_key instead of its serialized string version
    • And another that wraps the native STM StmVerificationKeyPoP in the ProtocolVerificationKey type that would be more than just an alias
    • Both approaches have advantages and drawbacks: we will keep working in an exploratory mode to find the optimal approach that will be then implemented on the other concerned types
    • We have pushed our work on this ensemble/668/refacto_crypto_entities branch

  • We have also created the pre-release for the new distribution 2329.0-pre. It has been successfully deployed on the pre-release-preview network. This is also the release candidate for the first Mithril distribution that runs on Cardano mainnet that is scheduled to be launched in beta by end of week

  • Finally, we have worked on the following PRs:

2023-07-18

2023-07-17

  • We have spent most of the day pairing on the issue Design & implement basic stress test tool for aggregator#991:

    • We have consolidated the foundations of the stress tester in a separated binary directory
    • We have implemented custom arguments (with working default values)
    • The scenario execution is now cleaner and displays logs and loaders during execution
    • We have worked on a general wait function for calls on aggregator REST API as it a recurrent need in the scenario
    • We have discussed about how to compute the stake distribution for the generated signers: we have agreed that during the signer key registration preparation step, we will also generate the associated stake distribution in the expected JSON format. This stake distribution will be used as is by the mock cardano cli
    • We have created the draft PR Aggregator Stress Test #1062

  • We have also worked on:

    • Fixing new clippy warnings with the release of Rust 1.71.0 with the PR Upgrade Rust 1.71.0 #1060 which has been closed and merged
    • Preparing the mainnet infrastructure in the new GCP project. We will resume once we have enough rights to givemandatory roles to the service account

2023-07-13

2023-07-12

2023-07-11

2023-07-10

2023-07-06

2023-07-05

2023-07-04

2023-07-03

  • We have worked on the following issues:
    • testing-preview network does not create certificates #1015: after investigation, it appears that this problem that occurs sporadically is due to the current implementation of the verification key storer. We are working on a fix and a PR should be created tomorrow
    • Add infrastructure monitoring #987: the prometheus endpoint is now able to scrape metrics from Cardano nodes. We are currently implementing logs gathering with promtail and loki. The draft PR has been updated and it should be ready to be reviewed tomorrow
    • Remove certificate hash from Artifact #932: a PR has been created, but we need to do some refactoring on the message adapters to avoid using Serialize/DeSerialize traits when it is not mandatory. It should be ready for review tomorrow
    • Deploy mainnet infrastructure #988: we have started to prepare the new production project on GCP for the mainnet aggregator
    • We have also started preparing the next distribution: we have made a test on a dev network to make sure that the re-computation of the hashes of the certificate chain is working properly on real networks data. This is the case: this will avoid to re-genesis the certificate chain. We will continue preparing the distribution tomorrow
Clone this wiki locally