Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update tdex to v1.0.0 #754

Merged
merged 18 commits into from
Sep 18, 2024
Merged

Update tdex to v1.0.0 #754

merged 18 commits into from
Sep 18, 2024

Conversation

altafan
Copy link
Contributor

@altafan altafan commented Aug 28, 2023

This updates TDEX from v0.9.1 to v1.0.0.

The compose file defines a brand new oceand service required by the new version of tdexd v1.0.0. Updated also the dashboard to v1.0.1.

* Update docker-compose

* Update images

* Fixes

* Expose dashboard port

* Fixes

* Update dashboard image

* Update dashboard image

* Get ready for being merged

* Get ready for the merge
Copy link
Contributor

@nmfretz nmfretz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this update @altafan! I have left some suggestions below for you to address before I test.

Also, are you okay if I push a commit directly to your update-tdex branch? We need to add a pre-start hook to make sure that users who already have tdex installed receive the new ocean-data directory with appropriate ownership when they update.

tdex/docker-compose.yml Outdated Show resolved Hide resolved
tdex/docker-compose.yml Outdated Show resolved Hide resolved
tdex/docker-compose.yml Outdated Show resolved Hide resolved
tdex/docker-compose.yml Outdated Show resolved Hide resolved
tdex/hooks/pre-start Show resolved Hide resolved
Comment on lines +32 to +33
ports:
- ${APP_TDEX_PORT}:${APP_TDEX_PORT}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we don't need to map ports here, as Docker's internal DNS resolution will allow the containers in this compose file to connect to each other. Unless you are trying to expose this port on the host so that it is reachable externally?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah the idea is to make this service accessible from the outside world.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah okay, so in that case we need to change ${APP_TDEX_PORT} to something other than 9090 because this port has been taken by Urbit Bitcoin Connector, meaning if a user is running both TDEX and Urbit Bitcoin Connector there will be a port clash on the host and one of the apps will break:

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it shouldn't be a problem at all, do you suggest any port number?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what about 9000?

Copy link
Contributor

@ParthJadhav ParthJadhav Sep 25, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @altafan , port 9000 is currently occupied by portainer. To avoid conflicts, please consider using 9002, which is the closest available option. Before selecting a port, it's a good practice to search within this repository to ensure it doesn't overlap with existing port assignments.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm sorry! I just realized I changed this for the ease of testing and it wasn't meant to be a committed change. I reverted to port 9090 🙏

@altafan
Copy link
Contributor Author

altafan commented Sep 14, 2023

Thanks for this update @altafan! I have left some suggestions below for you to address before I test.

Also, are you okay if I push a commit directly to your update-tdex branch? We need to add a pre-start hook to make sure that users who already have tdex installed receive the new ocean-data directory with appropriate ownership when they update.

Thank you @nmfretz for the review!

I guess I have to give you access to my repo in order to directly push changes? Can you wait until all required changes are fixed?

@nmfretz
Copy link
Contributor

nmfretz commented Sep 14, 2023

I guess I have to give you access to my repo in order to directly push changes? Can you wait until all required changes are fixed?

Actually, here's what I think will be needed. You can add this above the Tor Hidden Service logic in the existing pre-start.

set -euo pipefail

APP_DATA_DIR="$(readlink -f $(dirname "${BASH_SOURCE[0]}")/..)"
OCEAN_DATA_DIR="${APP_DATA_DIR}/ocean-data"

[ ! -d "${OCEAN_DATA_DIR}" ] && mkdir -p "${OCEAN_DATA_DIR}" && chown 1000:1000 "${OCEAN_DATA_DIR}"

@nmfretz
Copy link
Contributor

nmfretz commented Sep 26, 2023

Hey @altafan can you please update the exposed daemon port to 9002: #754 (comment)

Also, what would be the best way for us to test external connection in the tdex ecosystem?

@altafan
Copy link
Contributor Author

altafan commented Sep 27, 2023

Also, what would be the best way for us to test external connection in the tdex ecosystem?

@nmfretz I made a test on a remote host, and once I installed the tdex app, I just went to <ip>:9090 to see a page similar to the following one (the url might be different):

@tiero
Copy link
Contributor

tiero commented Oct 4, 2023

Any news? @nmfretz

@nmfretz
Copy link
Contributor

nmfretz commented Oct 6, 2023

oh sorry, @tiero and @altafan can you guys push the quick fix noted here: #754 (comment). Port 9090 clashes with an existing app. 9092 is free for example.

Also, since this is an update, we can now add release notes to the umbrel-app.yml file. See Lightning app as an example:

releaseNotes: >-
This release is a minor release to fix a memory leak inadvertently introduced by optimizations to the mempool scanning logic.
Read the full release notes for additional information and detailed changes at https://github.com/lightningnetwork/lnd/blob/master/docs/release-notes/release-notes-0.16.4.md

@altafan
Copy link
Contributor Author

altafan commented Oct 10, 2023

@nmfretz i pushed the changes. Now the port is 9092. I also followed your suggestion and added some release notes, thanks 🙏.

@nmfretz
Copy link
Contributor

nmfretz commented Oct 10, 2023

Thanks @altafan. I have made a few changes and we are nearly there:

  • in your recent commit 19e9840, you changed the port in the manifest to 9092. This port is actually for your dashboard UI, so should stay as 9094. I have changed that here a5bac52
  • updated the Caddyfile so that new installs start with the correct tdexd port (9092) 2f61879
  • added logic to the pre-start hook so that existing installs that are updating to this new version receive the change to the caddyfile f566598

I tested on a new install and it works well 👌. However, something breaks in the tdexd container when updating the app from the previous version. Here is the error I get from the tdexd container

$ sudo docker logs tdex_tdexd_1
INFO[0000] All 0 tables opened in 0s
INFO[0000] Discard stats nextEmptySlot: 0
INFO[0000] Set nextTxnTs to 0
INFO[0000] Deleting empty file: /home/tdex/.tdex-daemon/db/feeder/000007.vlog
INFO[0000] All 0 tables opened in 0s
INFO[0000] Discard stats nextEmptySlot: 0
INFO[0000] Set nextTxnTs to 0
INFO[0000] Deleting empty file: /home/tdex/.tdex-daemon/db/markets/000007.vlog
FATA[0000] failed to initialize grpc service             error="invalid opts: invalid app config: opening prices db: manifest has unsupported version: 7 (we support 8).\nPlease see https://github.com/dgraph-io/badger/blob/master/README.md#i-see-manifest-has-unsupported-version-x-we-support-y-error on how to fix this."

Could you please take a look into this? Once this is fixed we can go live!

@altafan
Copy link
Contributor Author

altafan commented Nov 15, 2023

@nmfretz sorry for the late reply

However, something breaks in the tdexd container when updating the app from the previous version. Here is the error I get from the tdexd container

The reason is that the database of the prev version of tdexd must be migrated in order to be compatible with the new version.
The migration tool is packed within the official docker image, do you think it's possible to setup a migration script that runs it if needed to solve the issue?

Otherwise, it could just be enough to delete the existing datadir, the daemon will create a new one at first startup and the user will restore the wallet, which seems an easier solution.

@altafan
Copy link
Contributor Author

altafan commented Nov 23, 2023

@nmfretz let's go for the easiest way and eventually remove the datadir if existing. Can you tell me if I can add this conditional operation to some hook script?

@tiero
Copy link
Contributor

tiero commented Dec 14, 2023

@smolgrrr any inputs here on what @altafan mentioned?

@nmfretz
Copy link
Contributor

nmfretz commented Jan 26, 2024

@tiero @altafan sorry for the delay.

Otherwise, it could just be enough to delete the existing datadir, the daemon will create a new one at first startup and the user will restore the wallet, which seems an easier solution.

For this option we'd run the risk of users clicking update without reading the release notes, so there could be a subset of users who update and don't have (or have lost) their recovery phrase, which would be pretty bad I think.

Additionally we'd have to make sure that this database deletion logic doesn't run on subsequent app starts/restarts/updates, which is totally possible but adds complexity.

The migration tool is packed within the official docker image, do you think it's possible to setup a migration script that runs it if needed to solve the issue?

Yes, we can do this with a pre-start script. The benefit here is that we users won't accidently lose access to their funds.

We could do something like this:

  • before starting the app, check if tdex-data dir has anything in it. If it doesn't then this is a new app install. We create a file in the data directory called something like HAS_MIGRATED and then exit the script and allow the app to start.
  • If tdex-data has data then check if the file called HAS_MIGRATED exists:
    • If it HAS_MIGRATED exists then we exit the script and allow the app to start normally. This will ensure that subsequent app restarts/updates don't nuke the database again.
    • If HAS_MIGRATED doesn't exist then we know that we need to migrate. We start the required container(s) for the migration tool and run the necessary commands to migrate. Then we create the HAS_MIGRATED file and exit the script so that the app starts up.

I can write this script on priority for you guys and test it out, I would just need guidance on the commands to run for migration. If I don't answer here, please feel free to DM me on X, Nostr, or Telegram!

@altafan
Copy link
Contributor Author

altafan commented Feb 13, 2024

@nmfretz the migration is an interactive process that requires the user to enter the password to unlock the data stored in the db. Can this be a problem for the workflow?

@tiero
Copy link
Contributor

tiero commented Jun 15, 2024

Guys can we merge this? People are still using the old deprecated version.

Let's override the old data directory with bash?

@tiero
Copy link
Contributor

tiero commented Jun 15, 2024

Would maybe make an additional App ie TDEX V1 and deprecate this one be better? So we assume user do not expect a migration being a full restore

@nmfretz
Copy link
Contributor

nmfretz commented Jun 18, 2024

Guys can we merge this? People are still using the old deprecated version.

Let's override the old data directory with bash?

Would maybe make an additional App ie TDEX V1 and deprecate this one be better? So we assume user do not expect a migration being a full restore

Hey @tiero. Alrighty, let's get this done. Without an automatic migration process on update, let's go with the following:

  1. A warning at the top of the release notes explaining to users that this update will require them to restore the wallet from their seed phrase.

  2. For existing installs we will keep the old data directory around so that in a worst-case scenario they can use this old database to restore using an older version of tdex. The way we can do this is just by simply changing the data directory bind mount on the host from say:

${APP_DATA_DIR}/tdex-data:/home/tdex/.tdex-daemon

to:

${APP_DATA_DIR}/tdexd:/home/tdex/.tdex-daemon

Let me make those changes and push them here so we can all test.

@nmfretz
Copy link
Contributor

nmfretz commented Jun 18, 2024

@tiero @altafan, here is the warning I have added to the top of the release notes. Does this look okay to you?

image

We could also include a link to your Telegram channel and say something like:

If you encounter any issues when recovering your wallet after update, please reach out to the official TDEX Telegram group: https://t.me/tdexnetwork

If any users reach out with issues after updating, their old tdex daemon data directory will be located at:
/home/umbrel/umbrel/app-data/tdex/tdex-data

The new tdex daemon data directory is located at:
/home/umbrel/umbrel/app-data/tdex/tdexd


Can you please take a look at this issue and see if you can recreate:

I just tested updating TDEX from v0.9.1 to v1.0.1. I went through the wallet recovery process and see this error in the UI:

image

I see this in the console:
image

But nothing in the networking tab, and nothing related in the container logs:

$ sudo docker logs -f tdex_tdexd_1
INFO[0000] All 0 tables opened in 0s
INFO[0000] Discard stats nextEmptySlot: 0
INFO[0000] Set nextTxnTs to 0
INFO[0000] All 0 tables opened in 0s
INFO[0000] Discard stats nextEmptySlot: 0
INFO[0000] Set nextTxnTs to 0
INFO[0000] All 0 tables opened in 0s
INFO[0000] Discard stats nextEmptySlot: 0
INFO[0000] Set nextTxnTs to 0
INFO[0000] All 0 tables opened in 0s
INFO[0000] Discard stats nextEmptySlot: 0
INFO[0000] Set nextTxnTs to 0
INFO[0000] All 0 tables opened in 0s
INFO[0000] Discard stats nextEmptySlot: 0
INFO[0000] Set nextTxnTs to 0
INFO[0000] starting daemon
INFO[0000] wallet interface is listening on :9092
DEBU[0008] /tdex_daemon.v2.WalletService/GetStatus
DEBU[0008] /tdex_daemon.v2.WalletService/GetInfo
DEBU[0016] /tdex_daemon.v2.WalletService/UnlockWallet
DEBU[0021] /tdex_daemon.v2.WalletService/GetStatus
DEBU[0021] /tdex_daemon.v2.WalletService/GetInfo
DEBU[0021] stopped server
INFO[0021] wallet interface is listening on :9092
INFO[0021] operator interface is listening on :9092
INFO[0021] trade interface is listening on :9092
DEBU[0044] /tdex_daemon.v2.WalletService/GetStatus
DEBU[0044] /tdex_daemon.v2.WalletService/GetInfo
DEBU[0063] /tdex_daemon.v2.WalletService/GetStatus
DEBU[0063] /tdex_daemon.v2.WalletService/GetInfo
DEBU[0600] Total allocated: 0.852GB, Heap allocated: 0.485GB, Allocated objects count: 1725134, Freed objects count: 1431569
DEBU[0600] Num of go routines: 102
$ sudo docker logs -f tdex_oceand_1
INFO[0000] All 1 tables opened in 0s
INFO[0000] Discard stats nextEmptySlot: 0
INFO[0000] Set nextTxnTs to 4
INFO[0000] All 0 tables opened in 0s
INFO[0000] Discard stats nextEmptySlot: 0
INFO[0000] Set nextTxnTs to 0
INFO[0000] All 0 tables opened in 0s
INFO[0000] Discard stats nextEmptySlot: 0
INFO[0000] Set nextTxnTs to 0
INFO[0000] service: registered wallet handler on public interface
INFO[0000] service: registered account handler on public interface
INFO[0000] service: registered transaction handler on public interface
INFO[0000] service: registered notification handler on public interface
DEBU[0000] scanner: start listening to messages from electrum server
INFO[0000] service: started blockchain scanner
INFO[0000] service: start listening on :18000
DEBU[0000] gRPC method: /ocean.v1.WalletService/Status
DEBU[0000] gRPC method: /ocean.v1.NotificationService/UtxosNotifications
DEBU[0000] gRPC method: /ocean.v1.NotificationService/TransactionNotifications
DEBU[0000] gRPC method: /ocean.v1.WalletService/GetInfo
DEBU[0009] gRPC method: /ocean.v1.WalletService/Status
DEBU[0009] gRPC method: /ocean.v1.WalletService/GetInfo
DEBU[0016] gRPC method: /ocean.v1.WalletService/Unlock
DEBU[0021] wallet repository: publish event WalletUnlocked
DEBU[0021] gRPC method: /ocean.v1.WalletService/GetInfo
DEBU[0021] gRPC method: /ocean.v1.WalletService/Status
DEBU[0022] gRPC method: /ocean.v1.WalletService/GetInfo
DEBU[0044] gRPC method: /ocean.v1.WalletService/Status
DEBU[0044] gRPC method: /ocean.v1.WalletService/GetInfo
DEBU[0064] gRPC method: /ocean.v1.WalletService/Status
DEBU[0064] gRPC method: /ocean.v1.WalletService/GetInfo
sudo docker logs -f tdex_dashboard_1
{"level":"info","ts":1718688563.9792867,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"info","ts":1718688563.9834294,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1718688563.9839225,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000764180"}
{"level":"info","ts":1718688563.9841962,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
{"level":"info","ts":1718688563.9845257,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1718688563.9845374,"msg":"serving initial configuration"}
{"level":"info","ts":1718688563.984608,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/data/caddy"}
{"level":"info","ts":1718688563.984663,"logger":"tls","msg":"finished cleaning storage units"}

On fresh install I see a different error:

image image

@nmfretz
Copy link
Contributor

nmfretz commented Jun 24, 2024

Hey @altafan @tiero just pinging again in case you missed this comment here: #754 (comment)

@tiero
Copy link
Contributor

tiero commented Jun 24, 2024

@altafan @Janaka-Steph Can you please double-check this when you got time? To test with latest version (without involving migration of db)

@nmfretz
Copy link
Contributor

nmfretz commented Aug 29, 2024

@altafan @Janaka-Steph @tiero, just pinging in case this has fallen off your radar.

@Janaka-Steph
Copy link

Janaka-Steph commented Sep 11, 2024

I just tried to run tdex 1.0.1 in multipass VM. I encountered the same error "verification failed: signature mismatch after caveat verification". But after refreshing the page I was able to access the Tdex provider and load the wallet.
Feel free to merge, it is a minor issue.

@nmfretz
Copy link
Contributor

nmfretz commented Sep 12, 2024

I just tried to run tdex 1.0.1 in multipass VM. I encountered the same error "verification failed: signature mismatch after caveat verification". But after refreshing the page I was able to access the Tdex provider and load the wallet.
Feel free to merge, it is a minor issue.

Thanks @Janaka-Steph. Did you also try a fresh install and wallet creation where I encounter this:
image

If you are comfortable moving forward with both issues then we can merge.

@nmfretz
Copy link
Contributor

nmfretz commented Sep 18, 2024

@Janaka-Steph @tiero pinging again for the above. Thanks.

@Janaka-Steph
Copy link

@nmfretz I just tried and I didn't get error. It should have been fixed. Ok for merge!

@nmfretz
Copy link
Contributor

nmfretz commented Sep 18, 2024

Thanks @Janaka-Steph! Clearing local storage has gotten rid of this issue for me FYI, in case anyone reaches out with a similar issue. Going live.

Copy link

🎉   Linting finished with no errors or warnings   🎉

Thank you for your submission! This is an automated linter that checks for common issues in pull requests to the Umbrel App Store.

Please review the linting results below and make any necessary changes to your submission.

Linting Results

Severity File Description
ℹ️ tdex/docker-compose.yml Mounted file/directory "/tdex/torrc" doesn't exist:
The volume "${APP_DATA_DIR}/torrc:/etc/tor/torrc:ro" tries to mount the file/directory "/tdex/torrc", but it is not present. This can lead to permission errors!
ℹ️ tdex/docker-compose.yml External port mapping "${APP_TDEX_PORT}:${APP_TDEX_PORT}":
Port mappings may be unnecessary for the app to function correctly. Docker's internal DNS resolves container names to IP addresses within the same network. External access to the web interface is handled by the app_proxy container. Port mappings are only needed if external access is required to a port not proxied by the app_proxy, or if an app needs to expose multiple ports for its functionality (e.g., DHCP, DNS, P2P, etc.).
ℹ️ tdex/docker-compose.yml Potentially using unsafe user in service "oceand":
The default container user "root" can lead to security vulnerabilities. If you are using the root user, please try to specify a different user (e.g. "1000:1000") in the compose file or try to set the UID/PUID and GID/PGID environment variables to 1000.
ℹ️ tdex/docker-compose.yml Potentially using unsafe user in service "tdexd":
The default container user "root" can lead to security vulnerabilities. If you are using the root user, please try to specify a different user (e.g. "1000:1000") in the compose file or try to set the UID/PUID and GID/PGID environment variables to 1000.
ℹ️ tdex/docker-compose.yml Potentially using unsafe user in service "dashboard":
The default container user "root" can lead to security vulnerabilities. If you are using the root user, please try to specify a different user (e.g. "1000:1000") in the compose file or try to set the UID/PUID and GID/PGID environment variables to 1000.
ℹ️ tdex/docker-compose.yml Potentially using unsafe user in service "caddy":
The default container user "root" can lead to security vulnerabilities. If you are using the root user, please try to specify a different user (e.g. "1000:1000") in the compose file or try to set the UID/PUID and GID/PGID environment variables to 1000.

Legend

Symbol Description
Error: This must be resolved before this PR can be merged.
⚠️ Warning: This is highly encouraged to be resolved, but is not strictly mandatory.
ℹ️ Info: This is just for your information.

@nmfretz nmfretz merged commit 67a8d8c into getumbrel:master Sep 18, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants