Merge pull request #995 from o1-labs/fix/revert-berkeley-upgrade

Revert berkeley upgrade instructions.

docs/berkeley-upgrade/appendix.mdx

@@ -0,0 +1,82 @@
+---
+title: Appendix
+sidebar_label: Appendix
+hide_title: true
+description: Berkeley Upgrade Appendix
+keywords:
+  - Berkeley
+  - upgrade
+  - appendix
+---
+
+# Appendix
+
+## Migration from o1labs/client-sdk to mina-signer
+
+The signing library `o1labs/client-sdk` was deprecated some time ago and will stop working after the Mina mainnet upgrade. All users should upgrade to use the [mina-signer](https://www.npmjs.com/package/mina-signer) library.
+
+Below you will find an example of how to use the `mina-signer` library. Please keep in mind the following:
+
+1. Make sure to adjust the `nonce` to the correct nonce on the account you want to use as "sender"
+1. Update the `url` variable with an existing Mina Node GraphQL endpoint
+
+```javascript
+import { Client } from 'mina-signer';
+
+// create the client and define the keypair
+
+const client = new Client({ network: 'testnet' }); // Mind the `network` client configuration option
+
+const senderPrivateKey = 'EKFd1Gx...'; // Sender's private key
+const senderPublicKey = 'B62qrDM...'; // Sender's public key, perhaps derived from the private key using `client.derivePublicKey(senderPrivateKey)`;
+
+// define and sign payment
+
+let payment = {
+  from: senderPublicKey,
+  to: 'B62qkBw...', // Recipient public key
+  amount: 100,
+  nonce: 1,
+  fee: 1000000,
+};
+
+const signedPayment = client.signPayment(payment, senderPrivateKey);
+
+// send payment to graphql endpoint
+
+const url = 'https://qanet.minaprotocol.network/graphql';
+
+const sendPaymentMutationQuery = `
+mutation SendPayment($input: SendPaymentInput!, $signature: SignatureInput!) {
+  sendPayment(input: $input, signature: $signature) {
+    payment {
+      hash
+    }
+  }
+}
+`;
+const graphQlVariables = {
+  input: signedPayment.data,
+  signature: signedPayment.signature,
+};
+const body = JSON.stringify({
+  query: sendPaymentMutationQuery,
+  variables: graphQlVariables,
+  operationName: 'SendPayment',
+});
+
+const paymentResponse = await fetch(url, {
+  method: 'POST',
+  headers: { 'Content-Type': 'application/json' },
+  body
+});
+
+const paymentResponseJson = await paymentResponse.json();
+if (paymentResponse.ok) {
+  console.log(`Transaction hash: ${paymentResponseJson.data.sendPayment.payment.hash}`);
+} else {
+  console.error(JSON.stringify(paymentResponseJson));
+}
+
+
+```

docs/berkeley-upgrade/archive-migration/appendix.mdx

@@ -0,0 +1,137 @@
+---
+title: Appendix
+sidebar_label: Appendix
+hide_title: true
+description: archive node schema changes between Mainnet and Berkeley
+keywords:
+  - Berkeley
+  - upgrade
+  - archive migration
+  - appendix
+  - mina archive node
+  - archive node
+---
+
+# Appendix
+
+## Archive node schema changes
+
+If you are using the Archive Node database directly for your system integrations, then you should understand all the changes that might impact your applications. The most important change is that the `balances` table in the Berkeley schema will no longer exist. In the new schema, it is replaced with the table `accounts_accessed` - from an application semantics point of view, the data in `accounts_accessed` is still the same.
+
+In the Berkeley protocol, accounts can now have the same public key but a different token_id. This means accounts are identified by both their public key and token_id, not just the public key. Consequently, the foreign key for the account in all tables is account_identifier_id instead of public_key_id.
+
+### Schema differences
+- **Removed Types**
+  - The options `create_token`, `create_account`, and `mint_tokens` have been removed from the user_command_type enumeration.
+- Indexes Dropped
+  - We've removed several indexes from tables, this may affect how you search and organize data:
+    - `idx_public_keys_id`
+    - `idx_public_keys_value`
+    - `idx_snarked_ledger_hashes_value`
+    - `idx_blocks_id`
+    - `idx_blocks_state_hash`
+- **Table Removed**
+  - The `balances` table is no longer available.
+- **New Tables Added**
+  - We've introduced the following new tables:
+    - `tokens`
+    - `token_symbols`
+    - `account_identifiers`
+    - `voting_for`
+    - `protocol_versions`
+    - `accounts_accessed`
+    - `accounts_created`
+    - `zkapp_commands`
+    - `blocks_zkapp_commands`
+    - `zkapp_field`
+    - `zkapp_field_array`
+    - `zkapp_states_nullable`
+    - `zkapp_states`
+    - `zkapp_action_states`
+    - `zkapp_events`
+    - `zkapp_verification_key_hashes`
+    - `zkapp_verification_keys`
+    - `zkapp_permissions`
+    - `zkapp_timing_info`
+    - `zkapp_uris`
+    - `zkapp_updates`
+    - `zkapp_balance_bounds`
+    - `zkapp_nonce_bounds`
+    - `zkapp_account_precondition`
+    - `zkapp_accounts`
+    - `zkapp_token_id_bounds`
+    - `zkapp_length_bounds`
+    - `zkapp_amount_bounds`
+    - `zkapp_global_slot_bounds`
+    - `zkapp_epoch_ledger`
+    - `zkapp_epoch_data`
+    - `zkapp_network_precondition`
+    - `zkapp_fee_payer_body`
+    - `zkapp_account_update_body`
+    - `zkapp_account_update`
+    - `zkapp_account_update_failures`
+- **Updated Tables**
+  - The following tables have been updated
+    - `timing_info`
+    - `user_commands`
+    - `internal_commands`
+    - `epoch_data`
+    - `blocks`
+    - `blocks_user_commands`
+    - `blocks_internal_commands`
+
+### Differences per table
+- **`timing_info`**
+  - Removed columns:
+    - `token`
+    - `initial_balance`
+- **`user_commands`**
+  - Removed columns:
+    - `fee_token`
+    - `token`
+- **`internal_commands`**
+  - Removed columns:
+    - `token`
+  - Renamed column
+    - `command_type` to `type`
+- **`epoch_data`**
+  - Added columns:
+    - `total_currency`
+    - `start_checkpoint`
+    - `lock_checkpoint`
+    - `epoch_length`
+- **`blocks`**
+  - Added columns:
+    - `last_vrf_output`
+    - `min_window_density`
+    - `sub_window_densities`
+    - `total_currency`
+    - `global_slot_since_hard_fork`
+    - `global_slot_since_genesis`
+    - `protocol_version_id`
+    - `proposed_protocol_version_id`
+  - Removed column:
+    - `global_slot`
+- **`blocks_user_commands`**
+  - Removed columns:
+    - `fee_payer_account_creation_fee_paid`
+    - `receiver_account_creation_fee_paid`
+    - `created_token`
+    - `fee_payer_balance`
+    - `source_balance`
+    - `receiver_balance`
+  - Added index:
+    - `idx_blocks_user_commands_sequence_no`
+- **`blocks_internal_commands`**
+  - Removed columns:
+    - `receiver_account_creation_fee_paid`
+    - `receiver_balance`
+  - Added indexes:
+    - `idx_blocks_internal_commands_sequence_no`
+    - `idx_blocks_internal_commands_secondary_sequence_no`
+
+### Rosetta API new operations
+
+The Berkeley upgrade introduces two new operation types:
+- `zkapp_fee_payer_dec`
+- `zkapp_balance_change`

docs/berkeley-upgrade/archive-migration/archive-migration-installation.mdx

@@ -0,0 +1,151 @@
+---
+title: Installing the archive migration package
+sidebar_label: Installing archive migration package
+hide_title: false
+description: Satisfying the archive migration prerequisites.
+keywords:
+  - Berkeley
+  - upgrade
+  - archive migration
+  - installing
+  - prerequisites
+  - mina archive node
+  - archive node
+---
+
+The archive node Berkeley migration package is sufficient for satisfying the migration from Devnet/Mainnet to Berkeley.
+However, it has some limitations. For example, the migration package does not migrate a non-canonical chain and it skips orphaned blocks that are not part of a canonical chain.
+
+To mitigate these limitations, the archive node maintenance package is available for use by archive node operators who want to maintain a copy of their Devnet and Mainnet databases for historical reasons.
+
+## Install with Google Cloud SDK
+
+The Google Cloud SDK installer does not always register a `google-cloud-sdk` apt package. The best way to install gsutil is using the apt repostory:
+
+```sh
+curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo gpg --dearmor -o /usr/share/keyrings/cloud.google.gpg
+echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
+sudo apt-get update && sudo apt-get install google-cloud-sdk
+```
+
+## Download the o1labs Mainnet archive database
+
+We strongly encourage you to perform the migration on your own data to preserve the benefits of decentralization. However, if you want to use the archive data that o1labs runs (for example, to bootstrap a new archive from SQL without waiting all day for the chain to download and replay), you can use the following steps:
+
+1. Download the Devnet/Mainnet archive data using cURL or gsutil:
+
+    - cURL:
+
+     For Devnet:
+     ```sh
+     curl https://storage.googleapis.com/mina-archive-dumps/devnet-archive-dump-{date}_0000.sql.tar.gz
+     ```
+
+     For Mainnet:
+     ```sh
+     curl https://storage.googleapis.com/mina-archive-dumps/mainnet-archive-dump-{date}_0000.sql.tar.gz
+       ```
+
+     To filter the dumps by date, replace `{date}` using the required `yyyy-dd-mm` format. For example, for March 15, 2024, use `2024-03-15`.
+
+     :warning: The majority of backups have the `0000` suffix. If a download with that name suffix is not available, try incrementing it. For example, `0001`, `0002`, and so on.
+
+    - gsutil:
+
+       ```sh
+       gsutil cp gs://mina-archive-dumps/mainnet-archive-dump-2024-01-15* .
+       ```
+
+2. Extract the tar package.
+
+   ```sh
+   tar -xvzf {network}-archive-dump-{date}_0000.sql.tar.gz {network}-archive-dump-{date}_0000.sql
+   ```
+
+3. Import the Devnet/Mainnet archive dump into the Berkeley database.
+
+    Run this command at the database server:
+
+    ```sh
+    psql -U {user} -f {network}-archive-dump-{date}_0000.sql
+    ```
+
+    The database in the dump **archive_balances_migrated** is created with the Devnet/Mainnet archive schema.
+
+    Note: This database does not have any Berkeley changes.
+
+## Ensure the location of Google Cloud bucket with the Devnet/Mainnet precomputed blocks
+
+The recommended method is to perform migration on your own data to preserve the benefits of decentralization.
+
+`gsutil cp gs://mina_network_block_data/{network}-*.json .`
+
+:warning: Precomputed blocks for the Mainnet network take ~800 GB of disk space. Plan for adequate time to download these blocks. The Berkeley migration app downloads them incrementally only when needed.
+
+## Validate the Devnet/Mainnet database
+
+The correct Devnet/Mainnet database state is crucial for a successful migration.
+
+[Missing blocks](/berkeley-upgrade/archive-migration/mainnet-database-maintenance#missing-blocks) is one the most frequent issues when dealing with the Devnet/Mainnet archive. Although this step is optional, it is strongly recommended that you verify the archive condition before you start the migration process.
+
+To learn how to maintain archive data, see [Devnet/Mainnet database maintenance](/berkeley-upgrade/archive-migration/mainnet-database-maintenance).
+
+## Download the migration applications
+
+Migration applications are distributed as part of the archive migration Docker and Debian packages.
+
+Choose the packages that are appropriate for your environment.
+
+### Debian packages
+
+To get the Debian packages:
+
+```
+CODENAME=bullseye
+CHANNEL=stable
+VERSION=3.0.1-e848ecb
+
+echo "deb [trusted=yes] http://packages.o1test.net $CODENAME $CHANNEL" | tee /etc/apt/sources.list.d/mina.list
+apt-get update
+apt-get install --allow-downgrades -y "mina-archive-migration=$VERSION"
+```
+
+### Docker image
+
+To get the Docker image:
+
+```
+docker pull gcr.io/o1labs-192920/mina-archive-migration:3.0.1-e848ecb-{codename}
+```
+
+Where supported codenames are:
+- bullseye
+- focal
+- buster
+
+
+## Devnet/Mainnet genesis ledger
+
+The Mina Devnet/Mainnet genesis ledger is stored in GitHub in the `mina` repository under the `genesis_ledgers` subfolder. However, if you are already running a daemon that is connected to the Mina Mainnet or the Devnet network, you already have the genesis ledger locally.
+
+## Berkeley database schema files
+
+You can get the Berkeley schema files from different locations:
+
+- GitHub repository from the `berkeley` branch.
+
+    Note: The `berkeley` branch can contain new updates regarding schema files, so always get the latest schema files instead of using an already downloaded schema.
+
+- Archive/Rosetta Docker from `berkeley` version
+
+### Example: Downloading schema sources from GitHub
+
+   ```sh
+   wget https://raw.githubusercontent.com/MinaProtocol/mina/berkeley/src/app/archive/zkapp_tables.sql
+
+   wget https://raw.githubusercontent.com/MinaProtocol/mina/berkeley/src/app/archive/create_schema.sql
+   ```
+
+## Next steps
+
+Congratulations on completing the essential preparation and verification steps. You are now ready to perform the migration steps in [Migrating Devnet/Mainnet Archive to Berkeley Archive](/berkeley-upgrade/archive-migration/migrating-archive-database-to-berkeley).