diff --git a/README.md b/README.md index 59ce70f6..f58c778d 100644 --- a/README.md +++ b/README.md @@ -1,45 +1,96 @@ -# Overview +## Overview + +The Shardeum JSON-RPC Server enables developers to interact with the Shardeum blockchain network. It allows dapps to post requests, retrieve information, and perform other related operations using JSON-RPC over HTTP. Additionally, the Shardeum JSON-RPC Server comes with an added REST API for debugging and monitoring purposes. + +For running existing dapps on Shardeum, refer to the [EVM JSON-RPC API](https://ethereum.org/en/developers/docs/apis/json-rpc/). + +## Table of Contents + +- [Project Structure](#project-structure) +- [Docker Setup](#docker-setup) + - [Start JSON-RPC Server](#start-json-rpc-server) + - [Check the Logs](#check-the-logs) + - [Clean the Setup](#clean-the-setup) +- [Developer Environment Setup](#developer-environment-setup) + - [Requirements](#requirements) + - [Installing Project Source Code](#installing-project-source-code) + - [Starting JSON-RPC Server](#starting-json-rpc-server) + - [Cleanup](#cleanup) +- [Usage](#usage) + - [REST API Endpoints](#rest-api-endpoints) + - [Debug Endpoints](#debug-endpoints) +- [Configuration](#configuration) +- [Notable Modules and Functions](#notable-modules-and-functions) + - [api.ts](#apits) + - [clients.ts](#clientsts) + - [index.ts](#indexts) + - [log_server.ts](#log_serverts) +- [Contributing](#contributing) +- [License](#license) +- [Contact](#contact) + +## Project Structure + +The project is organized as follows: -The Shardeum JSON-RPC Server enables developers to interact with the Shardeum blockchain network. It allows dapps to post request, retrieve information, and other related operations, using JSON-RPC over HTTP. Additionally, the Shardeum JSON-RPC Server comes with an added REST API for debugging and monitoring purposes. - -For running existing dapps on Shardeum, refer to EVM [JSON-RPC API](https://ethereum.org/en/developers/docs/apis/json-rpc/) - -## Docker setup +``` +json-rpc-server-dev/ +├── .appsec/ # Security baseline configurations +├── src/ # Source code +│ ├── api.ts # Main API implementation +│ ├── config.ts # Configuration settings +│ ├── logger.ts # Logging utility +│ ├── server.ts # Server setup and initialization +│ ├── types.ts # Type definitions +│ ├── utils/ # Utility functions +│ ├── cache/ # Cache management +│ ├── external/ # External service integrations +│ ├── middlewares/ # Middleware functions +│ ├── routes/ # API routes +│ ├── service/ # Core service implementations +│ ├── storage/ # Storage solutions +│ ├── websocket/ # WebSocket implementations +├── Dockerfile # Docker configuration +├── docker-compose.yml # Docker Compose configuration +├── README.md # Project documentation +├── package.json # Project metadata and dependencies +└── tsconfig.json # TypeScript configuration +``` -For developers deploying to production environments, simply use `docker compose` command. +## Docker Setup -env `NO_OF_RPC_SERVERS` creates replicas of rpc servers using pm2. default is 1. Default port is 8080, port for each replicas will increment by 1 on default port. i.e 8081, 8082, 8083, etc. +For developers deploying to production environments, use the `docker compose` command. The environment variable `NO_OF_RPC_SERVERS` creates replicas of RPC servers using PM2. The default is 1. The default port is 8080, and ports for each replica will increment by 1 (e.g., 8081, 8082, 8083). -### Start json-rpc-server +### Start JSON-RPC Server ```shell # Run services in detach mode docker compose up -d ``` -### Check the logs +### Check the Logs ```shell docker compose logs -f ``` -### Clean the setup +### Clean the Setup ```shell docker compose down ``` -# Developer Environment Setup +## Developer Environment Setup For end users, such as exchanges and large decentralized applications (dApps), seeking to deploy their own RPC server, it is recommended to run the Shardeum JSON-RPC server using Docker. It ensures all dependencies are installed and the server is running in a consistent environment. For developers who want to contribute to this project, running the server from source is recommended. You can use `npm` for installing the server locally. -## Requirements +### Requirements -If you are using `Docker`, in order to run the Shardeum JSON-RPC server, you must have the [Docker](https://docs.docker.com/get-docker/) daemon installed. +To run the Shardeum JSON-RPC server, you must have the [Docker](https://docs.docker.com/get-docker/) daemon installed. -## Installing project source code +### Installing Project Source Code -Let’s install the project source code, switch to `dev` branch and follow the below instructions: +Follow these steps to install the project source code and switch to the `dev` branch: ```bash git clone https://github.com/shardeum/json-rpc-server.git @@ -47,86 +98,139 @@ cd shardeum-json-rpc git switch dev ``` -If you are building using `Docker`, run the below command: +### Building the Project -```bash -make build -``` +- **Using Docker**: -This command will build Docker images for the service. + ```bash + make build + ``` -If you are building using `NPM`, run the below command, it will install all the required dependencies + This command will build Docker images for the service. -```bash -npm install -``` +- **Using NPM**: -## Starting JSON-RPC Server + ```bash + npm install + ``` -If you are building using `Docker`, you can start the JSON-RPC server by running the following command: + This command will install all the required dependencies. -```bash -make run -``` +### Starting JSON-RPC Server -This will start a container running the `shardeum-json-rpc` server image, available on port `8080`. The servers configuration fields can be viewed and edited in the `src/config.ts` file. Additionally, you can manage the server's access control lists by editing the `whitelist.json`, `blacklist.json` and `spammerlist.json`. +- **Using Docker**: -But if you are using NPM, use the below command to run the server: + ```bash + make run + ``` -```bash -npm run start -``` + This will start a container running the `shardeum-json-rpc` server image, available on port `8080`. Configuration fields can be viewed and edited in the `src/config.ts` file. Manage the server's access control lists by editing the `whitelist.json`, `blacklist.json`, and `spammerlist.json`. -If you want to modify the chainId or the port number, go to `src/config.ts` file: +- **Using NPM**: -```bash -chainId: 8082 -port: 8080 -``` + ```bash + npm run start + ``` -The RPC URL for using Metamask with Remix IDE and for running scripts is (default: ) + Modify the `chainId` or the `port` number in the `src/config.ts` file: -If you are contributing to this project, use Shardeum server to create the network from within the [validator repo](https://gitlab.com/shardus/archive/archive-server). You can find more details [here](https://github.com/shardeum/shardeum) + ```typescript + chainId: 8082 + port: 8080 + ``` -## Cleanup + The RPC URL for using Metamask with Remix IDE and for running scripts is (default: ). -If you are using `Docker`, you can stop the server by running: + For contributing to this project, use the Shardeum server to create the network from within the [validator repo](https://gitlab.com/shardus/archive/archive-server). More details can be found [here](https://github.com/shardeum/shardeum). -```bash -make stop -``` +### Cleanup -Additionally, you can remove the docker images by running: +- **Using Docker**: -```bash -make clean -``` + ```bash + make stop + ``` + + Remove the Docker images by running: + + ```bash + make clean + ``` + + This will remove all Docker images created by the server during the build process. + +## Usage + +### REST API Endpoints + +- **GET `/authenticate/:passphrase`**: Authenticate using the `passphrase` set in `config.ts` or system environment variable. +- **GET `/log/api-stats`**: Emit RPC interface call counts, average TPS, and other information. Supports querying by time range (`/log/api-stats?start={x}&end={x}`). +- **GET `/log/txs`**: Return the transactions made through the RPC server. Supports dynamic pagination (`/log/txs?max=30&page=9`). +- **GET `/log/status`**: Return the status of logging. +- **GET `/log/startTxCapture`**: Enable capturing of incoming transactions. +- **GET `/log/stopTxCapture`**: Disable capturing of incoming transactions. +- **GET `/log/startRPCCapture`**: Enable capturing of RPC interface call stats. +- **GET `/log/stopRPCCapture`**: Disable capturing of RPC interface call stats. +- **GET `/cleanStatTable`**: Trigger purging of the stats table. +- **GET `/cleanTxTable`**: Trigger purging of the transactions table. -This will remove all docker images created by the server during the build process. +### Debug Endpoints -# DEBUG Endpoints +These APIs are protected, preventing the general public from wiping out debug data. To authenticate, use `/authenticate/:passphrase`. The `passphrase` is set in `config.ts` or within the system environment variable. -These api are protected preventing general public to wiping out debug data to authenticate use `/authenticate/:passphrase`. `passphrase` is set in `config.ts` config file or within the system env variable. +- **GET `/log/api-stats`**: Emits RPC interface call counts and average TPS along with other information. Supports query by time range (e.g., `/log/api-stats?start={x}&end={x}`). The parameter value can be either `yyyy-mm-dd` or Unix epoch in milliseconds. +- **GET `/log/txs`**: Returns the transactions made through the RPC server. Supports dynamic pagination (e.g., `/log/txs?max=30&page=9`). Default values are `1000` for `max` and `0` for `page`. +- **GET `/log/status`**: Returns the status of logging, such as the date of recording start and whether or not recording is enabled. +- **GET `/log/startTxCapture`**: Sets the config value to true, enabling the capture of incoming transactions. +- **GET `/log/stopTxCapture`**: Sets the config value to false, disabling the capture of incoming transactions. +- **GET `/log/startRPCCapture`**: Sets the config value to true, enabling the capture of RPC interface call stats. +- **GET `/log/stopRPCCapture`**: Sets the config value to false, disabling the capture of RPC interface call stats. +- **GET `/cleanStatTable`**: Triggers purging of the table that stores interface stats. +- **GET `/cleanTxTable`**: Triggers purging of the table that stores transaction logging. -GET `/log/api-stats` this endpint emit the rpc interface call counts and avg tps along with a few a other information. This endpoint support query by time range. i.e `/log/api-stats?start={x}&end={x}`. The parameter value can be either `yyyy-mm-dd` or unix epoch in millisecond. (NOTE standard unix epoch is in seconds which does not work, it has to be in millisecond accuracy). Not setting any timestamp parameter will returns paginated json of all the entry in db. +## Configuration -GET `/log/txs` this endpoint return the txs it has been made through rpc server. This endpoint support dynmaic pagination. i.e `/log/txs?max=30&page=9`. -Default values are `1000` for `max` and `0` for page. +Important configuration files and parameters: -GET `/log/status` this endpint return status of logging such as date of recording start and whether or not recording is enabled. +- **`config.ts`**: Contains various configuration settings for the server. +- **Environment Variables**: + - `NO_OF_RPC_SERVERS`: Number of RPC server replicas (default is 1). + - `PORT`: Port number for the server (default is 8080). -GET `/log/startTxCapture` this endpoint set the config value to true which control whether to capture incoming txs and store in database. +## Notable Modules and Functions -GET `/log/stopRPCCapture` this endpoint set the config value to false which control whether to capture incoming rpc interface call stat and store in database +## `api.ts` -GET `/log/startRPCCapture` this endpoint set the config value to true which control whether to capture rpc interface call stat and store in database. +- **Core RPC Methods**: Handles various JSON-RPC methods such as `eth_call`, `eth_sendTransaction`, and more. +- **Utilities**: Includes utility functions for validating addresses, handling transactions, and interacting with the Ethereum ecosystem. +- **Error Handling**: Uses `serializeError` to format and return errors in a standardized way. -GET `/log/stopTxCapture` this endpoint set the config value to false which control whether to capture incoming txs and store in database +## `clients.ts` -GET `/cleanStatTable` this endpoint trigger purging of table that store interface stats +- **ClientList Class**: Manages active WebSocket clients and their subscription details. +- **Subscription Management**: Provides methods to add, remove, and manage client subscriptions. +- **Mapping**: Maps subscription IDs to request IDs and vice versa for efficient lookup and management. -GET `/cleanTxTable` this endpoint trigger purging of table that store transaction logging +## `index.ts` -# Contributing +- **WebSocket Server Setup**: Initializes and manages WebSocket server connections. +- **Event Handling**: Manages events for incoming connections, message handling, and disconnections. +- **Subscription Integration**: Uses `ClientList` to manage subscriptions and emit data to clients. + +## `log_server.ts` + +- **Log Server Connection**: Sets up and maintains WebSocket connection to the log server. +- **Event Handling**: Manages events for log-related messages. +- **Configuration Integration**: Uses configuration settings to manage log stream setup and maintenance. + +## Contributing Contributions are very welcome! Everyone interacting in our codebases, issue trackers, and any other form of communication, including chat rooms and mailing lists, is expected to follow our [code of conduct](CODE_OF_CONDUCT.md) so we can all enjoy the effort we put into this project. + +## License + +This project is licensed under the terms of the [MIT license](LICENSE). + +## Contact + +For any questions or suggestions, feel free to reach out to us at [support@shardeum.org](mailto:support@shardeum.org). diff --git a/pm2.js b/pm2.js index 70559fda..489f6dd0 100644 --- a/pm2.js +++ b/pm2.js @@ -1,44 +1,82 @@ -const pm2 = require('pm2') -const count = parseInt(process.argv[2]) -const config = require('./dist/src/config.js') -const startingPort = config.port ?? 8080 +const pm2 = require('pm2'); +const count = parseInt(process.argv[2]); -console.log(`count: ${count} starting port:${startingPort}`) +if (isNaN(count) || count <= 0) { + console.error('Invalid count value. Please provide a positive integer.'); + process.exit(1); +} + +const config = require('./dist/src/config.js'); +const startingPort = config.port ?? 8080; +console.log(`count: ${count}, starting port: ${startingPort}`); pm2.connect(function (err) { - if (err) { - console.error(err) - process.exit(2) - } - - for (let i = 0; i < count; i++) { - startRPC(startingPort + i) - } - setTimeout(() => { - console.log(`Run "pm2 list" to see started processes.`) - process.exit(0) - }, 3000) - return -}) - -function startRPC(port) { - const processName = 'rpc_' + port - pm2.start( - { - script: 'dist/src/server.js', - name: processName, - args: String(port), - }, - function (err, apps) { - if (err) { - console.error(err) - pm2.restart(processName, (err, proc) => { - pm2.disconnect() - }) - return - } else { - console.log(`Started RPC server at port ${port}`) - } + if (err) { + console.error('Failed to connect to PM2:', err); + process.exit(2); + } + + let processesStarted = 0; + let processesFailed = 0; + + for (let i = 0; i < count; i++) { + startRPC(startingPort + i, onProcessComplete); } - ) + + function onProcessComplete(err) { + if (err) { + processesFailed++; + } else { + processesStarted++; + } + if (processesStarted + processesFailed === count) { + setTimeout(() => { + console.log(`Run "pm2 list" to see started processes.`); + pm2.disconnect(() => { + process.exit(processesFailed > 0 ? 1 : 0); + }); + }, 3000); + } + } +}); + +function startRPC(port, callback, retries = 3) { + const processName = 'rpc_' + port; + pm2.start( + { + script: 'dist/src/server.js', + name: processName, + args: String(port), + }, + function (err, apps) { + if (err) { + console.error(`Failed to start RPC server at port ${port}:`, err); + if (retries > 0) { + console.log(`Retrying... attempts left: ${retries}`); + pm2.restart(processName, (restartErr, proc) => { + if (restartErr) { + console.error(`Failed to restart RPC server at port ${port}:`, restartErr); + return callback(restartErr); + } + console.log(`Restarted RPC server at port ${port}`); + return callback(null); + }); + } else { + return callback(err); + } + } else { + console.log(`Started RPC server at port ${port}`); + callback(null); + } + } + ); } + +// Graceful shutdown handling +process.on('SIGINT', () => { + console.info('SIGINT signal received.'); + pm2.disconnect(() => { + console.log('PM2 disconnected.'); + process.exit(0); + }); +});