Skip to content

Commit

Permalink
services: Ingest tss transactions (#46)
Browse files Browse the repository at this point in the history 
* tables and interfaces for TSS

* TSS tables and the channel interface

* remove empty lines

* update

* adding semicolons

adding semicolons

* moving all migrations to one file

* make hash primary key instead of xdr

* missing ,

* remove the index on try_transaction_xdr and add column/index on try_transaction_hash

...also remove outgoing/incoming_status and have one status column

* Squashed commit of the following:

commit fb807aa
Author: gouthamp-stellar <[email protected]>
Date:   Tue Sep 3 10:28:49 2024 -0700

    remove the index on try_transaction_xdr and add column/index on try_transaction_hash

    ...also remove outgoing/incoming_status and have one status column

commit 6fc0dc2
Author: gouthamp-stellar <[email protected]>
Date:   Tue Sep 3 08:58:46 2024 -0700

    missing ,

commit a9cf4e3
Author: gouthamp-stellar <[email protected]>
Date:   Tue Sep 3 01:55:23 2024 -0700

    make hash primary key instead of xdr

commit c0f9d32
Author: gouthamp-stellar <[email protected]>
Date:   Fri Aug 30 15:18:27 2024 -0700

    moving all migrations to one file

commit 2de9898
Author: gouthamp-stellar <[email protected]>
Date:   Fri Aug 30 15:16:53 2024 -0700

    adding semicolons

    adding semicolons

commit 373c71a
Author: gouthamp-stellar <[email protected]>
Date:   Fri Aug 30 15:12:24 2024 -0700

    update

commit 3f9f9f0
Author: gouthamp-stellar <[email protected]>
Date:   Fri Aug 30 15:12:00 2024 -0700

    remove empty lines

commit 9920f48
Author: gouthamp-stellar <[email protected]>
Date:   Fri Aug 30 15:06:40 2024 -0700

    TSS tables and the channel interface

commit a58d519
Author: gouthamp-stellar <[email protected]>
Date:   Fri Aug 30 10:55:22 2024 -0700

    tables and interfaces for TSS

* commit #2

* changing from RpcIngestTxResponse ->  RpcGetIngestTxResponse

* latest changes

* add tests for ValidateOptions

* changes based on comments

* latest changes based on changes to interface

* string -> RPCTXStatus

* adding a transaction_builder.go which takes a list of operation xdrs and builds a transaction out of it

* moving transaction_service to the utils dir

* upper case Channel methods

* latest changes

* p.txService.NetworkPassPhrase()

* last commit before writing unit tests

* Making the transaction service more injectible and adding fields to the Payload

* typo

* typo

* Update 2024-08-28.0-tss_transactions.sql

* lint errors

* go mod tidy

* test cases + other changes

* remoce unused mocks

* error handler service returns errorHandlerService

* changes based on comments

* lint deadcode error - suppress for now

* removed deadcode

* changes after comments on transaction service pr

* TSS Error Handler Service

* removing print statements

* Update transaction_service.go

* responding to comments

* remove commented code

* tx service changes

* remove comment

* remove comment

* latest tx service changes

* adding a router + utils file

* removing println

* changed function name

* commit #1

* Code() helper function on RPCTXCode

* Code()

* Delete .env.swp

* adding a helpers file

* removing BuildTestFeeBumpTransaction

* adding to serve.go etc

* casing

* better test for Send

* resolving merge conflicts

* removing mockSleep

* delete channels/mocks.go

* sleep -> time.Sleep

* incorporating Daniel's changes + comments

* delete files

* remove unused code

* name change

* remove commented code

* moving the mocks file inside servicesmocks dir

* name changes

* refactor

* changes to serve.go

* checking error on route.Route

* check error

* merging main

* fixing parsesendresp tests

* commit changes

* merging in changes from eror_handler_service

* delete file

* commit current changes before merging branch

* rpc ingestor

* delete file

* fix lint errors

* remove dead code

* fix broken test + delete unused code

* tidy mod

* remove commented code

* delete file

* changes based on comments

* removing test case that is not relevant anymore

* changes based on prev pr comments

* remove fmt.Println

* merge latest error_handler_service branch + small changes

* %s -> %w

* U -> u

* variable for channel name

* account for NewStore returning an error

* fix build

* merge main

* merged latest

* typo in desc string

* millisecond wait time

* changes based on comments

* test case for payment ingestion + RPC setup + removing current core setup
  • Loading branch information
@gouthamp-stellar
gouthamp-stellar authored Oct 23, 2024
1 parent 6553fa9 commit 3123476
Show file tree
Hide file tree
Showing 32 changed files with 1,300 additions and 825 deletions.
22 changes: 5 additions & 17 deletions Dockerfile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,28 +8,16 @@ RUN go mod download
COPY . ./
RUN go build -o /bin/wallet-backend -ldflags "-X main.GitCommit=$GIT_COMMIT" .

# Use the official stellar/soroban-rpc image as the base
FROM stellar/soroban-rpc

# Install bash or sh
RUN apt-get update && apt-get install -y bash

# Step 2: Install Stellar Core and copy over app binary
FROM ubuntu:jammy AS core-build

ARG STELLAR_CORE_VERSION

RUN apt-get update && \
apt-get install -y --no-install-recommends ca-certificates curl wget gnupg apt-utils gpg && \
curl -sSL https://apt.stellar.org/SDF.asc | gpg --dearmor >/etc/apt/trusted.gpg.d/SDF.gpg && \
echo "deb https://apt.stellar.org jammy stable" >/etc/apt/sources.list.d/SDF.list && \
echo "deb https://apt.stellar.org jammy testing" >/etc/apt/sources.list.d/SDF-testing.list && \
echo "deb https://apt.stellar.org jammy unstable" >/etc/apt/sources.list.d/SDF-unstable.list && \
apt-get update && \
apt-get install -y stellar-core=${STELLAR_CORE_VERSION} && \
apt-get clean

COPY --from=api-build /bin/wallet-backend /app/
COPY --from=api-build /src/wallet-backend/internal/ingest/config/stellar-core_pubnet.cfg /app/config/stellar-core_pubnet.cfg
COPY --from=api-build /src/wallet-backend/internal/ingest/config/stellar-core_testnet.cfg /app/config/stellar-core_testnet.cfg

ENV CAPTIVE_CORE_BIN_PATH /usr/bin/stellar-core
ENV CAPTIVE_CORE_CONFIG_DIR /app/config/

EXPOSE 8001
WORKDIR /app
Expand Down
46 changes: 19 additions & 27 deletions cmd/ingest.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@ package cmd
import (
"fmt"
"go/types"
"net/http"
"time"

_ "github.com/lib/pq"
"github.com/spf13/cobra"
Expand All @@ -11,6 +13,7 @@ import (
"github.com/stellar/wallet-backend/cmd/utils"
"github.com/stellar/wallet-backend/internal/apptracker/sentry"
"github.com/stellar/wallet-backend/internal/ingest"
tsschannels "github.com/stellar/wallet-backend/internal/tss/channels"
)

type ingestCmd struct{}
Expand All @@ -22,28 +25,15 @@ func (c *ingestCmd) Command() *cobra.Command {
cfgOpts := config.ConfigOptions{
utils.DatabaseURLOption(&cfg.DatabaseURL),
utils.LogLevelOption(&cfg.LogLevel),
utils.NetworkPassphraseOption(&cfg.NetworkPassphrase),
utils.SentryDSNOption(&sentryDSN),
utils.StellarEnvironmentOption(&stellarEnvironment),
utils.RPCURLOption(&cfg.RPCURL),
{
Name: "captive-core-bin-path",
Usage: "Path to Captive Core's binary file.",
OptType: types.String,
CustomSetValue: utils.SetConfigOptionCaptiveCoreBinPath,
ConfigKey: &cfg.CaptiveCoreBinPath,
FlagDefault: "/usr/local/bin/stellar-core",
Required: true,
},
{
Name: "captive-core-config-dir",
Usage: "Path to Captive Core's configuration files directory.",
OptType: types.String,
CustomSetValue: utils.SetConfigOptionCaptiveCoreConfigDir,
ConfigKey: &cfg.CaptiveCoreConfigDir,
FlagDefault: "./internal/ingest/config",
Required: true,
},
utils.StartLedgerOption(&cfg.StartLedger),
utils.EndLedgerOption(&cfg.EndLedger),
utils.WebhookHandlerServiceChannelMaxBufferSizeOption(&cfg.WebhookChannelMaxBufferSize),
utils.WebhookHandlerServiceChannelMaxWorkersOptions(&cfg.WebhookChannelMaxWorkers),
utils.WebhookHandlerServiceChannelMaxRetriesOption(&cfg.WebhookChannelMaxRetries),
utils.WebhookHandlerServiceChannelMinWaitBtwnRetriesMSOption(&cfg.WebhookChannelWaitBtwnTriesMS),
{
Name: "ledger-cursor-name",
Usage: "Name of last synced ledger cursor, used to keep track of the last ledger ingested by the service. When starting up, ingestion will resume from the ledger number stored in this record. It should be an unique name per container as different containers would overwrite the cursor value of its peers when using the same cursor name.",
Expand All @@ -60,14 +50,6 @@ func (c *ingestCmd) Command() *cobra.Command {
FlagDefault: 0,
Required: false,
},
{
Name: "end",
Usage: "Ledger number up to which ingestion should run. When not present, ingestion run indefinitely (live ingestion requires it to be empty).",
OptType: types.Int,
ConfigKey: &cfg.EndLedger,
FlagDefault: 0,
Required: false,
},
}

cmd := &cobra.Command{
Expand All @@ -85,11 +67,21 @@ func (c *ingestCmd) Command() *cobra.Command {
return fmt.Errorf("initializing app tracker: %w", err)
}
cfg.AppTracker = appTracker
cfg.WebhookChannel = tsschannels.NewWebhookChannel(tsschannels.WebhookChannelConfigs{
HTTPClient: &http.Client{Timeout: 30 * time.Second},
MaxBufferSize: cfg.WebhookChannelMaxBufferSize,
MaxWorkers: cfg.WebhookChannelMaxWorkers,
MaxRetries: cfg.WebhookChannelMaxRetries,
MinWaitBtwnRetriesMS: cfg.WebhookChannelWaitBtwnTriesMS,
})
return nil
},
RunE: func(_ *cobra.Command, _ []string) error {
return c.Run(cfg)
},
PersistentPostRun: func(_ *cobra.Command, _ []string) {
cfg.WebhookChannel.Stop()
},
}

if err := cfgOpts.Init(cmd); err != nil {
Expand Down
44 changes: 0 additions & 44 deletions cmd/utils/custom_set_value.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,7 @@ package utils

import (
"encoding/json"
"errors"
"fmt"
"os"
"strings"

"github.com/sirupsen/logrus"
Expand Down Expand Up @@ -83,48 +81,6 @@ func SetConfigOptionStellarPrivateKey(co *config.ConfigOption) error {
return nil
}

func SetConfigOptionCaptiveCoreBinPath(co *config.ConfigOption) error {
binPath := viper.GetString(co.Name)

fileInfo, err := os.Stat(binPath)
if errors.Is(err, os.ErrNotExist) {
return fmt.Errorf("binary file %s does not exist", binPath)
}

if fileInfo.IsDir() {
return fmt.Errorf("binary file path %s is a directory, not a file", binPath)
}

key, ok := co.ConfigKey.(*string)
if !ok {
return unexpectedTypeError(key, co)
}
*key = binPath

return nil
}

func SetConfigOptionCaptiveCoreConfigDir(co *config.ConfigOption) error {
dirPath := viper.GetString(co.Name)

fileInfo, err := os.Stat(dirPath)
if errors.Is(err, os.ErrNotExist) {
return fmt.Errorf("captive core configuration files dir %s does not exist", dirPath)
}

if !fileInfo.IsDir() {
return fmt.Errorf("captive core configuration files dir %s is not a directory", dirPath)
}

key, ok := co.ConfigKey.(*string)
if !ok {
return unexpectedTypeError(key, co)
}
*key = dirPath

return nil
}

func SetConfigOptionAssets(co *config.ConfigOption) error {
assetsJSON := viper.GetString(co.Name)

Expand Down
91 changes: 0 additions & 91 deletions cmd/utils/custom_set_value_test.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -220,97 +220,6 @@ func Test_SetConfigOptionLogLevel(t *testing.T) {
}
}

func TestSetConfigOptionCaptiveCoreBinPath(t *testing.T) {
opts := struct{ binPath string }{}

co := config.ConfigOption{
Name: "captive-core-bin-path",
OptType: types.String,
CustomSetValue: SetConfigOptionCaptiveCoreBinPath,
ConfigKey: &opts.binPath,
}

testCases := []customSetterTestCase[string]{
{
name: "returns an error if the file path is not set, should be caught by the Require() function",
wantErrContains: "binary file does not exist",
},
{
name: "returns an error if the path is invalid",
args: []string{"--captive-core-bin-path", "/a/random/path/bin"},
wantErrContains: "binary file /a/random/path/bin does not exist",
},
{
name: "returns an error if the path format is invalid",
args: []string{"--captive-core-bin-path", "^7JcrS8J4q@V0$c"},
wantErrContains: "binary file ^7JcrS8J4q@V0$c does not exist",
},
{
name: "returns an error if the path is a directory, not a file",
args: []string{"--captive-core-bin-path", "./"},
wantErrContains: "binary file path ./ is a directory, not a file",
},
{
name: "sets to ENV var value",
envValue: "./custom_set_value_test.go",
wantResult: "./custom_set_value_test.go",
},
}

for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
opts.binPath = ""
customSetterTester(t, tc, co)
})
}
}

func TestSetConfigOptionCaptiveCoreConfigDir(t *testing.T) {
opts := struct{ binPath string }{}

co := config.ConfigOption{
Name: "captive-core-config-dir",
OptType: types.String,
CustomSetValue: SetConfigOptionCaptiveCoreConfigDir,
ConfigKey: &opts.binPath,
}

testCases := []customSetterTestCase[string]{
{
name: "returns an error if the file path is not set, should be caught by the Require() function",
wantErrContains: "captive core configuration files dir does not exist",
},
{
name: "returns an error if the path is invalid",
envValue: "/a/random/path",
wantErrContains: "captive core configuration files dir /a/random/path does not exist",
},
{
name: "returns an error if the path format is invalid",
envValue: "^7JcrS8J4q@V0$c",
wantErrContains: "captive core configuration files dir ^7JcrS8J4q@V0$c does not exist",
},

{
name: "returns an error if the path is a file, not a directory",
envValue: "./custom_set_value_test.go",
wantErrContains: "captive core configuration files dir ./custom_set_value_test.go is not a directory",
},
{
name: "sets to ENV var value",
envValue: "../../internal/ingest/config",
wantResult: "../../internal/ingest/config",
},
}

for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
opts.binPath = ""
customSetterTester(t, tc, co)
})
}
}

func TestSetConfigOptionAssets(t *testing.T) {
opts := struct{ assets []entities.Asset }{}

Expand Down
26 changes: 25 additions & 1 deletion cmd/utils/global_options.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ func RPCURLOption(configKey *string) *config.ConfigOption {
Usage: "The URL of the RPC Server.",
OptType: types.String,
ConfigKey: configKey,
FlagDefault: "localhost:8080",
FlagDefault: "http://soroban-rpc:8000",
Required: true,
}
}
Expand Down Expand Up @@ -142,6 +142,30 @@ func DistributionAccountSignatureClientProviderOption(configKey *signing.Signatu
}
}

func StartLedgerOption(configKey *int) *config.ConfigOption {
return &config.ConfigOption{
Name: "start-ledger",
Usage: "ledger number to start getting transactions from",
OptType: types.Int,
ConfigKey: configKey,
FlagDefault: 0,
Required: true,
}

}

func EndLedgerOption(configKey *int) *config.ConfigOption {
return &config.ConfigOption{
Name: "end-ledger",
Usage: "ledger number to end on",
OptType: types.Int,
ConfigKey: configKey,
FlagDefault: 0,
Required: true,
}

}

func AWSOptions(awsRegionConfigKey *string, kmsKeyARN *string, required bool) config.ConfigOptions {
awsOpts := config.ConfigOptions{
{
Expand Down
Loading

0 comments on commit 3123476

Please sign in to comment.