Code

Fixes to SSH and use of SFTP#31

Code

Closed

aronchick wants to merge 213 commits intomainbacalhau-project/andaime:mainfrom aronchick/finish-aws-spotbacalhau-project/andaime:aronchick/finish-aws-spotCopy head branch name to clipboard

+35,575-16,199

Commits

Commits on Nov 8, 2024

• refactor: Mark CDK removal complete and update SOP next steps

  aronchick

  committed
  2765d4c

• refactor: Remove CDK dependencies from go.mod

  aronchick

  committed
  03497e5

• chore: Revert SOP status for CDK removal task

  aronchick

  committed
  2311edc

• chore: Update dependencies to latest versions

  aronchick

  committed
  9932944

• Based on the changes, I'll help you remove CDK-specific code. Could you show me the files that import or use CDK-related packages? Typically, these would be in:

  Show description for 33ef9d0
  aronchick

  committed
  33ef9d0

• refactor: Remove CDK dependencies from create_deployment

  aronchick

  committed
  d92127b

• refactor: Remove CDK dependencies from AWS provider interface

  aronchick

  committed
  39eba29

• refactor: Remove CDK dependencies and replace with direct AWS SDK VPC creation

  aronchick

  committed
  3f89bf3

• refactor: Remove unused AWS CDK imports from provider.go

  aronchick

  committed
  bbe4257

• refactor: Remove AWS CDK dependencies from AWS provider

  aronchick

  committed
  7683866

• fix: Update AWS resource filter to use hardcoded deployment tag

  aronchick

  committed
  b26e54c

• feat: Add AWS CDK dependencies to go.mod

  aronchick

  committed
  574ba8c

• refactor: Remove CDK dependencies and replace with direct AWS SDK calls

  aronchick

  committed
  cc06f28

• refactor: Remove unused AWS CDK dependencies and clean up imports

  aronchick

  committed
  fc544c5

Commits on Nov 9, 2024

• refactor: Update AWS provider to remove CDK Stack references and fix method names

  aronchick

  committed
  deda6f1

• refactor: Remove unused imports in AWS provider and test files

  aronchick

  committed
  3936557

• refactor: Remove unused Stack-related fields and references in AWS provider

  aronchick

  committed
  c7bb489

• refactor: Fix indentation in AWS VPC creation method

  aronchick

  committed
  ef4d8b6

• refactor: Remove CloudFormation stack references and update VPC creation

  aronchick

  committed
  810fc20

• feat: Remove CloudFormation client from AWS provider

  aronchick

  committed
  2ea36e3

• fix: Correct CloudFormation client method call in diagnostics

  aronchick

  committed
  21db87d

• refactor: Remove CloudFormation references and update EC2Clienter interface

  aronchick

  committed
  e46db91

• refactor: Remove CloudFormation diagnostics and stack deletion logic

  aronchick

  committed
  c57800e

• refactor: Remove unused CloudFormation code and clean up AWS provider

  aronchick

  committed
  6bf9aac

• refactor: Remove CloudFormation dependencies and migrate to direct EC2 SDK calls

  aronchick

  committed
  017e23f

• refactor: Update AWS provider to use client from struct and remove unused imports

  aronchick

  committed
  1f7cc7b

• fix: Remove unused CloudFormation client initialization in AWS provider

  aronchick

  committed
  228c4ca

• refactor: Add DescribeAvailabilityZones method to LiveEC2Client

  aronchick

  committed
  9582e1e

• initial unit testing for aws provider

  aronchick

  committed
  85f3dd6

• refactor: Mark completed tasks in SOP and update documentation status

  aronchick

  committed
  41cca81

• docs: Add comprehensive documentation for AWS CreateInfrastructure and Destroy methods

  aronchick

  committed
  600facb

• docs: Add detailed documentation for CreateInfrastructure method

  aronchick

  committed
  4e77555

• feat: Add integration and performance tests for AWS provider

  aronchick

  committed
  fa0882f

• docs: Update SOP with completed testing and error handling tasks

  aronchick

  committed
  91e3422

• docs: Complete AWS provider documentation for API, migration, and configuration

  aronchick

  committed
  64202c9

• feat: Ensure EC2 client initialization in AWS deployment creation

  aronchick

  committed
  9cd8c33

• refactor: Ensure EC2 client initialization in AWS deployment creation

  aronchick

  committed
  a81720b

• fix: Initialize AWS EC2 client correctly in create deployment

  aronchick

  committed
  4ebcb2f

Commits on Nov 10, 2024

• updated naming and caps

  aronchick

  committed
  a6e95e7

• feat: Add VPC availability check and network propagation delay for AWS deployment

  aronchick

  committed
  8ac0443

• fix: Update VPC state type import in AWS provider

  aronchick

  committed
  96e9762

• refactor: Implement exponential backoff for VPC availability check

  aronchick

  committed
  2480ffb

• refactor: Update AWS provider test to mock VPC status check

  aronchick

  committed
  b0362b3

• feat: Add network connectivity check with exponential backoff for AWS deployment

  aronchick

  committed
  7052ba7

• refactor: Add DescribeRouteTables method to EC2Clienter interface

  aronchick

  committed
  ff5cdbb

• refactor: Update AWS provider imports and filter types

  aronchick

  committed
  f9adf6d

• refactor: Adjust import order and method visibility for AWS provider

  aronchick

  committed
  f924f36

• feat: Add VPC ID tracking in config during create and destroy

  aronchick

  committed
  1d83ffb

• feat: Add display and viper imports to AWS VPC provider

  aronchick

  committed
  36672f7

• refactor: Update network connectivity wait method call in create deployment

  aronchick

  committed
  1e37c3f

• feat: Add SSH connectivity check before Bacalhau cluster provisioning

  aronchick

  committed
  55b433a

• refactor: Add SSH connectivity check before Bacalhau cluster provisioning

  aronchick

  committed
  976dfe0

• feat: Add parallel VM deployment with SSH polling for AWS provider

  aronchick

  committed
  bd5a154

• refactor: Update AWS compute operations package name to match existing provider

  aronchick

  committed
  a4e235a

• refactor: Fix SSH config and error handling in AWS and GCP providers

  aronchick

  committed
  e947578

• refactor: Fix SSH config and provider method calls in AWS and GCP providers

  aronchick

  committed
  bd0e0f4

• refactor: Update AWS compute operations with new SSH config and method names

  aronchick

  committed
  e5f2221

• feat: Implement parallel AWS VM deployment with resource state tracking

  aronchick

  committed
  262b0b3

• feat: Update AWS compute operations to use EC2Client interface methods

  aronchick

  committed
  a38ec37

• feat: Update AWS provider with missing fields and methods

  aronchick

  committed
  0ce39c2

• refactor: Update AWS VM creation method and type handling

  aronchick

  committed
  3b36d77

• refactor: Implement full EC2Clienter interface with WaitUntilInstanceRunning method

  aronchick

  committed
  7f11c36

• refactor: Remove duplicate imports and method declarations in aws_compute_operations.go

  aronchick

  committed
  d9f5a78

• refactor: Update AWS compute operations with type and method adjustments

  aronchick

  committed
  94a4e1d

• refactor: Add LiveEC2Client implementation with AWS EC2 methods

  aronchick

  committed
  3f1efd2

• refactor: Consolidate AWS EC2 client implementation into single file

  aronchick

  committed
  29ca5fb

• refactor: Update EC2 client creation with config loading and interface type

  aronchick

  committed
  843a712

• feat: Add DeleteSecurityGroup method to EC2Clienter interface and LiveEC2Client

  aronchick

  committed
  e88b8da

• feat: Add DeleteSubnet method and fix WaitUntilInstanceRunning and CreateVM return types

  aronchick

  committed
  2670b32

• feat: Uncomment security group methods in AWS EC2 client interface

  aronchick

  committed
  a134173

• refactor: Remove unnecessary whitespace in AWSProvider struct

  aronchick

  committed
  7659f13

• refactor: Move WaitUntilInstanceRunning from EC2Clienter to AWSProvider

  aronchick

  committed
  1198696

• refactor: Remove empty EC2 client implementation file

  aronchick

  committed
  16f5ebe

• feat: Implement LiveEC2Client with full EC2Clienter interface methods

  aronchick

  committed
  fca8121

• feat: Add CreateSecurityGroup method to LiveEC2Client

  aronchick

  committed
  72cc0bf

• refactor: Remove WaitUntilInstanceRunning method from LiveEC2Client

  aronchick

  committed
  2e6e210

• fix: Add missing EC2 client methods to implement interface

  aronchick

  committed
  7ffab0e

• refactor: Update SSH configuration to use private key material instead of path

  aronchick

  committed
  b3076e2

• feat: Improve GCP SSH connection resilience with exponential backoff

  aronchick

  committed
  c11d127

• test: Fix SSH mocking in GCP integration test

  aronchick

  committed
  451d79f

• fix: Improve SSH mocking and timeout handling in GCP integration tests

  aronchick

  committed
  88a5804

Commits on Nov 11, 2024

• feat: Add NewAWSProviderFunc for easier provider instantiation

  aronchick

  committed
  bf48b5c

• fix: Add VPC limit handling and cleanup for AWS integration tests

  aronchick

  committed
  fd24524

• feat: Add AWS deployment support to integration test suite

  aronchick

  committed
  1a66f02

• test: Add comprehensive AWS EC2 client mocking for infrastructure creation

  aronchick

  committed
  2251397

• refactor: Improve EC2Clienter interface method signatures for readability

  aronchick

  committed
  651d87b

• refactor: Add DeleteVpc and DescribeRouteTables methods to EC2Clienter interface

  aronchick

  committed
  3f392fb

• refactor: Remove unused DescribeRouteTables method from AWS compute operations

  aronchick

  committed
  84ff5ed

• refactor: Add mocks for AWS networking operations in integration tests

  aronchick

  committed
  94a3d3e

• refactor: Remove duplicate AWS EC2 client method implementations

  aronchick

  committed
  de2e925

• tests pass for AWS

  aronchick

  committed
  7ccd7c1

• fix: Add default AMI fallback for AWS VM deployment

  aronchick

  committed
  c4d2167

• fix: Update Azure mock to match dynamic deployment names

  aronchick

  committed
  e86f08a

• feat: Save AWS VPC ID to config file after creation

  aronchick

  committed
  3530c6a

• feat: Add VPC ID config saving with test support in AWS provider

  aronchick

  committed
  af6a97d

• feat: Add CreateVpc method to AWS provider for VPC creation

  aronchick

  committed
  8707377

• feat: Save AWS VPC ID to config file immediately after creation

  aronchick

  committed
  15b3ed6

• refactor: Update ec2 types import and references in provider_test.go

  aronchick

  committed
  157b931

• refactor: Update test config handling to use CLI-specified config file

  aronchick

  committed
  b8f8ae3

• refactor: Simplify viper config setup in GCP integration test

  aronchick

  committed
  8b0eabb

• refactor: Remove tempConfigFile references in GCP integration test

  aronchick

  committed
  e0e0a95

• refactor: Update import statements and remove unused MockEC2Client struct

  aronchick

  committed
  cd6aa29

• adding improved testing code

  aronchick

  committed
  af9962b

• feat: Improve config file handling for AWS deployment creation

  aronchick

  committed
  9e99236

• refactor: Update deployment config writing to use direct struct fields

  aronchick

  committed
  fd662bc

• chore: Add config flag to AWS create deployment command

  aronchick

  committed
  8caaaa8

• refactor: Add detailed network connectivity logging for AWS provider

  aronchick

  committed
  9782bb7

• feat: Add detailed resource state tracking and display updates during AWS infrastructure provisioning

  aronchick

  committed
  62e6dcf

• test: Add comprehensive tests for AWS provider resource tracking and display updates

  aronchick

  committed
  57b7f3b

• tests passing

  aronchick

  committed
  b0d8357

• feat: Add detailed debug logging for network infrastructure provisioning

  aronchick

  committed
  ed22aaa

• refactor: Simplify logging and improve log message formatting in AWS provider

  aronchick

  committed
  65ed30c

• refactor: Simplify AWS deployment config and update VPC ID immediately

  aronchick

  committed
  76eb06e

• refactor: Simplify VPC config saving with inline model declaration

  aronchick

  committed
  56e6ea1

• feat: Increase update queue size and add detailed network debugging

  aronchick

  committed
  06e38dd

• refactor: Fix route state logging in AWS provider test

  aronchick

  committed
  d6d8cea

• feat: Enhance AWS infrastructure creation with multi-AZ subnets and internet gateway

  aronchick

  committed
  c7629d7

• feat: Add dynamic Ubuntu AMI lookup for AWS deployments

  aronchick

  committed
  ea5d0a3

• fix: Improve update queue processing and error handling in AWS provider

  aronchick

  committed
  5cf0102

• refactor: Fix resource polling and logging in AWS deployment

  aronchick

  committed
  9f33b34

• refactor: Remove unused logger and simplify resource polling error handling

  aronchick

  committed
  bfb23ff

• refactor: Modify startResourcePolling to return error

  aronchick

  committed
  2643f59

• fix: Improve Ubuntu AMI lookup with better filtering and logging

  aronchick

  committed
  1afcff0

• refactor: Improve deployment destroy error handling and messaging

  aronchick

  committed
  242adbe

• refactor: Implement comprehensive VPC deletion and config handling for AWS destroy

  aronchick

  committed
  b775fb1

• refactor: Simplify VPC deletion by leveraging AWS automatic resource cleanup

  aronchick

  committed
  d2a2b81

• feat: Add region-specific AMI lookup for AWS VM deployments

  aronchick

  committed
  60ba986

• refactor: Improve logging formatting in AWS provider test suite

  aronchick

  committed
  6c28154

• fix: Update AWS provider to support region-specific AMI lookup

  aronchick

  committed
  4d3f5f8

• refactor: Update resource polling and VM deployment error handling

  aronchick

  committed
  602cb4f

• fix: Pass region-specific AMI IDs to DeployVMsInParallel method

  aronchick

  committed
  7c8b1c2

• refactor: Update AWS provider to fix AMI lookup and deployment method signature

  aronchick

  committed
  ef3901d

• tests pass on merge

  aronchick

  committed
  96d1515

Commits on Nov 12, 2024

• feat: Add placeholder GetUbuntuAMIForRegion function for AWS

  aronchick

  committed
  99c53ca

• feat: Add function to retrieve latest Ubuntu AMI dynamically from AWS

  aronchick

  committed
  ebd883e

• added new ami functions

  aronchick

  committed
  7474cb2

Commits on Nov 27, 2024

• merge from main

  aronchick

  committed
  fd8e3f8

• finished merge from main

  aronchick

  committed
  8ce3e97

• tests pass

  aronchick

  committed
  e5df82b

• feat: Add security group creation with allowed ports for AWS infrastructure

  aronchick

  committed
  24e9521

• feat: Improve AWS deployment cleanup and VPC deletion logic

  aronchick

  committed
  455510e

• test: fix AWS provider test mocking configuration

  aronchick

  committed
  d76ab57

Commits on Nov 28, 2024

• feat: Add security group mocks to AWS infrastructure creation test

  aronchick

  committed
  1af5499

• The changes look good. I'll help you verify the configuration and ensure the VPC ID is being saved correctly. Here are a few steps we can take:

  Show description for 3ceae52
  aronchick

  committed
  3ceae52

• refactor: Improve AWS VPC cleanup and config management in Destroy method

  aronchick

  committed
  1f8f994

• feat: Add security group methods to EC2Clienter interface

  aronchick

  committed
  2b17e76

• fix: Add DescribeSubnets method to EC2Clienter interface and filter deployments with empty VPC IDs

  aronchick

  committed
  7716c40

• adding testing for aws destroy

  aronchick

  committed
  21c6e91

Commits on Nov 29, 2024

• tests passing, vpc_id being removed

  aronchick

  committed
  7d6be55

• merge from main

  aronchick

  committed
  441288a

• feat: Add support for specifying AWS key pair name via configuration

  aronchick

  committed
  3fdb6ec

• feat: Remove Viper dependency and add SSH key import for AWS provider

  aronchick

  committed
  8f19767

• feat: Enhance SSH key pair generation with unique names and timestamps

  aronchick

  committed
  1389f8e

• feat: Add random seed initialization in AWS provider init function

  aronchick

  committed
  46e74e7

• refactor: Remove duplicate imports after init() function

  aronchick

  committed
  39fbdb2

• refactor: Replace AWS key pair import with user data SSH key injection

  aronchick

  committed
  243b609

• adding ssh-user and public key to user data for aws

  aronchick

  committed
  4c135f7

• refactor: Add detailed logging for AWS VM deployment configuration errors

  aronchick

  committed
  ddb425c

• fix: Correct AWS SDK import path for smithy-go package

  aronchick

  committed
  0ad9210

• refactor: Improve AWS error handling and remove unused imports

  aronchick

  committed
  48d27c5

Commits on Nov 30, 2024

• kicking ci

  aronchick

  committed
  f70a336

• tests passing again

  aronchick

  committed
  4decfc9

• removing large binary

  aronchick

  committed
  1dcaef7

• refactor: Simplify SSH client and session interfaces

  aronchick

  committed
  6ea00bb

• Based on the context and the proposed changes, here's a concise commit message:

  Show description for c32a1f7
  aronchick

  committed
  c32a1f7

• refactor: Remove type declarations from ssh_config.go

  aronchick

  committed
  1379fec

• refactor: Simplify SSH dialer implementation and improve error handling

  aronchick

  committed
  e34ea6c

• refactor: Remove duplicate type declarations in sshutils package

  aronchick

  committed
  e8bf36e

• feat: Add SSH interfaces and utility types for SSH operations

  aronchick

  committed
  f4d3d64

• refactor: Fix SSH utility compilation errors and method implementations

  aronchick

  committed
  e42574c

• fix: Resolve SSH interface and implementation compilation errors

  aronchick

  committed
  28fd28c

• fix: Resolve SSH utils compilation errors and improve code quality

  aronchick

  committed
  95fcfe9

• refactor: Improve SSH file transfer and service installation methods

  aronchick

  committed
  1f969ef

• refactor: Simplify SSH mock client and config generation

  aronchick

  committed
  f04aa7c

• refactor: Improve error handling in SSH utility methods

  aronchick

  committed
  3ece852

• refactor: Enhance SFTP client interface with directory creation and file mode support

  aronchick

  committed
  88d119f

• refactor: Update SFTP and SSH client implementations to resolve compilation errors

  aronchick

  committed
  52db7cd

• refactor: Update SSH service methods to return command output and error

  aronchick

  committed
  d428eec

• fix: Update SSH service methods to return only error

  aronchick

  committed
  7d76b8b

• fix: Update RestartService method signature and implementation

  aronchick

  committed
  1180f74

• refactor: Update mock SSH service methods to return output string

  aronchick

  committed
  5a87e36

• test: Update mock RestartService calls to match new signature

  aronchick

  committed
  0b8c74b

• feat: Update RestartService mock expectations across test files

  aronchick

  committed
  e06e757

Commits on Dec 1, 2024

• add testing for ssh config

  aronchick

  committed
  c140957

• refactor: Remove SSHDialer interface and replace with direct SSH dialing functions

  aronchick

  committed
  869bd43

• refactor: Remove SSHDial methods and initialization to break import cycle

  aronchick

  committed
  af1db7e

• moved sshutils into interfaces and mocks

  aronchick

  committed
  5dcdd1c

• fix: Update SSH utils test suite to improve mocking and error handling

  aronchick

  committed
  a13b971

• fix: Refactor InstallSystemdService to use SFTP instead of StdinPipe

  aronchick

  committed
  45c3ec7

• fix: Update SSH utils test to mock GetClient and SFTP client creation

  aronchick

  committed
  ca5c8d0

• Based on the test output and the changes I suggested, here's a concise commit message:

  Show description for 43214e7
  aronchick

  committed
  43214e7

• refactor: Simplify systemd service operations test logic

  aronchick

  committed
  95539dd

• fix: Update SSH utils test suite to improve mocking and test coverage

  Show description for 30ffe4e
  aronchick

  committed
  30ffe4e

Commits on Dec 2, 2024

• test: Update test mock to use predefined Docker output constant

  aronchick

  committed
  749f6cf

• refactor: Clear mock expectations in SetupTest to prevent unexpected mock matches

  aronchick

  committed
  f0fa69e

• fix: Resolve nil pointer dereference in AWS provider EC2 client creation

  aronchick

  committed
  4f8b360

• all tests passing

  aronchick

  committed
  bd8b021

• fix: Add robust error handling for SSH session methods

  aronchick

  committed
  2cffc1f

• refactor: Remove duplicate SSH session method implementations

  aronchick

  committed
  497cd73

• fix: Add Close method to SSHSessionWrapper and remove unused imports

  aronchick

  committed
  fcd2526

• refactor: Move SSH interfaces to pkg/models/interfaces/sshutils

  aronchick

  committed
  2c362c7

• refactor: Remove duplicate SSHClienter interface declaration

  aronchick

  committed
  0179b9c

• fix: Implement SSH wrapper interfaces to resolve build errors

  aronchick

  committed
  03f06f2

• refactor: Remove unused import from ssh_session_wrapper.go

  aronchick

  committed
  eea1549

• refactor: Add SSH client reference to SSHSessionWrapper for improved connection management

  aronchick

  committed
  dd531c8

• refactor: Improve SSH connection logging and error handling

  aronchick

  committed
  5ec0a64

• refactor: Enhance SSH connection logging and error handling for better diagnostics

  aronchick

  committed
  82a8417

• refactor: Improve SSH connection logging and error handling

  aronchick

  committed
  29c5c17

• refactor: Add detailed SSH connection logging and key validation

  aronchick

  committed
  ab281a2

Commits on Dec 3, 2024

• debugged deployment

  aronchick

  committed
  74ec3ed

Commits on Dec 4, 2024

• adding coderabbit status

  aronchick

  committed
  a568ed8

Commits on Dec 9, 2024

• tests passing

  aronchick

  committed
  ec85df8

• merge

  aronchick

  committed
  e440730