[abctl] Storage not getting created on windows #40641

Sivivatu · 2024-07-01T08:29:23Z

What happened?

When Installing abctl on Windows (with WSL) the installer fails with the error:

  ERROR   Failed to install airbyte/airbyte Helm Chart
  ERROR   Unable to install Airbyte locally
  ERROR   could not install airbyte chart: could not install helm: failed pre-install: 1 error occurred:
                * pod airbyte-abctl-airbyte-bootloader failed

this is being run from in the linux subsystem (WSL2) with docker installed.

full Error output:

 fatpunk    abctl-v0.6.0-linux-amd64   36ms⠀  
 $ sudo ./abctl local install                                                        bash  79  20:48:03 
  INFO    Using Kubernetes provider:
            Provider: kind
            Kubeconfig: /root/.airbyte/abctl/abctl.kubeconfig
            Context: kind-airbyte-abctl
 SUCCESS  Found Docker installation: version 26.1.4
 SUCCESS  Port 8000 appears to be available
  INFO    No existing cluster found, cluster 'airbyte-abctl' will be created
 SUCCESS  Cluster 'airbyte-abctl' created
  INFO    Namespace 'airbyte-abctl' created
  INFO    Persistent volume 'airbyte-minio-pv' created
  INFO    Persistent volume 'airbyte-volume-db' created
  INFO    Persistent volume claim 'airbyte-minio-pv-claim-airbyte-minio-0' created
  INFO    Persistent volume claim 'airbyte-volume-db-airbyte-db-0' created
 WARNING  Encountered an issue deploying Airbyte:
            Pod: airbyte-db-0.17dd42dfa14c67f5
            Reason: BackOff
            Message: Back-off restarting failed container airbyte-db-container in pod airbyte-db-0_airbyte-abctl(12ddb005-440d-4687-934f-0fb9179f4a25)
            Count: 6
            Logs: mkdir: can't create directory '/var/lib/postgresql/data/pgdata': Permission denied
 WARNING  Encountered an issue deploying Airbyte:
            Pod: airbyte-minio-0.17dd42de3a833a83
            Reason: BackOff
            Message: Back-off restarting failed container airbyte-minio in pod airbyte-minio-0_airbyte-abctl(37d9f725-b4db-40a3-879e-90ba5477f59b)
            Count: 6
            Logs: API: SYSTEM()
          Time: 19:49:57 UTC 06/28/2024
          Error: unable to rename (/storage/.minio.sys/tmp -> /storage/.minio.sys/tmp-old/fe0cf3cd-8cc7-4e6c-b4fc-c497812698d1) file access denied, drive may be faulty please investigate (*fmt.wrapError)
                 6: internal/logger/logger.go:258:logger.LogIf()
                 5: cmd/prepare-storage.go:89:cmd.bgFormatErasureCleanupTmp()
                 4: cmd/xl-storage.go:263:cmd.newXLStorage()
                 3: cmd/object-api-common.go:63:cmd.newStorageAPI()
                 2: cmd/format-erasure.go:673:cmd.initStorageDisksWithErrors.func1()
                 1: github.com/minio/pkg/[email protected]/sync/errgroup/errgroup.go:123:errgroup.(*Group).Go.func1()

          API: SYSTEM()
          Time: 19:49:57 UTC 06/28/2024
          Error: unable to create (/storage/.minio.sys/tmp) file access denied, drive may be faulty please investigate (*fmt.wrapError)
                 6: internal/logger/logger.go:258:logger.LogIf()
                 5: cmd/prepare-storage.go:96:cmd.bgFormatErasureCleanupTmp()
                 4: cmd/xl-storage.go:263:cmd.newXLStorage()
                 3: cmd/object-api-common.go:63:cmd.newStorageAPI()
                 2: cmd/format-erasure.go:673:cmd.initStorageDisksWithErrors.func1()
                 1: github.com/minio/pkg/[email protected]/sync/errgroup/errgroup.go:123:errgroup.(*Group).Go.func1()
          ERROR Unable to use the drive /storage: file access denied: Invalid arguments specified
 WARNING  Encountered an issue deploying Airbyte:
            Pod: airbyte-db-0.17dd42dfa14c67f5
            Reason: BackOff
            Message: Back-off restarting failed container airbyte-db-container in pod airbyte-db-0_airbyte-abctl(12ddb005-440d-4687-934f-0fb9179f4a25)
            Count: 7
            Logs: mkdir: can't create directory '/var/lib/postgresql/data/pgdata': Permission denied
 WARNING  Encountered an issue deploying Airbyte:
            Pod: airbyte-minio-0.17dd42de3a833a83
            Reason: BackOff
            Message: Back-off restarting failed container airbyte-minio in pod airbyte-minio-0_airbyte-abctl(37d9f725-b4db-40a3-879e-90ba5477f59b)
            Count: 7
            Logs: API: SYSTEM()
          Time: 19:50:41 UTC 06/28/2024
          Error: unable to rename (/storage/.minio.sys/tmp -> /storage/.minio.sys/tmp-old/4f8ec462-1f1f-489f-8aa7-9b774547e91c) file access denied, drive may be faulty please investigate (*fmt.wrapError)
                 6: internal/logger/logger.go:258:logger.LogIf()
                 5: cmd/prepare-storage.go:89:cmd.bgFormatErasureCleanupTmp()
                 4: cmd/xl-storage.go:263:cmd.newXLStorage()
                 3: cmd/object-api-common.go:63:cmd.newStorageAPI()
                 2: cmd/format-erasure.go:673:cmd.initStorageDisksWithErrors.func1()
                 1: github.com/minio/pkg/[email protected]/sync/errgroup/errgroup.go:123:errgroup.(*Group).Go.func1()

          API: SYSTEM()
          Time: 19:50:41 UTC 06/28/2024
          Error: unable to create (/storage/.minio.sys/tmp) file access denied, drive may be faulty please investigate (*fmt.wrapError)
                 6: internal/logger/logger.go:258:logger.LogIf()
                 5: cmd/prepare-storage.go:96:cmd.bgFormatErasureCleanupTmp()
                 4: cmd/xl-storage.go:263:cmd.newXLStorage()
                 3: cmd/object-api-common.go:63:cmd.newStorageAPI()
                 2: cmd/format-erasure.go:673:cmd.initStorageDisksWithErrors.func1()
                 1: github.com/minio/pkg/[email protected]/sync/errgroup/errgroup.go:123:errgroup.(*Group).Go.func1()
          ERROR Unable to use the drive /storage: file access denied: Invalid arguments specified
 WARNING  Encountered an issue deploying Airbyte:
            Pod: airbyte-db-0.17dd42dfa14c67f5
            Reason: BackOff
            Message: Back-off restarting failed container airbyte-db-container in pod airbyte-db-0_airbyte-abctl(12ddb005-440d-4687-934f-0fb9179f4a25)
            Count: 8
            Logs: mkdir: can't create directory '/var/lib/postgresql/data/pgdata': Permission denied
 WARNING  Encountered an issue deploying Airbyte:
            Pod: airbyte-db-0.17dd42dfa14c67f5
            Reason: BackOff
            Message: Back-off restarting failed container airbyte-db-container in pod airbyte-db-0_airbyte-abctl(12ddb005-440d-4687-934f-0fb9179f4a25)
            Count: 9
            Logs: mkdir: can't create directory '/var/lib/postgresql/data/pgdata': Permission denied
 WARNING  Encountered an issue deploying Airbyte:
            Pod: airbyte-minio-0.17dd42de3a833a83
            Reason: BackOff
            Message: Back-off restarting failed container airbyte-minio in pod airbyte-minio-0_airbyte-abctl(37d9f725-b4db-40a3-879e-90ba5477f59b)
            Count: 8
            Logs: API: SYSTEM()
          Time: 19:50:41 UTC 06/28/2024
          Error: unable to rename (/storage/.minio.sys/tmp -> /storage/.minio.sys/tmp-old/4f8ec462-1f1f-489f-8aa7-9b774547e91c) file access denied, drive may be faulty please investigate (*fmt.wrapError)
                 6: internal/logger/logger.go:258:logger.LogIf()
                 5: cmd/prepare-storage.go:89:cmd.bgFormatErasureCleanupTmp()
                 4: cmd/xl-storage.go:263:cmd.newXLStorage()
                 3: cmd/object-api-common.go:63:cmd.newStorageAPI()
                 2: cmd/format-erasure.go:673:cmd.initStorageDisksWithErrors.func1()
                 1: github.com/minio/pkg/[email protected]/sync/errgroup/errgroup.go:123:errgroup.(*Group).Go.func1()

          API: SYSTEM()
          Time: 19:50:41 UTC 06/28/2024
          Error: unable to create (/storage/.minio.sys/tmp) file access denied, drive may be faulty please investigate (*fmt.wrapError)
                 6: internal/logger/logger.go:258:logger.LogIf()
                 5: cmd/prepare-storage.go:96:cmd.bgFormatErasureCleanupTmp()
                 4: cmd/xl-storage.go:263:cmd.newXLStorage()
                 3: cmd/object-api-common.go:63:cmd.newStorageAPI()
                 2: cmd/format-erasure.go:673:cmd.initStorageDisksWithErrors.func1()
                 1: github.com/minio/pkg/[email protected]/sync/errgroup/errgroup.go:123:errgroup.(*Group).Go.func1()
          ERROR Unable to use the drive /storage: file access denied: Invalid arguments specified
 WARNING  Encountered an issue deploying Airbyte:
            Pod: airbyte-minio-0.17dd42de3a833a83
            Reason: BackOff
            Message: Back-off restarting failed container airbyte-minio in pod airbyte-minio-0_airbyte-abctl(37d9f725-b4db-40a3-879e-90ba5477f59b)
            Count: 9
            Logs: API: SYSTEM()
          Time: 19:50:41 UTC 06/28/2024
          Error: unable to rename (/storage/.minio.sys/tmp -> /storage/.minio.sys/tmp-old/4f8ec462-1f1f-489f-8aa7-9b774547e91c) file access denied, drive may be faulty please investigate (*fmt.wrapError)
                 6: internal/logger/logger.go:258:logger.LogIf()
                 5: cmd/prepare-storage.go:89:cmd.bgFormatErasureCleanupTmp()
                 4: cmd/xl-storage.go:263:cmd.newXLStorage()
                 3: cmd/object-api-common.go:63:cmd.newStorageAPI()
                 2: cmd/format-erasure.go:673:cmd.initStorageDisksWithErrors.func1()
                 1: github.com/minio/pkg/[email protected]/sync/errgroup/errgroup.go:123:errgroup.(*Group).Go.func1()

          API: SYSTEM()
          Time: 19:50:41 UTC 06/28/2024
          Error: unable to create (/storage/.minio.sys/tmp) file access denied, drive may be faulty please investigate (*fmt.wrapError)
                 6: internal/logger/logger.go:258:logger.LogIf()
                 5: cmd/prepare-storage.go:96:cmd.bgFormatErasureCleanupTmp()
                 4: cmd/xl-storage.go:263:cmd.newXLStorage()
                 3: cmd/object-api-common.go:63:cmd.newStorageAPI()
                 2: cmd/format-erasure.go:673:cmd.initStorageDisksWithErrors.func1()
                 1: github.com/minio/pkg/[email protected]/sync/errgroup/errgroup.go:123:errgroup.(*Group).Go.func1()
          ERROR Unable to use the drive /storage: file access denied: Invalid arguments specified
 WARNING  Encountered an issue deploying Airbyte:
            Pod: airbyte-minio-0.17dd42de3a833a83
            Reason: BackOff
            Message: Back-off restarting failed container airbyte-minio in pod airbyte-minio-0_airbyte-abctl(37d9f725-b4db-40a3-879e-90ba5477f59b)
            Count: 21
            Logs: API: SYSTEM()
          Time: 20:27:50 UTC 06/28/2024
          Error: unable to rename (/storage/.minio.sys/tmp -> /storage/.minio.sys/tmp-old/e8d4fe46-d2b3-4685-ba9e-ee93afd548a7) file access denied, drive may be faulty please investigate (*fmt.wrapError)
                 6: internal/logger/logger.go:258:logger.LogIf()
                 5: cmd/prepare-storage.go:89:cmd.bgFormatErasureCleanupTmp()
                 4: cmd/xl-storage.go:263:cmd.newXLStorage()
                 3: cmd/object-api-common.go:63:cmd.newStorageAPI()
                 2: cmd/format-erasure.go:673:cmd.initStorageDisksWithErrors.func1()
                 1: github.com/minio/pkg/[email protected]/sync/errgroup/errgroup.go:123:errgroup.(*Group).Go.func1()

          API: SYSTEM()
          Time: 20:27:50 UTC 06/28/2024
          Error: unable to create (/storage/.minio.sys/tmp) file access denied, drive may be faulty please investigate (*fmt.wrapError)
                 6: internal/logger/logger.go:258:logger.LogIf()
                 5: cmd/prepare-storage.go:96:cmd.bgFormatErasureCleanupTmp()
                 4: cmd/xl-storage.go:263:cmd.newXLStorage()
                 3: cmd/object-api-common.go:63:cmd.newStorageAPI()
                 2: cmd/format-erasure.go:673:cmd.initStorageDisksWithErrors.func1()
                 1: github.com/minio/pkg/[email protected]/sync/errgroup/errgroup.go:123:errgroup.(*Group).Go.func1()
          ERROR Unable to use the drive /storage: file access denied: Invalid arguments specified
 WARNING  Encountered an issue deploying Airbyte:
            Pod: airbyte-db-0.17dd42dfa14c67f5
            Reason: BackOff
            Message: Back-off restarting failed container airbyte-db-container in pod airbyte-db-0_airbyte-abctl(12ddb005-440d-4687-934f-0fb9179f4a25)
            Count: 23
            Logs: mkdir: can't create directory '/var/lib/postgresql/data/pgdata': Permission denied
  ERROR   Failed to install airbyte/airbyte Helm Chart
  ERROR   Unable to install Airbyte locally
  ERROR   could not install airbyte chart: could not install helm: failed pre-install: 1 error occurred:
                * pod airbyte-abctl-airbyte-bootloader failed

What did you expect to happen?

Expected airbyte to install and be accessable via localhost:8000

Abctl Version

attempting to install abctl-v0.6.0-linux-amd64

$ abctl version
# paste output here

Docker Version

 $ docker version
Client:
 Version:           24.0.7
 API version:       1.43
 Go version:        go1.21.1
 Git commit:        24.0.7-0ubuntu2~23.10.1
 Built:             Wed Mar 13 20:05:50 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Desktop
 Engine:
  Version:          26.1.4
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.11
  Git commit:       de5c9cf
  Built:            Wed Jun  5 11:29:22 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.33
  GitCommit:        d2d58213f83a351ca8f528a95fbd145f5654e957
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

OS Version

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here
 $ wmic os get Caption, Version, BuildNumber, OSArchitecture                               pwsh   79  20:25:05 
BuildNumber  Caption                   OSArchitecture  Version
22631        Microsoft Windows 11 Pro  64-bit          10.0.22631

 $ wsl --version                                                                           pwsh   79  09:25:54 
WSL version: 2.1.5.0
Kernel version: 5.15.146.1-2
WSLg version: 1.0.60
MSRDC version: 1.2.5105
Direct3D version: 1.611.1-81528511
DXCore version: 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp
Windows version: 10.0.22631.3810

On Linux:

$ cat /etc/os-release

paste output here

$ cat /etc/os-release   bash  79  09:26:12
PRETTY_NAME="Ubuntu 23.10"
NAME="Ubuntu"
VERSION_ID="23.10"
VERSION="23.10 (Mantic Minotaur)"
VERSION_CODENAME=mantic
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=mantic
LOGO=ubuntu-logo

flobr · 2024-07-01T17:12:47Z

I have exactly the same problem with Windows 10 Pro & Docker Desktop 4.31.1. I tested it with Powershell and cygwin. Both show the same errors.

duncan4b · 2024-07-02T08:06:14Z

same here, same error in windows

flobr · 2024-07-02T08:10:46Z

v0.5.0 of airbyte works (if you ignore the "Message: Readiness probe failed:" messages)

marcosmarxm · 2024-07-03T16:55:06Z

Hello everyone, 👋 thank you for reporting the issue and providing more information. @bgroff can you check this?

burningion · 2024-07-04T13:03:51Z

Just here to make a note, had the same error experience on Ubuntu linux.

Same problem creating the pgdata directory (I had to make it manually and then chown it to my user), followed by the same error above:

Error: unable to create (/storage/.minio.sys/tmp) file access denied, drive may be faulty please investigate (*fmt.wrapError)

stsigounis · 2024-07-10T02:50:42Z

Regarding The first post above

I'm running Windows 10 Pro and I just tried to install abctl-v0.7.1-windows-amd64. The same error still persists, I've included the warning received prior to the error.

WARNING Encountered an issue deploying Airbyte:
Pod: airbyte-db-0.17e0b8e47b1a9d8a
Reason: BackOff
Message: Back-off restarting failed container airbyte-db-container in pod airbyte-db-0_airbyte-abctl(d6f2a0c4-3e55-41e0-9669-391c444110a7)
Count: 23
Logs: chmod: /var/lib/postgresql/data/pgdata: Operation not permitted
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

      The database cluster will be initialized with locale "en_US.utf8".
      The default database encoding has accordingly been set to "UTF8".
      The default text search configuration will be set to "english".

      Data page checksums are disabled.

      initdb: error: could not change permissions of directory "/var/lib/postgresql/data/pgdata": Operation not permitted
      fixing permissions on existing directory /var/lib/postgresql/data/pgdata ...

ERROR Failed to install airbyte/airbyte Helm Chart
ERROR Unable to install Airbyte locally
ERROR could not install airbyte chart: could not install helm: failed pre-install: 1 error occurred:
* pod airbyte-abctl-airbyte-bootloader failed

bgroff · 2024-09-30T05:05:42Z

Airbyte has been unable to reproduce this error. Is this still happening for folks? Going to close this thread as it has been quite, but if you are still experiencing this with the latest version of abctl, please feel free to open this again.

flobr · 2024-09-30T17:40:23Z

Migrating from 0.5.0 failed completely. But installing the new version 0.18.0 from scratch worked like a charm!

wajdifoothill · 2024-12-31T18:50:46Z

same issue here

@bgroff

Sivivatu added area/abctl Issues with the abctl quickstart cli needs-triage type/bug Something isn't working labels Jul 1, 2024

octavia-squidington-iii added team/triage autoteam community team/use and removed team/triage labels Jul 1, 2024

marcosmarxm changed the title ~~Storage not getting created on windows~~ [abctl] Storage not getting created on windows Jul 3, 2024

marcosmarxm added team/deployments and removed needs-triage labels Jul 3, 2024

wkeifenheim mentioned this issue Aug 7, 2024

[abctl] minimum requirements not specified in documentation, gives error when try to use less resources than required #40231

Closed

nataliekwong added the proj/abctl label Sep 12, 2024

bgroff closed this as completed Sep 30, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[abctl] Storage not getting created on windows #40641

[abctl] Storage not getting created on windows #40641

Sivivatu commented Jul 1, 2024 •

edited

Loading

On Linux:

paste output here

flobr commented Jul 1, 2024

duncan4b commented Jul 2, 2024

flobr commented Jul 2, 2024

marcosmarxm commented Jul 3, 2024

burningion commented Jul 4, 2024

stsigounis commented Jul 10, 2024

bgroff commented Sep 30, 2024

flobr commented Sep 30, 2024

wajdifoothill commented Dec 31, 2024

[abctl] Storage not getting created on windows #40641

[abctl] Storage not getting created on windows #40641

Comments

Sivivatu commented Jul 1, 2024 • edited Loading

What happened?

What did you expect to happen?

Abctl Version

Docker Version

OS Version

On Linux:

paste output here

flobr commented Jul 1, 2024

duncan4b commented Jul 2, 2024

flobr commented Jul 2, 2024

marcosmarxm commented Jul 3, 2024

burningion commented Jul 4, 2024

stsigounis commented Jul 10, 2024

bgroff commented Sep 30, 2024

flobr commented Sep 30, 2024

wajdifoothill commented Dec 31, 2024

Sivivatu commented Jul 1, 2024 •

edited

Loading