Skip to content

Filebase S3 authentication always failed after upgrade to v0.5.3 #895

@mprokopiev

Description

@mprokopiev

Bug Description

Following the #811 (and fix in #812), I upgraded to latest litestream release however the same error keeps appearing (while 0.3.13 is working).

Environment

Litestream version:

v0.5.3

Configuration

litestream.yml
logging:
  level: DEBUG
dbs:
  - path: ${NTFY_AUTH_FILE}
    replica:
      url: s3://litestream/ntfyv5
      endpoint: https://s3.filebase.com

Logs

Log output
# litestream replicate -exec "ntfy serve"
time=2025-12-13T21:19:18.943+02:00 level=INFO msg=litestream version=v0.5.3 level=DEBUG
time=2025-12-13T21:19:18.943+02:00 level=INFO msg="initialized db" path=/var/lib/ntfy/user.db
time=2025-12-13T21:19:18.943+02:00 level=INFO msg="replicating to" type=s3 sync-interval=1s bucket=litestream path=ntfyv5 region="" endpoint=https://s3.filebase.com
time=2025-12-13T21:19:18.943+02:00 level=INFO msg="starting compaction monitor" level=2 interval=5m0s
time=2025-12-13T21:19:18.943+02:00 level=DEBUG msg="db not ready, skipping" level=2 path=/var/lib/ntfy/user.db
time=2025-12-13T21:19:18.943+02:00 level=INFO msg="starting compaction monitor" level=1 interval=30s
time=2025-12-13T21:19:18.944+02:00 level=DEBUG msg="db not ready, skipping" level=1 path=/var/lib/ntfy/user.db
time=2025-12-13T21:19:18.944+02:00 level=INFO msg="starting L0 retention monitor" interval=15s retention=5m0s
time=2025-12-13T21:19:18.944+02:00 level=INFO msg="starting compaction monitor" level=3 interval=1h0m0s
time=2025-12-13T21:19:18.944+02:00 level=INFO msg="starting compaction monitor" level=9 interval=24h0m0s
time=2025-12-13T21:19:18.944+02:00 level=DEBUG msg="db not ready, skipping" level=3 path=/var/lib/ntfy/user.db
time=2025-12-13T21:19:18.944+02:00 level=DEBUG msg="db not ready, skipping" level=9 path=/var/lib/ntfy/user.db
2025/12/13 21:19:19 INFO Listening on :8080[http], ntfy 2.15.0, log level is INFO (tag=startup)
time=2025-12-13T21:19:20.376+02:00 level=DEBUG msg=verify saltMatch=false prevWALOffset=12392
time=2025-12-13T21:19:20.377+02:00 level=DEBUG msg=sync db=user.db txid=0000000000000003 offset=16512 snap=true reason="last page does not exist in last ltx file, wal overwritten by another process"
time=2025-12-13T21:19:20.397+02:00 level=DEBUG msg="db sync" db=user.db status=ok
time=2025-12-13T21:19:20.631+02:00 level=DEBUG msg="replica sync" db=user.db replica=s3 txid=0000000000000003
time=2025-12-13T21:19:20.776+02:00 level=ERROR msg="monitor error" db=user.db replica=s3 error="write ltx file: s3: upload to ntfyv5/0000/0000000000000001-0000000000000001.ltx: operation error S3: PutObject, https response error StatusCode: 403, RequestID: 8f395d73b204d0e957c172af7f2dcb1d, HostID: ZmlsZWJhc2UtNmI5Y2M2OTY5Yy1zaDQ2aA==, api error AccessDenied: Access Denied"
time=2025-12-13T21:19:20.944+02:00 level=DEBUG msg=verify saltMatch=true prevWALOffset=12392
time=2025-12-13T21:19:20.944+02:00 level=DEBUG msg=verify.2 lastPageMatch=true
time=2025-12-13T21:19:20.944+02:00 level=DEBUG msg=sync db=user.db txid=0000000000000004 offset=16512
time=2025-12-13T21:19:21.612+02:00 level=DEBUG msg="replica sync" db=user.db replica=s3 txid=0000000000000003
time=2025-12-13T21:19:21.848+02:00 level=ERROR msg="monitor error" db=user.db replica=s3 error="write ltx file: s3: upload to ntfyv5/0000/0000000000000001-0000000000000001.ltx: operation error S3: PutObject, https response error StatusCode: 403, RequestID: 299ebc65ae03db2b026ab7933d1adb1d, HostID: ZmlsZWJhc2UtNmI5Y2M2OTY5Yy1sdHdsZg==, api error AccessDenied: Access Denied"

The rest of the details are pretty much the same as in the previously reported issue. Env vars with S3 credentails are there (and valid - 0.3.13 can use them):

# env | grep ACCESS
LITESTREAM_ACCESS_KEY_ID=D4xx
LITESTREAM_SECRET_ACCESS_KEY=KiNxxx

I didn't try building the release from commit e40c8a5985ac5223bef4bff124b1a912630809fe initially, when PR was merger (was waiting for the new release). Now I built the binary using above commit and resulted binary reported the same error.

As per the docs, it is recommended to use sign-payload and require-content-md5 as default - false and true. I tried setting those explicitly with no effect (any combination of those options gave the same result).

Am I missing something?

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions