Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New Storage Driver: Minio #962

Merged
merged 38 commits into from
Sep 5, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
21eb34f
minio
mstgnz Aug 23, 2023
23de4dd
Fix README
mstgnz Aug 23, 2023
580a711
Fix README
mstgnz Aug 23, 2023
bec8bb6
Merge branch 'main' into main
mstgnz Aug 23, 2023
d3d3a58
Added Conn method to return the Minio instance
mstgnz Aug 23, 2023
7f002b3
go mod tidy
mstgnz Aug 23, 2023
244330f
Added to Minio storage
mstgnz Aug 23, 2023
5b49d36
added to minio gosec and depeddabot and drafter
mstgnz Aug 23, 2023
4493457
minio test
mstgnz Aug 23, 2023
68fe393
minio drafter
mstgnz Aug 23, 2023
dab578a
gosec
mstgnz Aug 23, 2023
d62aeb5
Merge branch 'main' into main
mstgnz Aug 23, 2023
b3fe37d
change go version
mstgnz Aug 23, 2023
b2636aa
Merge branch 'main' of github.com:mstgnz/fiber-storage
mstgnz Aug 23, 2023
b597a36
The stretchr/testify package was used for testing
mstgnz Aug 23, 2023
dda7fcf
The Reset flag added to New flag
mstgnz Aug 23, 2023
fd63105
min go version supported added to README
mstgnz Aug 23, 2023
cd33dc9
test fixed
mstgnz Aug 23, 2023
d64fa18
fixed test
mstgnz Aug 23, 2023
4c8f4dc
fix minio test
mstgnz Aug 23, 2023
0ccc5d3
UPDATE README
mstgnz Aug 23, 2023
ca32702
Update README
mstgnz Aug 25, 2023
949c597
definitions added
mstgnz Aug 25, 2023
3d59844
updated test for key not in get requests
mstgnz Aug 25, 2023
c49bca7
change to default bucket name - fiber-bucket
mstgnz Aug 26, 2023
e0523db
update comment for CheckBucket
mstgnz Aug 26, 2023
cb6a705
Update README
mstgnz Aug 26, 2023
cdad223
update comment
mstgnz Aug 26, 2023
67218b0
bucket check control removed and update test
mstgnz Aug 26, 2023
1887a32
update credentials key name
mstgnz Aug 29, 2023
edf7bf2
added yo bytebufferpool
mstgnz Aug 29, 2023
f4a12a2
Merge branch 'main' into main
mstgnz Aug 29, 2023
79fdc2e
fix get
mstgnz Aug 31, 2023
fee5f98
Merge branch 'main' into main
mstgnz Sep 2, 2023
6eba76b
fix bytebufferpool
mstgnz Sep 3, 2023
2e2f6ca
Merge branch 'main' into main
mstgnz Sep 4, 2023
8400db5
Update test-minio.yml
gaby Sep 4, 2023
b6dbad7
Update test-minio.yml
gaby Sep 4, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,12 @@ updates:
- "🤖 Dependencies"
schedule:
interval: "daily"
- package-ecosystem: "gomod"
directory: "/minio/" # Location of package manifests
labels:
- "🤖 Dependencies"
schedule:
interval: "daily"
- package-ecosystem: "gomod"
directory: "/mongodb/" # Location of package manifests
labels:
Expand Down
50 changes: 50 additions & 0 deletions .github/release-drafter-minio.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
name-template: 'Minio - v$RESOLVED_VERSION'
tag-template: 'minio/v$RESOLVED_VERSION'
tag-prefix: minio/v
include-paths:
- minio
categories:
- title: '❗ Breaking Changes'
labels:
- '❗ BreakingChange'
- title: '🚀 New'
labels:
- '✏️ Feature'
- title: '🧹 Updates'
labels:
- '🧹 Updates'
- '🤖 Dependencies'
- title: '🐛 Fixes'
labels:
- '☢️ Bug'
- title: '📚 Documentation'
labels:
- '📒 Documentation'
change-template: '- $TITLE (#$NUMBER)'
change-title-escapes: '\<*_&' # You can add # and @ to disable mentions, and add ` to disable code blocks.
exclude-contributors:
- dependabot
- dependabot[bot]
version-resolver:
major:
labels:
- 'major'
- '❗ BreakingChange'
minor:
labels:
- 'minor'
- '✏️ Feature'
patch:
labels:
- 'patch'
- '📒 Documentation'
- '☢️ Bug'
- '🤖 Dependencies'
- '🧹 Updates'
default: patch
template: |
$CHANGES

**Full Changelog**: https://github.com/$OWNER/$REPOSITORY/compare/$PREVIOUS_TAG...minio/v$RESOLVED_VERSION

Thank you $CONTRIBUTORS for making this update possible.
4 changes: 4 additions & 0 deletions .github/workflows/gosec.yml
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,10 @@ jobs:
working-directory: ./memory
run: gosec ./...
# -----
- name: Run Gosec (minio)
working-directory: ./minio
run: gosec ./...
# -----
- name: Run Gosec (mongodb)
working-directory: ./mongodb
run: gosec ./...
Expand Down
19 changes: 19 additions & 0 deletions .github/workflows/release-drafter-minio.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
name: Release Drafter Minio
on:
push:
# branches to consider in the event; optional, defaults to all
branches:
- master
- main
paths:
- 'minio/**'
jobs:
draft_release_minio:
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- uses: release-drafter/release-drafter@v5
with:
config-name: release-drafter-minio.yml
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
33 changes: 33 additions & 0 deletions .github/workflows/test-minio.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
on:
push:
branches:
- master
- main
paths:
- 'minio/**'
pull_request:
paths:
- 'minio/**'
name: "Tests Minio"
jobs:
Tests:
runs-on: ubuntu-latest
strategy:
matrix:
go-version:
- 1.19.x
- 1.20.x
- 1.21.x
steps:
- name: Install MinIO
run: |
docker run -d --restart always -p 9000:9000 --name storage-minio -e MINIO_ROOT_USER='minio-user' -e MINIO_ROOT_PASSWORD='minio-password' minio/minio server /data

- name: Fetch Repository
uses: actions/checkout@v3
- name: Install Go
uses: actions/setup-go@v4
with:
go-version: '${{ matrix.go-version }}'
- name: Run Test
run: cd ./minio && go test ./... -v -race
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@ type Storage interface {
- [Etcd](./etcd/README.md) <a href="https://github.com/gofiber/storage/actions?query=workflow%3A%22Tests+Etcd%22"> <img src="https://img.shields.io/github/actions/workflow/status/gofiber/storage/test-etcd.yml?branch=main&label=%F0%9F%A7%AA%20&style=flat&color=75C46B" /> </a>
- [Memcache](./memcache/README.md) <a href="https://github.com/gofiber/storage/actions?query=workflow%3A%22Tests+Memcache%22"> <img src="https://img.shields.io/github/actions/workflow/status/gofiber/storage/test-memcache.yml?branch=main&label=%F0%9F%A7%AA%20&style=flat&color=75C46B" /> </a>
- [Memory](./memory/README.md) <a href="https://github.com/gofiber/storage/actions?query=workflow%3A%22Tests+Local+Storage%22"> <img src="https://img.shields.io/github/actions/workflow/status/gofiber/storage/test-memory.yml?branch=main&label=%F0%9F%A7%AA%20&style=flat&color=75C46B" /> </a>
- [Minio](./minio/README.md) <a href="https://github.com/gofiber/storage/actions?query=workflow%3A%22Tests+Minio%22"> <img src="https://img.shields.io/github/actions/workflow/status/gofiber/storage/test-minio.yml?branch=main&label=%F0%9F%A7%AA%20&style=flat&color=75C46B" /> </a>
- [MongoDB](./mongodb/README.md) <a href="https://github.com/gofiber/storage/actions?query=workflow%3A%22Tests+Mongodb%22"> <img src="https://img.shields.io/github/actions/workflow/status/gofiber/storage/test-mongodb.yml?branch=main&label=%F0%9F%A7%AA%20&style=flat&color=75C46B" /> </a>
- [MSSQL](./mssql/README.md) <a href="https://github.com/gofiber/storage/actions?query=workflow%3A%22Tests+MSSQL%22"> <img src="https://img.shields.io/github/actions/workflow/status/gofiber/storage/test-mssql.yml?branch=main&label=%F0%9F%A7%AA%20&style=flat&color=75C46B" /> </a>
- [MySQL](./mysql/README.md) <a href="https://github.com/gofiber/storage/actions?query=workflow%3A%22Tests+MySQL%22"> <img src="https://img.shields.io/github/actions/workflow/status/gofiber/storage/test-mysql.yml?branch=main&label=%F0%9F%A7%AA%20&style=flat&color=75C46B" /> </a>
Expand Down
126 changes: 126 additions & 0 deletions minio/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
# Minio

A Minio storage driver using [minio/minio-go](https://github.com/minio/minio-go).

**Note: Requires Go 1.19 and above**

gaby marked this conversation as resolved.
Show resolved Hide resolved
### Table of Contents
- [Signatures](#signatures)
- [Installation](#installation)
- [Examples](#examples)
- [Config](#config)
- [Default Config](#default-config)

### Signatures
```go
func New(config ...Config) Storage
func (s *Storage) Get(key string) ([]byte, error)
func (s *Storage) Set(key string, val []byte, exp time.Duration) error
func (s *Storage) Delete(key string) error
func (s *Storage) Reset() error
func (s *Storage) Close() error
func (s *Storage) CheckBucket() error
gaby marked this conversation as resolved.
Show resolved Hide resolved
func (s *Storage) CreateBucket() error
func (s *Storage) RemoveBucket() error
func (s *Storage) Conn() *minio.Client
```
### Installation
Install the Minio implementation:
```bash
go get github.com/gofiber/storage/minio
```
And then run minio on Docker
```bash
docker run -d --restart always -p 9000:9000 -p 9001:9001 --name storage-minio --volume=minio:/var/lib/minio -e MINIO_ROOT_USER='minio-user' -e MINIO_ROOT_PASSWORD='minio-password' minio/minio server --console-address ":9001" /var/lib/minio
```

### Examples
Import the storage package.
```go
import "github.com/gofiber/storage/minio"
```

You can use the following possibilities to create a storage:
```go
// Initialize default config
store := minio.New()

// Initialize custom config
store := minio.New(minio.Config{
Bucket: "fiber-bucket",
Endpoint: "localhost:9000",
Credentials: Credentials{
accessKeyID: "minio-user",
secretAccessKey: "minio-password",
},
})
```

### Config
```go
// Config defines the config for storage.
type Config struct {
// Bucket
// Default fiber-bucket
Bucket string

// Endpoint is a host name or an IP address
Endpoint string

// Region Set this value to override region cache
// Optional
Region string

// Token Set this value to provide x-amz-security-token (AWS S3 specific)
// Optional, Default is false
Token string

// Secure If set to true, https is used instead of http.
// Default is false
Secure bool

// Reset clears any existing keys in existing Bucket
// Optional. Default is false
Reset bool

// Credentials Minio access key and Minio secret key.
// Need to be defined
Credentials Credentials

// GetObjectOptions Options for GET requests specifying additional options like encryption, If-Match
GetObjectOptions minio.GetObjectOptions

// PutObjectOptions
// Allows user to set optional custom metadata, content headers, encryption keys and number of threads for multipart upload operation.
PutObjectOptions minio.PutObjectOptions

// ListObjectsOptions Options per to list objects
ListObjectsOptions minio.ListObjectsOptions

// RemoveObjectOptions Allows user to set options
RemoveObjectOptions minio.RemoveObjectOptions
}
```

### Default Config
The default configuration lacks Bucket, Region, and Endpoint which are all required and must be overwritten:
```go
// ConfigDefault is the default config
var ConfigDefault = Config{
Bucket: "fiber-bucket",
Endpoint: "",
Region: "",
Token: "",
Secure: false,
Reset: false,
Credentials: Credentials{},
GetObjectOptions: minio.GetObjectOptions{},
PutObjectOptions: minio.PutObjectOptions{},
ListObjectsOptions: minio.ListObjectsOptions{},
RemoveObjectOptions: minio.RemoveObjectOptions{},
}
type Credentials struct {
AccessKeyID string
SecretAccessKey string
}
```
88 changes: 88 additions & 0 deletions minio/config.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
package minio

import (
"github.com/minio/minio-go/v7"
)

// Config defines the config for storage.
type Config struct {
// Bucket
// Default fiber-bucket
Bucket string

// Endpoint is a host name or an IP address
Endpoint string

// Region Set this value to override region cache
// Optional
Region string

// Token Set this value to provide x-amz-security-token (AWS S3 specific)
// Optional, Default is false
Token string

// Secure If set to true, https is used instead of http.
// Default is false
Secure bool

// Reset clears any existing keys in existing Bucket
// Optional. Default is false
Reset bool

// Credentials Minio access key and Minio secret key.
// Need to be defined
Credentials Credentials

// GetObjectOptions Options for GET requests specifying additional options like encryption, If-Match
GetObjectOptions minio.GetObjectOptions

// PutObjectOptions
// Allows user to set optional custom metadata, content headers, encryption keys and number of threads for multipart upload operation.
PutObjectOptions minio.PutObjectOptions

// ListObjectsOptions Options per to list objects
ListObjectsOptions minio.ListObjectsOptions

// RemoveObjectOptions Allows user to set options
RemoveObjectOptions minio.RemoveObjectOptions
}

type Credentials struct {
// AccessKeyID is like user-id that uniquely identifies your account.
AccessKeyID string
// SecretAccessKey is the password to your account.
SecretAccessKey string
}

// ConfigDefault is the default config
var ConfigDefault = Config{
Bucket: "fiber-bucket",
Endpoint: "",
Region: "",
Token: "",
Secure: false,
Reset: false,
Credentials: Credentials{},
GetObjectOptions: minio.GetObjectOptions{},
PutObjectOptions: minio.PutObjectOptions{},
ListObjectsOptions: minio.ListObjectsOptions{},
RemoveObjectOptions: minio.RemoveObjectOptions{},
}

// Helper function to set default values
func configDefault(config ...Config) Config {
// Return default config if nothing provided
if len(config) < 1 {
return ConfigDefault
}

// Override default config
cfg := config[0]

// Set default values
if cfg.Bucket == "" {
cfg.Bucket = ConfigDefault.Bucket
}

return cfg
}
31 changes: 31 additions & 0 deletions minio/go.mod
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
module github.com/gofiber/storage/minio

go 1.19

require (
github.com/minio/minio-go/v7 v7.0.63
github.com/stretchr/testify v1.8.4
github.com/valyala/bytebufferpool v1.0.0
)

require (
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/google/uuid v1.3.1 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/klauspost/compress v1.16.7 // indirect
github.com/klauspost/cpuid/v2 v2.2.5 // indirect
github.com/minio/md5-simd v1.1.2 // indirect
github.com/minio/sha256-simd v1.0.1 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/rs/xid v1.5.0 // indirect
github.com/sirupsen/logrus v1.9.3 // indirect
golang.org/x/crypto v0.12.0 // indirect
golang.org/x/net v0.14.0 // indirect
golang.org/x/sys v0.11.0 // indirect
golang.org/x/text v0.12.0 // indirect
gopkg.in/ini.v1 v1.67.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
)
Loading
Loading