Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add aws-s3 driver (http) #361

Draft
wants to merge 32 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
8efab74
chore(playground): Create a nitro app
becem-gharbi Dec 29, 2023
a4a7669
chore(playground): Create endpoints for testing
becem-gharbi Dec 29, 2023
8e75dda
feat(driver): Create s3 driver starter
becem-gharbi Dec 29, 2023
c8bb29d
chore(playground): Mount s3 driver
becem-gharbi Dec 29, 2023
b91c365
refactor(s3): Add AWS client
becem-gharbi Dec 29, 2023
e940ce1
feat(s3): Implement `getItemRaw` and `setItemRaw`
becem-gharbi Dec 29, 2023
e9b3b6f
feat(s3): Implement `getMeta`
becem-gharbi Dec 29, 2023
880b914
feat(s3): Implement `getKeys`
becem-gharbi Dec 29, 2023
34d1d06
refactor(s3): Create `_getMeta` and `_getKeys` abstraction
becem-gharbi Dec 29, 2023
a5a5f66
refactor(s3): Mark `clear` as not implemented
becem-gharbi Dec 29, 2023
cac2e8f
chore(s3): Add aws doc links
becem-gharbi Dec 29, 2023
6552a95
chore(playground): Set options for driver's methods
becem-gharbi Dec 29, 2023
2594fa8
refactor(s3): No significant change
becem-gharbi Jan 2, 2024
ca3e9a6
chore: Add `playground` to gitignore
becem-gharbi Jan 3, 2024
4dca7ca
refactor(s3): Create `_getItemRaw` and `_setItemRaw` abstraction
becem-gharbi Jan 3, 2024
8accaf2
feat(s3): Implement `getItem` and `setItem`
becem-gharbi Jan 3, 2024
d1f1ce1
types(s3): Change `headers` type to the same as `http` driver
becem-gharbi Jan 3, 2024
15be314
refactor(s3): No significant change
becem-gharbi Jan 3, 2024
a50fa5c
feat(s3): Implement `clear`
becem-gharbi Jan 3, 2024
69ca7c8
refactor(s3): Remove extra arguments
becem-gharbi Jan 3, 2024
4807d79
fix(s3): Fallback to remove single items if `accountId` option not pr…
becem-gharbi Jan 6, 2024
b585542
docs: Add s3 driver documentation
becem-gharbi Jan 6, 2024
6958320
perf(s3): Lazy load `AwsClient` instance
becem-gharbi Jan 6, 2024
4c87924
types(s3): Fix type issues
becem-gharbi Jan 6, 2024
7b24a35
test: Add s3 driver test
becem-gharbi Jan 6, 2024
67bc19b
fix(s3): Properly resolve `awsUrl`
becem-gharbi Jan 6, 2024
381c40d
refactor(s3): No significant change
becem-gharbi Jan 6, 2024
8b3a948
refactor(s3): No significant change
becem-gharbi Jan 14, 2024
8e86f0a
fix(s3): Include query params in signature
becem-gharbi Jan 14, 2024
73bf8c0
fix(s3): Remove trailing slash on key normalization
becem-gharbi Jan 14, 2024
967a971
fix(s3): Internally set `Content-Length` on `setItem`
becem-gharbi Jan 16, 2024
6540535
docs(s3): Mention required headers on `setItemRaw`
becem-gharbi Jan 16, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -9,4 +9,6 @@ tmp
/test.*
__*
.vercel
.netlify
.netlify
playground
.env
37 changes: 37 additions & 0 deletions docs/content/6.drivers/s3.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# S3

This driver allows data storage to Amazon S3 compatible providers. The data can be KV for document storage or Binary for file storage. It implements a direct HTTP client for a lightweight bundle size and compatibility with edge environments such as Cloudflare Workers and Netlify Edge Functions.

#### Setup

To start please make sure to create a Bucket and an access key with permissions for Object Read & Write. This key will be used for the following object-level operations [HeadObject](https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html), [ListObjectsV2](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html), [GetObject](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html), [PutObject](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html), [DeleteObject](https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObject.html), [DeleteObjects](https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjects.html).

Then please make sure to set the driver's options.

```ts
import { createStorage } from "unstorage";
import s3Driver from "unstorage/drivers/s3";

const storage = createStorage({
driver: s3Driver({
accessKeyId: "", // The client id or access key id
secretAccessKey: "", // The client secret or access key secret
accountId: "", // The account ID (optional)
bucket: "", // The bucket name
endpoint: "", // The bucket endpoint
region: "", // The bucket region
}),
});
```

#### Usage

This driver supports the following methods

- `getItem`/`getItemRaw` To retreive KV or Binary data respectively. Note that the internal response headers can be retreived from `headers` option.
- `setItem`/`setItemRaw` To upload KV or Binary data respectively. Note that the internal request headers can be extended by `headers` option. On `setItemRaw` it's recommended to set `Content-Length` and `Content-Type` headers as they may not be automatically resolved by the provider.
- `removeItem` To delete a single Object.
- `clear` To delete multiple Objects. Note that if `accountId` config option is provided then **DeleteObjects** operation is performed. Otherwise multiple **DeleteObject** operations are performed. This is because **DeleteObjects** operation may not be supported by the provider.
- `getMeta` To retreive the user-defined metadata. Note that `setMeta` is not supported because metadata can only be set on upload (`setItem`/`setItemRaw`) via `meta` option.
- `getKeys` Gets the keys of objects in a specific folder.
- `hasItem` Checks if Object exists.
7 changes: 6 additions & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -44,16 +44,19 @@
},
"dependencies": {
"anymatch": "^3.1.3",
"aws4fetch": "^1.0.17",
"chokidar": "^3.5.3",
"destr": "^2.0.2",
"h3": "^1.9.0",
"ioredis": "^5.3.2",
"jstoxml": "^3.2.10",
"listhen": "^1.5.5",
"lru-cache": "^10.1.0",
"mri": "^1.2.0",
"node-fetch-native": "^1.4.1",
"ofetch": "^1.3.3",
"ufo": "^1.3.2"
"ufo": "^1.3.2",
"xml2js": "^0.6.2"
},
"devDependencies": {
"@azure/app-configuration": "^1.5.0",
Expand All @@ -68,8 +71,10 @@
"@planetscale/database": "^1.13.0",
"@types/ioredis-mock": "^8.2.5",
"@types/jsdom": "^21.1.6",
"@types/jstoxml": "^2.0.4",
"@types/mri": "^1.1.5",
"@types/node": "^20.10.5",
"@types/xml2js": "^0.4.14",
"@upstash/redis": "^1.25.2",
"@vercel/kv": "^0.2.4",
"@vitejs/plugin-vue": "^4.5.2",
Expand Down
36 changes: 33 additions & 3 deletions pnpm-lock.yaml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading