Skip to content

Commit

Permalink
Replace providers, rewrite steps (#3295)
Browse files Browse the repository at this point in the history
Replace paid/discontinued providers with free/actual providers. Replace
specific steps for each provider with basic steps, applicable for all

Co-authored-by: Michael Hoepler <[email protected]>
  • Loading branch information
AlekseyKorzik and MichaelHoepler authored Jan 25, 2024
1 parent 4a2eb7e commit cb319de
Showing 1 changed file with 5 additions and 55 deletions.
60 changes: 5 additions & 55 deletions docs/docs/setting-up/data-ingestion/pin.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,63 +5,13 @@ description: "How to pin data to public storage"
---
# Pinning Data

If you have data that you want to make available to your Bacalhau jobs (or other people), you can pin it using a pinning service like Web3.Storage, Pinata, etc. Pinning services store data on behalf of users. The pinning provider is essentially guaranteeing that your data will be available if someone knows the CID. Most pinning services offer you a free tier, so you can try them out without spending any money.
If you have data that you want to make available to your Bacalhau jobs (or other people), you can pin it using a pinning service like Pinata, NFT.Storage, Thirdweb, etc. Pinning services store data on behalf of users. The pinning provider is essentially guaranteeing that your data will be available if someone knows the CID. Most pinning services offer you a free tier, so you can try them out without spending any money.

## Web3.Storage

This example will demonstrate how to pin data using Web3.Storage. Web3.Storage is a pinning service that is built on top of IPFS and Filecoin. It is free to use for small amounts of data and has a generous free tier. You can find more information about Web3.Storage [here](https://web3.storage/).
## Basic steps

- First you need to create an [account](https://web3.storage/login/) (if you don't have one already).
- Next, sign in and browse to the [Create API Key](https://web3.storage/tokens/?create=true) page. Follow the instructions to create an API key. Once created, you will need to copy the API key to your clipboard.
To use a pinning service, you will almost always need to create an account. After registration, you get an API token, which is necessary to control and access the files. Then you need to upload files - usually services provide a web interface, CLI and code samples for integration into your application. Once you upload the files you will get its CID, which looks like this: `QmUyUg8en7G6RVL5uhyoLBxSWFgRMdMraCRWFcDdXKWEL9`. Now you can access pinned data from the jobs via this CID.

### Ways to pin using web3.storage

1. **Pin a local file using their test client**: To test that your API key is working, use [web3.storage's test client](https://bafybeic5r5yxjh5xpmeczfp34ysrjcoa66pllnjgffahopzrl5yhex7d7i.ipfs.dweb.link/).

You can now see (or upload) your file via the web3.storage [account page](https://web3.storage/account/).

:::warning
Note that you shouldn't share your API key with anyone. Delete this API key once you have finished with this example.
:::info
Data source can be specified via `--input` flag, see the [CLI Guide](../../dev/cli-reference/all-flags.md#docker-run) for more details
:::

2. **Pin a local file via curl**: You can also pin a file via curl. Please view the [API documentation](https://web3.storage/docs/reference/http-api/) to see all available commands. This example submits a single file to be pinned.

```bash
export TOKEN=YOUR_API_KEY
echo hello world > foo.txt
curl -X POST https://api.web3.storage/upload -H "Authorization: Bearer ${TOKEN}" -H "X-NAME: foo.txt" -d @foo.txt
```

3. **Pin multiple local files via Node.JS**: Web3.Storage has a [node.js library](https://web3.storage/docs/reference/js-client-library/) to interact with their API. The following example requires node.js to be installed. The following code uses a docker container. The javascript code is located on [their website](https://web3.storage/docs/intro/#create-the-upload-script) or on [github](https://github.com/bacalhau-project/examples/blob/main/data-ingestion/nodejs/put-files.js).

First, create some files to upload.

```python
%%writefile nodejs/test1.txt
First test file
```

Then run the following command, which uses the environmental variable `TOKEN` to authenticate with the API.

```bash
export TOKEN=YOUR_API_KEY
docker run --rm --env TOKEN=$TOKEN -v $PWD/nodejs:/nodejs node:18-alpine ash -c 'cd /nodejs && npm install && node put-files.js --token=$TOKEN test1.txt test2.txt'
```

The response will return the CID of the file, which can now be used as an input to Bacalhau.

4. **Pin a file from a URL via Curl**: You can use curl to download a file and then re-upload it to web3.storage. For example:

```bash
export TOKEN=YOUR_API_KEY
curl -o train-images-idx3-ubyte.gz http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz
curl -X POST https://api.web3.storage/upload -H "Authorization: Bearer ${TOKEN}" -H "X-NAME: train-images-idx3-ubyte.gz" -d @train-images-idx3-ubyte.gz
```


5. **Pin a file from a URL via Node.JS**: You can combine the node.js example above with a `wget` to then upload it to web3.storage.

```bash
docker run --rm --env TOKEN=$TOKEN -v $PWD/nodejs:/nodejs node:18-alpine ash -c 'cd /nodejs && wget http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz && npm install && node put-files.js --token=$TOKEN train-images-idx3-ubyte.gz'
```

0 comments on commit cb319de

Please sign in to comment.