Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: import from a remote source #118

Merged
merged 11 commits into from
May 4, 2023
Merged

feat: import from a remote source #118

merged 11 commits into from
May 4, 2023

Conversation

alvin-reyes
Copy link
Collaborator

@alvin-reyes alvin-reyes commented May 2, 2023

Changes

  • introduce transfer_parameters to allow import deals to pull from a remote source when making a deal
  • introduce batch imports to asynchronously batch import deal calls.

Transfer Parameters

This allows users to specify a remote source for import deals.

  • This PR introduces a new metadata field called transfer_parameters that's only valid/allowed for import type deals
  • This new field will enable the deal-maker to specify a URL source of file.

Sample below:

When the transfer_parameters is specified, delta will use the URL as the source of the import.

    "cid": "bafybeicgdjdvwes3e5aaicqljrlv6hpdfsducknrjvsq66d4gsvepolk6y",
    "miner": "f01963614",
    "piece_commitment": {
        "piece_cid": "baga6ea4seaqdgtooshkmlmbu4hesd5asp7g3p4yd7s4tjmrbkx676ba5mlu2kcy",
        "padded_piece_size": 2097152
    },
    "transfer_parameters": {
        "url":"https://ipfs.io/ipfs/bafybeicgdjdvwes3e5aaicqljrlv6hpdfsducknrjvsq66d4gsvepolk6y"
    },
    "connection_mode": "import",
    "size": 1157859,
    "remove_unsealed_copy": false,
    "skip_ipni_announce": true,
    "duration_in_days": 521,
    "start_epoch_in_days": 7
}

Batch Imports

Introduced the following new endpoints

  • /deal/batch/imports - this takes an array of import deal metadata and creates a batch job that runs async. The user gets a batch id that can be used to check the status of the job and it's associated content.
  • /open/stats/batch/imports/:batchId - this is to check the status of the batch and the contents associated to it.

@alvin-reyes alvin-reyes self-assigned this May 2, 2023
@alvin-reyes alvin-reyes linked an issue May 2, 2023 that may be closed by this pull request
@alvin-reyes alvin-reyes marked this pull request as ready for review May 3, 2023 02:22
@alvin-reyes alvin-reyes merged commit be4cf72 into main May 4, 2023
@alvin-reyes alvin-reyes deleted the auto-import branch May 4, 2023 15:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Download URL for Import deals
1 participant