Need Assistance: Uploading Large Datasets(> 10M) via API/SDK

[anata404]

Dear Ultralytics Community,

I am integrating the Ultralytics HUB into my data pipeline using the hub_sdk (version 0.0.17). My goal is to upload large datasets programmatically, but I've encountered a limitation with the current implementation of the upload_dataset method.

Issue Overview

While using the following code snippet:

from hub_sdk import HUBClient

credentials = {"api_key": "<YOUR-API-KEY>"}
client = HUBClient(credentials)

# Select the dataset
dataset = client.dataset("<Dataset ID>")  # Substitute with the real dataset ID

# Upload the dataset file
dataset.upload_dataset(file="<Dataset File>")  # Ensure the correct file path is specified
print("Dataset has been uploaded.")

I discovered that the upload_dataset method works, but it imposes a 10MB file size limit, which prevents me from uploading larger datasets directly.

Steps Taken So Far

1. Web UI Uploads: I successfully uploaded large datasets using the Ultralytics HUB Web Interface, but my use case requires automation for programmatic uploads.
2. API Investigation: I attempted to identify API endpoints that could support pre-signed URL generation for uploading large datasets, but I couldn’t locate sufficient documentation.
3. I opened an Issue #963 for potential workarounds or relevant updates, but no solution was found.

Request for Assistance

1. Workaround for Large Files: Is there a recommended approach for uploading datasets larger than 10MB using the hub_sdk? For example:

• Splitting datasets into smaller chunks programmatically.
• Using a direct API method for pre-signed URL uploads.

2. API Documentation: Are there any detailed guides or examples for utilizing the API directly for pre-signed URL generation?
3. Future Updates: Are there plans to enhance the upload_dataset method in the hub_sdk to support larger files or provide a seamless programmatic alternative?

Why This Matters

This enhancement or workaround would significantly improve the SDK's usability for automating workflows, especially for users managing large datasets in production environments.

Thank you in advance for your guidance and support. I’m eager to hear your suggestions or recommendations!

[anata404]

Closed for #971