feat: archivistactl import #358
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #358 +/- ##
==========================================
- Coverage 82.40% 1.64% -80.76%
==========================================
Files 10 121 +111
Lines 358 28956 +28598
==========================================
+ Hits 295 477 +182
- Misses 43 28422 +28379
- Partials 20 57 +37 ☔ View full report in Codecov by Sentry. |
kairoaraujo
force-pushed
the
feat/import
branch
2 times, most recently
from
d8ff65a to
fe986e1
Compare
Introduces a new `archivistactl` import It allows Archivista users to import DSSE Envelopes directly to the Archivista database. The feature allows direct import, which can help importing huge amount of data as it use concurrency (go routines) to process. Signed-off-by: Kairo Araujo <[email protected]>
|
Definitely like this -- there's been a few times where I've hacked this in but never actually got it in a mergable state. I think one thing we may eventually want to do is also support smarter bulk loading on the server side. We can do some better SQL optimizations when we know we're going to load in a bunch of data, but we can worry about that later |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it
Introduces a new
archivistactlimportIt allows Archivista users to import DSSE Envelopes directly to the Archivista database.
The feature allows direct import, which can help import huge amounts of data as it uses concurrency (go routines) to process.
Performance examples
Importing 2100 new DSSE Envelopes
Using default:
--max-concurrent 3Using
--max-concurrent 10Which issue(s) this PR fixes (optional)
(optional, using
fixes #<issue number>(, fixes #<issue_number>, ...)format, will close the issue(s) when the PR gets merged)*Related #319
Acceptance Criteria Met
Special notes for your reviewer:
TODO:
--exist-firstfail in the first import error instead of skippingI can add the TODO as new commits or as new PRs.