MANTA-5503 buckets-mdapi should have batch metadata updates#65
MANTA-5503 buckets-mdapi should have batch metadata updates#65
Conversation
cneira
force-pushed
the
MANTA-5484
branch
2 times, most recently
from
31f930e to
e01c570
Compare
danmcd
left a comment
danmcd
left a comment
There was a problem hiding this comment.
Quick pass 0. Missing MPL 2.0 headers everywhere and a copyright bump that shouldn't be in an otherwise unchanged file.
Pass 1 on Friday.
New Fast RPC endpoint that accepts multiple UpdateObjectPayload items, groups them by vnode, and executes each vnode group within a single PostgreSQL transaction. Objects in the same vnode succeed or fail atomically. === AI-METADATA === AI-Role: code generation Human-Role: code review, implementation guidance, design decisions Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
danmcd
left a comment
danmcd
left a comment
There was a problem hiding this comment.
I'm worried the stated performance bounds in the comments are incorrect.
This is an incremental. Only rpc_handlers.rs remain for me to review.
danmcd
left a comment
danmcd
left a comment
There was a problem hiding this comment.
Pass 1 complete. Will return after blocking comments addressed.
| // When tests run in parallel, each test must share the same | ||
| // RegisteredMetrics to avoid duplicate-registration panics. | ||
| // Uses static mut + Once because Mutex::new is not const fn | ||
| // in Rust 1.40 (the target toolchain on SmartOS). |
There was a problem hiding this comment.
Ahhh, I was looking at the import of Once above and wondering. This answers it mostly. One more question I'd like answered here: what Rust gives us Mutex::new ? A lookup suggests 1.63, please mention that here?
And a non-blocking followup for my own knowledge: What do we need to do to bump our Rust to 1.63 or more? Looks like pkgsrc LTS 24.4 has 1.81 in it, e.g.
There was a problem hiding this comment.
A reminder: you should mention that once we move to or past the 1.63 toolchain we should revisit this.
Fix syntax error (missing dot and semicolon) on the 'which pg_tmp' expect call and replace generic "failed to execute process" messages with descriptive ones that name the command and suggest what to check. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
danmcd
left a comment
danmcd
left a comment
There was a problem hiding this comment.
Missing one unresolved in rpc_handlers.rs .
3 participants
Today, manta-rebalancer does not acknowledge manta-buckets-api objects, which means those are not rebalanced/evacuated to the appropriate storage nodes.
To evacuate an object, the following events must happen:
[This is similar to what was done in ] (https://github.com/cneira/monitor-reef/blob/15383e082579a663ca16a4a4b79e1e46b30e3e21/libs/moray/src/objects.rs#L282)
This PR deals with (1). This new RPC endpoint helps to minimize network round-trips by updating objects via batches. Consumers of this new call will receive an object of type
BatchUpdateObjectsResponse, which contains the failed payload of objects, to be able to be resubmitted again.Current update object functionality in mdapi has added a
sharksfield as optional; this was intentional to not break current consumers of mdapi. If the shark metadata is not present on the request, then the object's shark metadata will not be touched.The only consumer of this feature today will be manta-rebalancer.