[WIP][SPARK-53455][CONNECT] Add CloneSession RPC
#52200
Draft
+924
−3
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
vicennial
changed the title
CloneSession RPCCloneSession RPC
hvanhovell
reviewed
Sep 2, 2025
| optional string client_type = 3; | ||
|
|
||
| // (Optional) | ||
| // The session_id for the new cloned session. If not provided, a new UUID will be generated. |
There was a problem hiding this comment.
Is the server allowed to return a different session id if you provide this id?
hvanhovell
reviewed
Sep 2, 2025
| * Create a clone of this Spark Connect session on the server side with a custom session ID. | ||
| * The server-side session is cloned with all its current state (SQL configurations, temporary | ||
| * views, registered functions, catalog state) copied over to a new independent session with | ||
| * the specified session ID. The returned cloned session will remain isolated from this session. |
There was a problem hiding this comment.
'remain' is a bit weird since the session is new.
hvanhovell
reviewed
Sep 2, 2025
| @@ -1139,4 +1187,8 @@ service SparkConnectService { | |||
|
|
|||
| // FetchErrorDetails retrieves the matched exception with details based on a provided error id. | |||
| rpc FetchErrorDetails(FetchErrorDetailsRequest) returns (FetchErrorDetailsResponse) {} | |||
|
|
|||
| // Clone a session. Creates a new session that shares the same configuration and state | |||
There was a problem hiding this comment.
Can we make this as descriptive as the Scala API doc?
hvanhovell
reviewed
Sep 2, 2025
| * @note This creates a new server-side session with the specified session ID while preserving | ||
| * the current session's configuration and state. | ||
| */ | ||
| def cloneSession(sessionId: String): SparkSession = { |
There was a problem hiding this comment.
Should we make this part of the public API?
Please annotate this with @DeveloperAPI.
hvanhovell
reviewed
Sep 2, 2025
|
|
||
| // Next ID: 5 | ||
| message CloneSessionResponse { | ||
| // Session id of the original session that was cloned. |
hvanhovell
reviewed
Sep 2, 2025
| // configuration, catalog, session state, temporary views, and registered functions | ||
| val clonedSparkSession = sourceSessionHolder.session.cloneSession() | ||
|
|
||
| val newHolder = SessionHolder(newKey.userId, newKey.sessionId, clonedSparkSession) |
There was a problem hiding this comment.
Shouldn't we clone more state here? For example MLCache, or DataFrameCache?
hvanhovell
reviewed
Sep 2, 2025
| assert(ex.getMessage.contains("Session not found")) | ||
| } | ||
|
|
||
| test("successful clone session creates new session") { |
There was a problem hiding this comment.
Should we test session independence?
hvanhovell
reviewed
Sep 2, 2025
| }, | ||
| "TARGET_SESSION_ID_FORMAT" : { | ||
| "message" : [ | ||
| "Target session ID <targetSessionId> for clone operation must be an UUID string of the format '00112233-4455-6677-8899-aabbccddeeff'." |
There was a problem hiding this comment.
Isn't there a different error for this as well?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Adds a new RPC
CloneSessionto theSparkConnectService.Why are the changes needed?
Spark Connect introduced the concept of resource isolation (via
ArtifactManager, which has been ported to classic Spark) and thus, jars/pyfiles/artifacts added to each session are isolated from other sessions.A slight rough edge is that if a user wishes to fork the state of a session but maintain independence, the only possible way is to create a new session and reupload/reinit all base jars/artifacts/pyfiles, etc.
Support for cloning through the API helps address the rough edge while maintaining all the benefits of session resource isolation.
Does this PR introduce any user-facing change?
Yes
How was this patch tested?
New individual unit tests along with new test suites.
Was this patch authored or co-authored using generative AI tooling?
Co-authored with assistance from Claude Code.