Wire encryption into serialization layer#957
Wire encryption into serialization layer#957TooTallNate wants to merge 1 commit intonate/vercel-encryptionfrom
Conversation
🦋 Changeset detectedLatest commit: 180ed09 The changes in this PR will be included in the next version bump. This PR includes changesets to release 18 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
🧪 E2E Test ResultsNo test result files found. ❌ Some E2E test jobs failed:
Check the workflow run for details. |
There was a problem hiding this comment.
Pull request overview
This PR implements end-to-end encryption for workflow user data by wiring encryption functionality into the serialization layer. It builds on previous PRs that generated runId client-side (#954), made serialization functions async (#955), and added the Vercel encryption implementation (#956).
Changes:
- Adds encryption/decryption helper functions (
maybeEncrypt,maybeDecrypt,isEncrypted,peekFormatPrefix) and unused stream encryption utilities (getEncryptStream,getDecryptStream) - Adds
ENCRYPTED('encr') format prefix toSerializationFormatenum - Wires encryption into all 8 dehydrate/hydrate functions by calling
maybeEncryptafter serialization andmaybeDecryptbefore deserialization - Implements inline stream encryption in
WorkflowServerWritableStreamandWorkflowServerReadableStream - Passes
runIdtoWorkflowServerReadableStreamfor decryption context - Adds 8 comprehensive integration tests using a mock XOR encryptor
- Removes unused
_prefixes fromencryptorandrunIdparameters (now actively used) - Adds type casts (
as any[],as any,as TResult) to hydration call sites to preserve type information
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/core/src/serialization.ts | Core encryption implementation: adds format prefix, helper functions, encryption wiring in dehydrate/hydrate functions, and inline stream encryption/decryption |
| packages/core/src/workflow.ts | Adds type cast for hydrated workflow arguments |
| packages/core/src/runtime/step-handler.ts | Adds type cast for hydrated step arguments |
| packages/core/src/runtime/run.ts | Adds type cast for hydrated workflow return value |
| packages/core/src/serialization.test.ts | Adds 8 encryption integration tests with mock XOR encryptor |
| .changeset/e2e-encryption.md | Documents the end-to-end encryption feature addition |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| export function getEncryptStream( | ||
| encryptor: Encryptor, | ||
| runId: string | ||
| ): TransformStream<Uint8Array, Uint8Array> { | ||
| return new TransformStream<Uint8Array, Uint8Array>({ | ||
| async transform(chunk, controller) { | ||
| if (!encryptor.encrypt) { | ||
| // No encryption available, pass through unchanged | ||
| controller.enqueue(chunk); | ||
| return; | ||
| } | ||
|
|
||
| const encrypted = await encryptor.encrypt(chunk, { runId }); | ||
| controller.enqueue(encrypted); | ||
| }, | ||
| }); | ||
| } | ||
|
|
||
| /** | ||
| * Create a transform stream that decrypts each chunk. | ||
| * Used to decrypt stream data read from storage. | ||
| * | ||
| * @param encryptor - Encryptor instance with decrypt function | ||
| * @param runId - Run ID for decryption context | ||
| * @returns TransformStream that decrypts Uint8Array chunks | ||
| */ | ||
| export function getDecryptStream( | ||
| encryptor: Encryptor, | ||
| runId: string | ||
| ): TransformStream<Uint8Array, Uint8Array> { | ||
| return new TransformStream<Uint8Array, Uint8Array>({ | ||
| async transform(chunk, controller) { | ||
| // Check if chunk is encrypted | ||
| if (!isEncrypted(chunk)) { | ||
| // Not encrypted, pass through unchanged | ||
| controller.enqueue(chunk); | ||
| return; | ||
| } | ||
|
|
||
| if (!encryptor.decrypt) { | ||
| throw new WorkflowRuntimeError( | ||
| 'Encrypted stream data encountered but Encryptor does not support decryption. ' + | ||
| 'Ensure VERCEL_DEPLOYMENT_KEY is set or provide an Encryptor with decryption support.' | ||
| ); | ||
| } | ||
|
|
||
| const decrypted = await encryptor.decrypt(chunk, { runId }); | ||
| controller.enqueue(decrypted); | ||
| }, | ||
| }); | ||
| } | ||
|
|
There was a problem hiding this comment.
The functions getEncryptStream and getDecryptStream are defined but never used in the codebase. Stream encryption and decryption are instead implemented inline within WorkflowServerWritableStream (lines 419-424) and WorkflowServerReadableStream (lines 364-379).
These functions should either be removed as dead code, or the inline implementations should be refactored to use these TransformStream factories for consistency and maintainability.
| export function getEncryptStream( | |
| encryptor: Encryptor, | |
| runId: string | |
| ): TransformStream<Uint8Array, Uint8Array> { | |
| return new TransformStream<Uint8Array, Uint8Array>({ | |
| async transform(chunk, controller) { | |
| if (!encryptor.encrypt) { | |
| // No encryption available, pass through unchanged | |
| controller.enqueue(chunk); | |
| return; | |
| } | |
| const encrypted = await encryptor.encrypt(chunk, { runId }); | |
| controller.enqueue(encrypted); | |
| }, | |
| }); | |
| } | |
| /** | |
| * Create a transform stream that decrypts each chunk. | |
| * Used to decrypt stream data read from storage. | |
| * | |
| * @param encryptor - Encryptor instance with decrypt function | |
| * @param runId - Run ID for decryption context | |
| * @returns TransformStream that decrypts Uint8Array chunks | |
| */ | |
| export function getDecryptStream( | |
| encryptor: Encryptor, | |
| runId: string | |
| ): TransformStream<Uint8Array, Uint8Array> { | |
| return new TransformStream<Uint8Array, Uint8Array>({ | |
| async transform(chunk, controller) { | |
| // Check if chunk is encrypted | |
| if (!isEncrypted(chunk)) { | |
| // Not encrypted, pass through unchanged | |
| controller.enqueue(chunk); | |
| return; | |
| } | |
| if (!encryptor.decrypt) { | |
| throw new WorkflowRuntimeError( | |
| 'Encrypted stream data encountered but Encryptor does not support decryption. ' + | |
| 'Ensure VERCEL_DEPLOYMENT_KEY is set or provide an Encryptor with decryption support.' | |
| ); | |
| } | |
| const decrypted = await encryptor.decrypt(chunk, { runId }); | |
| controller.enqueue(decrypted); | |
| }, | |
| }); | |
| } |
| const args = (await hydrateWorkflowArguments( | ||
| workflowRun.input, | ||
| workflowRun.runId, | ||
| world, | ||
| vmGlobalThis | ||
| ); | ||
| )) as any[]; |
There was a problem hiding this comment.
The return value of hydrateWorkflowArguments is cast to any[] using a type assertion. While this is necessary because the function returns Promise<unknown>, this cast bypasses type safety.
The hydrated workflow arguments should be an array to be spread into the workflow function call on line 643. Consider changing the return type of hydrateWorkflowArguments to Promise<any[]> to make this contract explicit, or add runtime validation to ensure the result is actually an array before casting.
| const hydratedInput = (await hydrateStepArguments( | ||
| step.input, | ||
| workflowRunId, | ||
| world, | ||
| ops | ||
| ); | ||
| )) as any; |
There was a problem hiding this comment.
The return value of hydrateStepArguments is cast to any using a type assertion. While this is necessary because the function returns Promise<unknown>, this cast bypasses type safety.
The result is expected to have properties args, thisVal, and closureVars (used on lines 273, 274, 300). Consider creating a dedicated interface for the step input structure (e.g., StepInput { args: any[], thisVal?: any, closureVars?: Record<string, any> }) and changing the return type to make this contract explicit.
| return (await hydrateWorkflowReturnValue( | ||
| run.output, | ||
| this.runId, | ||
| this.world | ||
| ); | ||
| )) as TResult; |
There was a problem hiding this comment.
The return value of hydrateWorkflowReturnValue is cast to TResult using a type assertion. While this is necessary because the function returns Promise<unknown>, this cast bypasses type safety and could lead to runtime errors if the deserialized value doesn't match the expected type.
However, this is likely acceptable in this context as workflow return types are user-defined and can't be validated at compile time. The cast ensures the caller gets the expected generic type.
| // Decrypt chunk if encrypted and runId is provided | ||
| let chunk = result.value; | ||
| if (runId && isEncrypted(chunk)) { | ||
| const world = getWorld(); | ||
| if (!world.decrypt) { | ||
| controller.error( | ||
| new WorkflowRuntimeError( | ||
| 'Encrypted stream data encountered but World does not support decryption. ' + | ||
| 'Ensure VERCEL_DEPLOYMENT_KEY is set.' | ||
| ) | ||
| ); | ||
| return; | ||
| } | ||
| chunk = await world.decrypt(chunk, { runId }); | ||
| } | ||
| controller.enqueue(chunk); |
There was a problem hiding this comment.
The decryption logic in WorkflowServerReadableStream only decrypts if runId is provided. However, if encrypted data is encountered when runId is undefined, the encrypted data will be passed through unchanged (line 379), which will likely cause downstream errors when trying to deserialize the encrypted bytes.
Consider adding an error check: if isEncrypted(chunk) returns true but runId is undefined, throw an error indicating that encrypted data was encountered but decryption context is missing.
| * Encrypt data if the world supports encryption. | ||
| * Returns original data if encryption is not available. | ||
| * | ||
| * @param data - Serialized data to encrypt | ||
| * @param world - World instance (may have encrypt function) |
There was a problem hiding this comment.
The JSDoc comment incorrectly states @param world - World instance (may have encrypt function) but the actual parameter is named encryptor and is of type Encryptor, not World. The documentation should be updated to reflect the actual parameter name and type.
| * Encrypt data if the world supports encryption. | |
| * Returns original data if encryption is not available. | |
| * | |
| * @param data - Serialized data to encrypt | |
| * @param world - World instance (may have encrypt function) | |
| * Encrypt data if the encryptor supports encryption. | |
| * Returns original data if encryption is not available. | |
| * | |
| * @param data - Serialized data to encrypt | |
| * @param encryptor - Encryptor instance (may have encrypt function) |
| /** | ||
| * Called from the `start()` function to serialize the workflow arguments | ||
| * into a format that can be saved to the database and then hydrated from | ||
| * within the workflow execution environment. | ||
| * | ||
| * @param value | ||
| * @param global | ||
| * @param runId | ||
| * @param value - The value to serialize | ||
| * @param runId - The workflow run ID (required for encryption context) | ||
| * @param world - World instance for encryption support | ||
| * @param ops - Promise array for stream operations | ||
| * @param global - Global object for serialization context | ||
| * @param v1Compat - Enable legacy v1 compatibility mode | ||
| * @returns The dehydrated value as binary data (Uint8Array) with format prefix |
There was a problem hiding this comment.
Several JSDoc comments throughout the dehydrate/hydrate functions incorrectly reference @param world - World instance for encryption support when the actual parameter is named encryptor and is of type Encryptor. This affects lines:
- 1427 (dehydrateWorkflowArguments)
- 1468 (hydrateWorkflowArguments)
- 1510 (dehydrateWorkflowReturnValue)
- 1545 (hydrateWorkflowReturnValue)
- 1590 (dehydrateStepArguments)
- 1629 (hydrateStepArguments)
- 1674 (dehydrateStepReturnValue)
- 1715 (hydrateStepReturnValue)
All of these should be updated to say @param encryptor - Encryptor instance for encryption support or similar.
| export async function dehydrateWorkflowReturnValue( | ||
| value: unknown, | ||
| _runId: string, | ||
| _encryptor: Encryptor, | ||
| global: Record<string, any> = globalThis, | ||
| v1Compat = false | ||
| runId: string, | ||
| encryptor: Encryptor, | ||
| global: Record<string, any> = globalThis |
There was a problem hiding this comment.
The v1Compat parameter and its associated logic (lines 1522-1524 in the original code) have been removed from dehydrateWorkflowReturnValue. However, this parameter is still present in the other dehydrate functions (dehydrateWorkflowArguments, dehydrateStepArguments, dehydrateStepReturnValue).
This creates an API inconsistency where three functions support v1 compatibility mode but one doesn't. This is likely unintentional, as the PR description doesn't mention removing v1 compatibility support.
Additionally, the test wrapper function at line 70-76 in serialization.test.ts is passing a v1Compat parameter to this function, which will be silently ignored.
|
Warning This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
This stack of pull requests is managed by Graphite. Learn more about stacking. |
TooTallNate
force-pushed
the
nate/vercel-encryption
branch
from
efd031c to
c1795c7
Compare
TooTallNate
force-pushed
the
nate/wire-encryption
branch
from
7e25694 to
e5b148e
Compare
TooTallNate
force-pushed
the
nate/vercel-encryption
branch
from
c1795c7 to
60fac27
Compare
TooTallNate
force-pushed
the
nate/wire-encryption
branch
from
e5b148e to
c686cb4
Compare
TooTallNate
force-pushed
the
nate/vercel-encryption
branch
from
60fac27 to
0d60577
Compare
TooTallNate
force-pushed
the
nate/wire-encryption
branch
from
c686cb4 to
61c8590
Compare
TooTallNate
force-pushed
the
nate/vercel-encryption
branch
from
0d60577 to
2fcf24e
Compare
TooTallNate
force-pushed
the
nate/wire-encryption
branch
from
61c8590 to
180ed09
Compare
1 participant
Summary
maybeEncrypt()/maybeDecrypt()helpers that conditionally encrypt/decrypt based on whether anEncryptoris providedpeekFormatPrefix(),isEncrypted()helper functionsENCRYPTED('encr') format prefix toSerializationFormatgetEncryptStream()/getDecryptStream()TransformStream factories for stream encryptionmaybeEncryptafter serialization, hydrate callsmaybeDecryptbefore deserializationWorkflowServerWritableStream(encrypts chunks before writing to storage)WorkflowServerReadableStream(decrypts chunks when reading from storage)runIdtoWorkflowServerReadableStreamfor decryption contextThis is the "light the fuse" PR that actually enables encryption. When a
Worldimplementation providesencrypt/decryptfunctions (e.g., viaVERCEL_DEPLOYMENT_KEYenv var), all workflow user data will be encrypted at rest.Stack
Test plan
All 311 core tests pass (303 existing + 8 new encryption integration tests). Build succeeds.