🤖 fix: add trace dumps for System 1 memory writer failures#1969
🤖 fix: add trace dumps for System 1 memory writer failures#1969
Conversation
b93c543 to
5f81afb
Compare
9b301d7 to
f2b2dab
Compare
f2b2dab to
ac2bd9b
Compare
ac2bd9b to
d89ca1e
Compare
d89ca1e to
4d218e5
Compare
8e5952c to
8ddbad0
Compare
4f4de49 to
d50f25c
Compare
|
@codex review Please re-review after rebase and hash alignment. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d50f25c4ac
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review Addressed latest failing checks after rebase (System1 settings tests + flake hash). Please re-review. |
|
@codex review Addressed the memory_write nullish schema comment and resolved the thread. Please re-review. |
|
@codex review Applied follow-up fix for failing integration test (duplicate General button match). Please re-review. |
|
@codex review Follow-up integration fix: updated thinkingPolicy test to match current Internal section heading in Agents settings. |
|
Codex Review: Didn't find any major issues. You're on a roll. ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
@codex review |
|
Codex Review: Didn't find any major issues. Chef's kiss. ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d41bc5e667
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
… heading The Agents settings page no longer renders the legacy "System1 Defaults (internal)" label. Assert the "Internal" heading instead before locating the System1 Bash card, matching the current TasksSection structure. Signed-off-by: Thomas Kosiewski <tk@coder.com> --- _Generated with `mux` • Model: `openai:gpt-5.3-codex` • Thinking: `xhigh` • Cost: `$57.73`_ <!-- mux-attribution: model=openai:gpt-5.3-codex thinking=xhigh costs=57.73 --> Change-Id: I76028cac1ebae4628ce7b3fb6a77eb5e6a6ae675
Change-Id: Ia70e53a105f8a2af10284a803e5549fb52373fb7 Signed-off-by: Thomas Kosiewski <tk@coder.com>
Change-Id: Ieaebdfcaa0b52afb53f3f85f9075fc509bd57803 Signed-off-by: Thomas Kosiewski <tk@coder.com>
Change-Id: Iec999f96974d7b245d6edea8ebd1f694b7b20ea8 Signed-off-by: Thomas Kosiewski <tk@coder.com>
Change-Id: Ie86193feec602f596107ec5fe28ba98b5c4cf1f1 Signed-off-by: Thomas Kosiewski <tk@coder.com>
Change-Id: I24d2ed4d4189293a2eff2ffbbb6699cb26a30849 Signed-off-by: Thomas Kosiewski <tk@coder.com>
Change-Id: Icc7bc67e0bbc95688dd6295cbd81609fe4467cda Signed-off-by: Thomas Kosiewski <tk@coder.com>
Change-Id: I617795e1b4091bee99bfa5faa06394b4dfcd8ae5 Signed-off-by: Thomas Kosiewski <tk@coder.com>
Change-Id: I72ba518ba43024c941f9cab7b2f33e2406204551 Signed-off-by: Thomas Kosiewski <tk@coder.com>
Change-Id: I2f187641921f17e9f67377a7191a261d7fd05e6d Signed-off-by: Thomas Kosiewski <tk@coder.com>
Change-Id: I636801b84dc11124b19d2dcae6c7a42464e8e367 Signed-off-by: Thomas Kosiewski <tk@coder.com>
Change-Id: I4d27f1f8e1c50b29c1bbdbbbb2874cb280960573 Signed-off-by: Thomas Kosiewski <tk@coder.com>
Add a new no_new_memories tool and require System1 memory runs to explicitly call either memory_write or no_new_memories.\n\n- Added no_new_memories tool definition + implementation + unit test\n- Updated system1_memory_writer prompt/tool list to include explicit no-op path\n- Enforced tool-policy-style retry reminder when neither required tool is called\n- Returned memoryAction in system1 memory writer results for downstream logging\n- Regenerated built-in agent docs/content and updated memory writer tests\n\n---\n\n_Generated with `mux` • Model: `openai:gpt-5.3-codex` • Thinking: `xhigh` • Cost: `8.91`_\n\n<!-- mux-attribution: model=openai:gpt-5.3-codex thinking=xhigh costs=58.91 --> Change-Id: I6b06e902b8a2ca03dcb3146b1eea54eef38480cf Signed-off-by: Thomas Kosiewski <tk@coder.com>
Drop an unused SearchableModelSelect import introduced during conflict reconciliation so post-rebase static checks pass.\n\n---\n\n_Generated with `mux` • Model: `openai:gpt-5.3-codex` • Thinking: `xhigh` • Cost: `8.91`_\n\n<!-- mux-attribution: model=openai:gpt-5.3-codex thinking=xhigh costs=58.91 --> Change-Id: Ib3db1c855c790290d0090199de49f9d7c7517c11 Signed-off-by: Thomas Kosiewski <tk@coder.com>
Address Codex feedback by keeping no_new_memories internal to the System1 memory-writer path instead of exposing it through the global base tool list used by normal agents.\n\n---\n\n_Generated with `mux` • Model: `openai:gpt-5.3-codex` • Thinking: `xhigh` • Cost: `8.91`_\n\n<!-- mux-attribution: model=openai:gpt-5.3-codex thinking=xhigh costs=58.91 --> Change-Id: I5d1e69a7f240dcb6e3afc6459fa2517374de1e87 Signed-off-by: Thomas Kosiewski <tk@coder.com>
Address Codex feedback by removing no_new_memories from the global nonRuntime tool registry.\n\nSystem1 memory writer still gets the tool via its dedicated internal tool wiring, but user-facing agents (exec/plan and descendants) no longer see this no-op tool in normal chat sessions.\n\n---\n\n_Generated with `mux` • Model: `openai:gpt-5.3-codex` • Thinking: `xhigh` • Cost: `8.91`_\n\n<!-- mux-attribution: model=openai:gpt-5.3-codex thinking=xhigh costs=58.91 --> Change-Id: I864d41ea556ceeee0d3680dd4190b2e875996efe Signed-off-by: Thomas Kosiewski <tk@coder.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 178812e7df
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
178812e to
59ca769
Compare
|
@codex review |
Change-Id: I032b87e0f6b32f6007aae666eb9e27e53ef0e8e8 Signed-off-by: Thomas Kosiewski <tk@coder.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 59ca76971d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
Change-Id: Iaa794ca6b0567b1c49fbd6327f50c78a6ecb626e Signed-off-by: Thomas Kosiewski <tk@coder.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fe3120b646
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Change-Id: I0eb4824bff452d732240811aa0d23abb0c6d62c4 Signed-off-by: Thomas Kosiewski <tk@coder.com>
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: df372bde22
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Change-Id: I3dc50fbc2228b22b8e072e7f63feb8deff060e1c Signed-off-by: Thomas Kosiewski <tk@coder.com>
|
@codex review |
Summary
Adds richer debug visibility for the System 1 memory writer:
memory_writetool call) or times out, writes a full execution trace (prompt/messages, step results, tool executions) todebug_obj/for offline inspection.Background
We saw cases where the memory writer appeared to “not run” (no
[system1][memory]logs) or would exit withtimedOut: trueand nomemory_writecall. Most lifecycle logs were debug-only, and failures lacked enough detail to understand what the model did.Implementation
MemoryWriterPolicy: adds explicit debug logs for early-return gates; threadstriggerMessageIdinto the runner.system1MemoryWriter: captures per-attempt messages,onStepFinishresults, and tool execution records; dumps a JSON trace to~/.mux*/debug_obj/<workspaceId>/system1_memory_writer/in debug mode when the run fails to update memory.Validation
make static-checktriggerMessageIdparameterRisks
Low. Changes are scoped to debug logging and failure-path diagnostics; the writer’s normal success path is unchanged.
📋 Implementation Plan
Debug: System 1 memory writer not running / missing
[system1][memory]logsContext / Why
You set Settings → System 1 → “Write Interval (messages)” to
1, expecting the background System 1 memory writer to run after each assistant turn and update the project memory file. You’re not seeing any[system1][memory]log lines and it looks like the writer never runs.From the current code:
experiments.system1 === true) and only runs for root workspaces (it skips child/subtask workspaces).infologs are emitted by thememory_writetool when a memory file is actually written.Evidence (repo)
src/node/services/aiService.tsstores a context at stream start and callsmemoryWriterPolicy.onAssistantStreamEnd(ctx)onstream-end.src/node/services/system1/memoryWriterPolicy.tsreturns early unless:ctx.system1Enabled === true!ctx.parentWorkspaceIdinterval = config.taskSettings.memoryWriterIntervalMessages(defaults to 2)sessions/<workspaceId>/system1-memory-writer-state.json.src/browser/utils/messages/sendOptions.tspassesexperiments.system1based onisExperimentEnabled(EXPERIMENT_IDS.SYSTEM_1).src/common/orpc/schemas/stream.tsdefinesexperiments.system1in the RPC schema.src/node/services/log.tssupportsMUX_LOG_LEVEL/MUX_DEBUG.src/node/services/tools/memory_write.tslogsinfoon successful writes.src/node/services/tools/memoryCommon.tswrites to<muxHome>/memories/<projectId>.md.src/common/constants/paths.tsdefines<muxHome>:~/.mux,~/.mux-dev(whenNODE_ENV=development), orMUX_ROOT.Approach A (recommended): add explicit “no changes” debug logs (~5–20 LoC)
src/node/services/system1/memoryWriterPolicy.ts, change the existing debug line"[system1][memory] Memory writer produced no output"to something explicit like"[system1][memory] Memory writer exited without updating memory (no memory_write call)".debug(so it only appears whenMUX_LOG_LEVEL=debug/MUX_DEBUG=1).timedOut,system1Model).debuglogs on the early-return gates inonAssistantStreamEnd:ctx.system1Enabled !== true)ctx.parentWorkspaceId)MUX_LOG_LEVEL=debug(orMUX_DEBUG=1), then send a message.Approach B: user-side diagnosis (0 LoC)
1) Confirm the hard gates
experiments.system1 !== true.parentWorkspaceIdis set).stream-end. If you interrupt/abort streams, it won’t schedule.2) Confirm the interval persisted to disk
~/.mux/~/.mux-dev/(whenNODE_ENV=development)$MUX_ROOT<muxHome>/config.jsoncontains:taskSettings.memoryWriterIntervalMessages: 13) Verify scheduling without logs (state file)
Even if you can’t see stdout/stderr, the scheduler persists a state file per workspace.
workspaceId:ls -lt <muxHome>/sessions | headmetadata.jsonuntil you find the workspace you’re testing.<muxHome>/sessions/<workspaceId>/system1-memory-writer-state.jsonExpected behavior with
interval=1:lastRunStartedAtupdates (timestamp)lastRunMessageIdupdatesturnsSinceLastRunreturns to0Interpretation:
onAssistantStreamEndisn’t running (most often: System 1 experiment still OFF, or streams are aborting).lastRunStartedAtnever set → interval not being read as1, or the run is permanently “in flight”.lastRunStartedAtchanges but no memory file changes → writer ran but didn’t callmemory_write(model/tool support or credentials issue).4) Verify memory output
<muxHome>/memories/*.md.ls -lt <muxHome>/memories | head5) Enable the right logs (debug)
To see the scheduler/runner messages (which include skip reasons), you need debug logging:
MUX_LOG_LEVEL=debugorMUX_DEBUG=1Important: for the desktop app, you must start mux from a shell that has those env vars (GUI launches won’t inherit shell env). Once enabled, look for:
[system1][memory] Skipping memory writer ...[system1][memory] Memory writer completed[system1][memory] Memory writer failedDebug dumps (only in debug mode) land in:
<muxHome>/debug_obj/6) Ensure the writer’s model supports tool calling
The memory writer uses:
agentAiDefaults.system1_memory_writer.modelString(if set)ctx.modelString)If that model/provider can’t do tool calling (or lacks credentials), the writer may never call
memory_write.Actions:
system1_memory_writer(via Settings → Agents or by editingconfig.json).Why you might see “nothing” even when it’s running
Most memory-writer logs are
debug. The onlyinfoline is from thememory_writetool after the model calls it. If the model never callsmemory_write(tool calling unsupported / credentials missing / etc.), you’ll see no[system1][memory]output without enabling debug logging.Generated with
mux• Model:openai:gpt-5.2• Thinking:high• Cost:$54.20