Skip to content

🤖 fix: add trace dumps for System 1 memory writer failures#1969

Open
ThomasK33 wants to merge 40 commits intomainfrom
memory-agent-hev5
Open

🤖 fix: add trace dumps for System 1 memory writer failures#1969
ThomasK33 wants to merge 40 commits intomainfrom
memory-agent-hev5

Conversation

@ThomasK33
Copy link
Member

@ThomasK33 ThomasK33 commented Jan 27, 2026

Summary

Adds richer debug visibility for the System 1 memory writer:

  • Emits explicit debug-level skip reasons when scheduling is gated (System 1 disabled / child workspace / invalid interval).
  • When a run completes without writing (no memory_write tool call) or times out, writes a full execution trace (prompt/messages, step results, tool executions) to debug_obj/ for offline inspection.

Background

We saw cases where the memory writer appeared to “not run” (no [system1][memory] logs) or would exit with timedOut: true and no memory_write call. Most lifecycle logs were debug-only, and failures lacked enough detail to understand what the model did.

Implementation

  • MemoryWriterPolicy: adds explicit debug logs for early-return gates; threads triggerMessageId into the runner.
  • system1MemoryWriter: captures per-attempt messages, onStepFinish results, and tool execution records; dumps a JSON trace to ~/.mux*/debug_obj/<workspaceId>/system1_memory_writer/ in debug mode when the run fails to update memory.

Validation

  • make static-check
  • unit tests updated for the new triggerMessageId parameter

Risks

Low. Changes are scoped to debug logging and failure-path diagnostics; the writer’s normal success path is unchanged.


📋 Implementation Plan

Debug: System 1 memory writer not running / missing [system1][memory] logs

Context / Why

You set Settings → System 1 → “Write Interval (messages)” to 1, expecting the background System 1 memory writer to run after each assistant turn and update the project memory file. You’re not seeing any [system1][memory] log lines and it looks like the writer never runs.

From the current code:

  • The memory writer is scheduled on assistant stream end.
  • It is gated by System 1 experiment enabled (experiments.system1 === true) and only runs for root workspaces (it skips child/subtask workspaces).
  • Most lifecycle logs are debug-level, so you won’t see them unless you enable debug logging. The only info logs are emitted by the memory_write tool when a memory file is actually written.

Evidence (repo)

  • Scheduling + gates:
    • src/node/services/aiService.ts stores a context at stream start and calls memoryWriterPolicy.onAssistantStreamEnd(ctx) on stream-end.
    • src/node/services/system1/memoryWriterPolicy.ts returns early unless:
      • ctx.system1Enabled === true
      • !ctx.parentWorkspaceId
      • interval = config.taskSettings.memoryWriterIntervalMessages (defaults to 2)
    • The scheduler persists state to sessions/<workspaceId>/system1-memory-writer-state.json.
  • Experiment flag plumbing:
    • src/browser/utils/messages/sendOptions.ts passes experiments.system1 based on isExperimentEnabled(EXPERIMENT_IDS.SYSTEM_1).
    • src/common/orpc/schemas/stream.ts defines experiments.system1 in the RPC schema.
  • Logging:
    • src/node/services/log.ts supports MUX_LOG_LEVEL / MUX_DEBUG.
    • src/node/services/tools/memory_write.ts logs info on successful writes.
  • Memory file location:
    • src/node/services/tools/memoryCommon.ts writes to <muxHome>/memories/<projectId>.md.
    • src/common/constants/paths.ts defines <muxHome>: ~/.mux, ~/.mux-dev (when NODE_ENV=development), or MUX_ROOT.

Approach A (recommended): add explicit “no changes” debug logs (~5–20 LoC)

  1. In src/node/services/system1/memoryWriterPolicy.ts, change the existing debug line
    "[system1][memory] Memory writer produced no output" to something explicit like
    "[system1][memory] Memory writer exited without updating memory (no memory_write call)".
    • Keep it debug (so it only appears when MUX_LOG_LEVEL=debug / MUX_DEBUG=1).
    • Preserve the existing fields (timedOut, system1Model).
  2. (Optional) Add debug logs on the early-return gates in onAssistantStreamEnd:
    • System 1 disabled (ctx.system1Enabled !== true)
    • child workspace (ctx.parentWorkspaceId)
    • invalid interval
  3. Verify:
    • Run mux with MUX_LOG_LEVEL=debug (or MUX_DEBUG=1), then send a message.
    • You should now see an explicit log even when the writer doesn’t write.

Approach B: user-side diagnosis (0 LoC)

1) Confirm the hard gates

  1. Enable System 1: Settings → Experiments → System 1 must be ON.
    • The interval alone is not enough; the backend hard-returns when experiments.system1 !== true.
  2. Use a root workspace:
    • The memory writer does not run for child/subtask workspaces (parentWorkspaceId is set).
  3. Let the assistant finish normally:
    • The writer is scheduled on stream-end. If you interrupt/abort streams, it won’t schedule.

2) Confirm the interval persisted to disk

  1. Find your mux home directory:
    • Default: ~/.mux/
    • Dev: ~/.mux-dev/ (when NODE_ENV=development)
    • Override: $MUX_ROOT
  2. Check <muxHome>/config.json contains:
    • taskSettings.memoryWriterIntervalMessages: 1

3) Verify scheduling without logs (state file)

Even if you can’t see stdout/stderr, the scheduler persists a state file per workspace.

  1. Identify your workspaceId:
    • Start by listing sessions sorted by recent activity:
      • ls -lt <muxHome>/sessions | head
    • Then inspect candidate folders’ metadata.json until you find the workspace you’re testing.
  2. Inspect:
    • <muxHome>/sessions/<workspaceId>/system1-memory-writer-state.json

Expected behavior with interval=1:

  • After each assistant completion:
    • lastRunStartedAt updates (timestamp)
    • lastRunMessageId updates
    • turnsSinceLastRun returns to 0

Interpretation:

  • No file / never updatesonAssistantStreamEnd isn’t running (most often: System 1 experiment still OFF, or streams are aborting).
  • File updates but lastRunStartedAt never set → interval not being read as 1, or the run is permanently “in flight”.
  • lastRunStartedAt changes but no memory file changes → writer ran but didn’t call memory_write (model/tool support or credentials issue).

4) Verify memory output

  • Memory files live at <muxHome>/memories/*.md.
  • After an assistant turn, check whether anything in that folder updates:
    • ls -lt <muxHome>/memories | head

5) Enable the right logs (debug)

To see the scheduler/runner messages (which include skip reasons), you need debug logging:

  • MUX_LOG_LEVEL=debug or MUX_DEBUG=1

Important: for the desktop app, you must start mux from a shell that has those env vars (GUI launches won’t inherit shell env). Once enabled, look for:

  • [system1][memory] Skipping memory writer ...
  • [system1][memory] Memory writer completed
  • [system1][memory] Memory writer failed

Debug dumps (only in debug mode) land in:

  • <muxHome>/debug_obj/

6) Ensure the writer’s model supports tool calling

The memory writer uses:

  • agentAiDefaults.system1_memory_writer.modelString (if set)
  • otherwise it falls back to the workspace’s current chat model (ctx.modelString)

If that model/provider can’t do tool calling (or lacks credentials), the writer may never call memory_write.

Actions:

  • Set an explicit tool-capable model for system1_memory_writer (via Settings → Agents or by editing config.json).
  • Confirm provider API keys are configured for that model.
Why you might see “nothing” even when it’s running

Most memory-writer logs are debug. The only info line is from the memory_write tool after the model calls it. If the model never calls memory_write (tool calling unsupported / credentials missing / etc.), you’ll see no [system1][memory] output without enabling debug logging.


Generated with mux • Model: openai:gpt-5.2 • Thinking: high • Cost: $54.20

@ThomasK33
Copy link
Member Author

@codex review

Please re-review after rebase and hash alignment.

Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d50f25c4ac

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ThomasK33
Copy link
Member Author

@codex review

Addressed latest failing checks after rebase (System1 settings tests + flake hash). Please re-review.

@ThomasK33
Copy link
Member Author

@codex review

Addressed the memory_write nullish schema comment and resolved the thread. Please re-review.

@ThomasK33
Copy link
Member Author

@codex review

Applied follow-up fix for failing integration test (duplicate General button match). Please re-review.

@ThomasK33
Copy link
Member Author

@codex review

Follow-up integration fix: updated thinkingPolicy test to match current Internal section heading in Agents settings.

@chatgpt-codex-connector
Copy link

Codex Review: Didn't find any major issues. You're on a roll.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ThomasK33
Copy link
Member Author

@codex review

2 similar comments
@ThomasK33
Copy link
Member Author

@codex review

@ThomasK33
Copy link
Member Author

@codex review

@chatgpt-codex-connector
Copy link

Codex Review: Didn't find any major issues. Chef's kiss.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ThomasK33
Copy link
Member Author

@codex review

Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d41bc5e667

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

… heading

The Agents settings page no longer renders the legacy "System1 Defaults (internal)"
label. Assert the "Internal" heading instead before locating the System1 Bash card,
matching the current TasksSection structure.

Signed-off-by: Thomas Kosiewski <tk@coder.com>

---

_Generated with `mux` • Model: `openai:gpt-5.3-codex` • Thinking: `xhigh` • Cost: `$57.73`_

<!-- mux-attribution: model=openai:gpt-5.3-codex thinking=xhigh costs=57.73 -->

Change-Id: I76028cac1ebae4628ce7b3fb6a77eb5e6a6ae675
Change-Id: Ia70e53a105f8a2af10284a803e5549fb52373fb7
Signed-off-by: Thomas Kosiewski <tk@coder.com>
Change-Id: Ieaebdfcaa0b52afb53f3f85f9075fc509bd57803
Signed-off-by: Thomas Kosiewski <tk@coder.com>
Change-Id: Iec999f96974d7b245d6edea8ebd1f694b7b20ea8
Signed-off-by: Thomas Kosiewski <tk@coder.com>
Change-Id: Ie86193feec602f596107ec5fe28ba98b5c4cf1f1
Signed-off-by: Thomas Kosiewski <tk@coder.com>
Change-Id: I24d2ed4d4189293a2eff2ffbbb6699cb26a30849
Signed-off-by: Thomas Kosiewski <tk@coder.com>
Change-Id: Icc7bc67e0bbc95688dd6295cbd81609fe4467cda
Signed-off-by: Thomas Kosiewski <tk@coder.com>
Change-Id: I617795e1b4091bee99bfa5faa06394b4dfcd8ae5
Signed-off-by: Thomas Kosiewski <tk@coder.com>
Change-Id: I72ba518ba43024c941f9cab7b2f33e2406204551
Signed-off-by: Thomas Kosiewski <tk@coder.com>
Change-Id: I2f187641921f17e9f67377a7191a261d7fd05e6d
Signed-off-by: Thomas Kosiewski <tk@coder.com>
Change-Id: I636801b84dc11124b19d2dcae6c7a42464e8e367
Signed-off-by: Thomas Kosiewski <tk@coder.com>
Change-Id: I4d27f1f8e1c50b29c1bbdbbbb2874cb280960573
Signed-off-by: Thomas Kosiewski <tk@coder.com>
Add a new no_new_memories tool and require System1 memory runs to explicitly call either memory_write or no_new_memories.\n\n- Added no_new_memories tool definition + implementation + unit test\n- Updated system1_memory_writer prompt/tool list to include explicit no-op path\n- Enforced tool-policy-style retry reminder when neither required tool is called\n- Returned memoryAction in system1 memory writer results for downstream logging\n- Regenerated built-in agent docs/content and updated memory writer tests\n\n---\n\n_Generated with `mux` • Model: `openai:gpt-5.3-codex` • Thinking: `xhigh` • Cost: `8.91`_\n\n<!-- mux-attribution: model=openai:gpt-5.3-codex thinking=xhigh costs=58.91 -->

Change-Id: I6b06e902b8a2ca03dcb3146b1eea54eef38480cf
Signed-off-by: Thomas Kosiewski <tk@coder.com>
Drop an unused SearchableModelSelect import introduced during conflict reconciliation so post-rebase static checks pass.\n\n---\n\n_Generated with `mux` • Model: `openai:gpt-5.3-codex` • Thinking: `xhigh` • Cost: `8.91`_\n\n<!-- mux-attribution: model=openai:gpt-5.3-codex thinking=xhigh costs=58.91 -->

Change-Id: Ib3db1c855c790290d0090199de49f9d7c7517c11
Signed-off-by: Thomas Kosiewski <tk@coder.com>
Address Codex feedback by keeping no_new_memories internal to the System1 memory-writer path instead of exposing it through the global base tool list used by normal agents.\n\n---\n\n_Generated with `mux` • Model: `openai:gpt-5.3-codex` • Thinking: `xhigh` • Cost: `8.91`_\n\n<!-- mux-attribution: model=openai:gpt-5.3-codex thinking=xhigh costs=58.91 -->

Change-Id: I5d1e69a7f240dcb6e3afc6459fa2517374de1e87
Signed-off-by: Thomas Kosiewski <tk@coder.com>
Address Codex feedback by removing no_new_memories from the global nonRuntime tool registry.\n\nSystem1 memory writer still gets the tool via its dedicated internal tool wiring, but user-facing agents (exec/plan and descendants) no longer see this no-op tool in normal chat sessions.\n\n---\n\n_Generated with `mux` • Model: `openai:gpt-5.3-codex` • Thinking: `xhigh` • Cost: `8.91`_\n\n<!-- mux-attribution: model=openai:gpt-5.3-codex thinking=xhigh costs=58.91 -->

Change-Id: I864d41ea556ceeee0d3680dd4190b2e875996efe
Signed-off-by: Thomas Kosiewski <tk@coder.com>
Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 178812e7df

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ThomasK33
Copy link
Member Author

@codex review

Change-Id: I032b87e0f6b32f6007aae666eb9e27e53ef0e8e8
Signed-off-by: Thomas Kosiewski <tk@coder.com>
Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 59ca76971d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ThomasK33
Copy link
Member Author

@codex review

Change-Id: Iaa794ca6b0567b1c49fbd6327f50c78a6ecb626e
Signed-off-by: Thomas Kosiewski <tk@coder.com>
Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fe3120b646

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Change-Id: I0eb4824bff452d732240811aa0d23abb0c6d62c4
Signed-off-by: Thomas Kosiewski <tk@coder.com>
@ThomasK33
Copy link
Member Author

@codex review

Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: df372bde22

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Change-Id: I3dc50fbc2228b22b8e072e7f63feb8deff060e1c
Signed-off-by: Thomas Kosiewski <tk@coder.com>
@ThomasK33
Copy link
Member Author

@codex review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or functionality

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

Comments