feat: regenerate in Chat, agent and Chatflow app

[xuzuodong]

Description

Fixes #7161

Support mode: Chatbot, Agent and Chatflow mode

1. The regenerate button can be accessed via

   • ✅ In explore page, i.e. installed App
   • ✅ Single debug view
   • ✅ Web app
   • ✅ Embedded app

2. The regenerate button won't be displayed in

   • ❌ Multiple debug view
   • ❌ Log view
   • ❌ Chatflow's chat record view

3. Completion and Workflow mode should not be affected by this PR.

4. All sent messages in Chatbot, Agent and Chatflow have parent_message_id field now for future considerations (important for implementing branched chat history tree).

Demonstration

1. Agent mode, accessed via Explore page:

2024-08-26.18.10.32.mov

2. Chatflow mode, accessed via embedding app:

2024-08-30.16.49.09.mov

Known issue

The Chatflow run history and log detail view are designed to display all historical messages. However, for now, it is flattened and displays all messages simultaneously, rather than offering an interactive UI to toggle between different threads.

I've already implemented this fix in a separated PR xuzuodong#1, so we can look into this later.

Database migration caveat

A new field, parent_message_id, will be added to the messages table. All existing messages will be set to a special value for future compatibility: 00000000-0000-0000-0000-000000000000.

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update, included: Dify Document
• Improvement, including but not limited to code refactoring, performance optimization, and UI/UX improvement
• Dependency upgrade

Testing Instructions

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

• Test A
• Test B

[takatost]

I tried regenerating both conversation histories 1 and 2. They display correctly in the WebApp. However, after regenerating, the regenerated records are also shown in the App Log, and they are sorted by time.

I think the App Log should, by default, only display the regenerated records. If possible, it would be great to add < > toggle buttons to allow viewing of the historical generation records.

[takatost]

The parent_message_id in the messages table has both UUID_NIL and None in the form of a null value, which is expected to be ?

[xuzuodong]

I think the App Log should, by default, only display the regenerated records. If possible, it would be great to add < > toggle buttons to allow viewing of the historical generation records.

This is exactly the next PR I will create (currently at xuzuodong#1). I separated this to another PR so that changes of files and codes won't be that much and can be easier to review.

[takatost]

I think the App Log should, by default, only display the regenerated records. If possible, it would be great to add < > toggle buttons to allow viewing of the historical generation records.

This is exactly the next PR I will create (currently at xuzuodong#1). I separated this to another PR so that changes of files and codes won't be that much and can be easier to review.

Is it possible to keep only the last record this time? Showing it all in the Log can be confusing for app maintainers.

[xuzuodong]

The parent_message_id in the messages table has both UUID_NIL and None in the form of a null value, which is expected to be ?

The parent_message_id value of all messages before this PR will be marked as UUID_NIL.

For the first message in a conversation and the first regenerated message, their parent_message_id will be set to NULL.

These two cases need to be distinguished to support left-right conversation switching in next implementation.

[xuzuodong]

Is it possible to keep only the last record this time? Showing it all in the Log can be confusing for app maintainers.

Sure, will do it today.

[takatost]

The parent_message_id in the messages table has both UUID_NIL and None in the form of a null value, which is expected to be ?

The parent_message_id value of all messages before this PR will be marked as UUID_NIL.

For the first message in a conversation and the first regenerated message, their parent_message_id will be set to NULL.

These two cases need to be distinguished to support left-right conversation switching in next implementation.

Does it mean that UUID_NIL means that it was generated only once, None means the first time, and having a specific UUID is the subsequent re-generation of the message record?

[xuzuodong]

Does it mean that UUID_NIL means that it was generated only once,

UUID_NIL is actually just a mark that says the message is from old version, and later when implementing the message navigating feature, these messages should be processed in a different way.

None means the first time

NULL means the message is a head message.

having a specific UUID is the subsequent re-generation of the message record?

And after this PR take effect, all non-head sent messages will have a valid UUID (non NULL and non UUID_NIL).

---

Let me show some images to explain this more accurately

1. Before this PR takes effect, all messages that already exist in the DB (or we can call this kind of messages old version messages, or legacy messages) will look like this:

2. After this PR takes effect, newly sent messages will look like:

Note that the first message with id 1 doesn't have a parent_message_id, meaning that it's a head message.

3. Now Let's say we regenerate message 3, which will introduce a new branch, and the newly generated message will also be based on message 2:

4. Now what if I want to regenerate message 1?

Here's what's going to happen:

The newly created message will also be a head message. That said, if a message's parent_message_id is NULL, this is no doubt a head message, but it can be either the first message of the conversation or a regeneration of the first message.

5. Plus, we can also regenerate a message that already exists before this PR takes affect:

[takatost]

I see what UUID_NIL means, if don't do that, you have to batch update the parent-child ID data via script, and there will be a longer data break when updating the service, it's really better this way.