From f337096854eb4ae0894636e13daeb8e53c403be3 Mon Sep 17 00:00:00 2001 From: ruthenian8 Date: Mon, 2 Oct 2023 14:09:02 +0300 Subject: [PATCH 01/15] add stub section --- docs/source/user_guides.rst | 7 +++++++ docs/source/user_guides/context_guide.rst | 14 ++++++++++++++ 2 files changed, 21 insertions(+) create mode 100644 docs/source/user_guides/context_guide.rst diff --git a/docs/source/user_guides.rst b/docs/source/user_guides.rst index 8724a1489..866d5b45e 100644 --- a/docs/source/user_guides.rst +++ b/docs/source/user_guides.rst @@ -9,6 +9,12 @@ those include but are not limited to: dialog graph creation, specifying start an setting transitions and conditions, using ``Context`` object in order to receive information about current script execution. +:doc:`Context guide <./user_guides/context_guide>` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The ``context tutorial`` walks you through the details of working with the +``Context`` object, the backbone of the DFF API, including most of the relevant fields and methods. + :doc:`Superset guide <./user_guides/superset_guide>` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -22,4 +28,5 @@ Superset dashboard shipped with DFF. :hidden: user_guides/basic_conceptions + user_guides/context_guide user_guides/superset_guide diff --git a/docs/source/user_guides/context_guide.rst b/docs/source/user_guides/context_guide.rst new file mode 100644 index 000000000..7ea0b903e --- /dev/null +++ b/docs/source/user_guides/context_guide.rst @@ -0,0 +1,14 @@ +Context guide +-------------- + +Introduction +~~~~~~~~~~~~ + +A Context is a data structure that is used to store information about the current state of a conversation. +It is used to keep track of the user's input, the current stage of the conversation, and any other +information that is relevant to the current context of a dialog. +The Context provides a convenient interface for working with data, allowing developers to easily add, +retrieve, and manipulate data as the conversation progresses. + +Another important feature of the context is data serialization. +The context can be easily serialized to a format that can be stored or transmitted, such as JSON. \ No newline at end of file From e1067469127ca7dbb846683b33373463226a958b Mon Sep 17 00:00:00 2001 From: ruthenian8 Date: Mon, 2 Oct 2023 14:42:24 +0300 Subject: [PATCH 02/15] update guide --- docs/source/user_guides/context_guide.rst | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/docs/source/user_guides/context_guide.rst b/docs/source/user_guides/context_guide.rst index 7ea0b903e..9f8912456 100644 --- a/docs/source/user_guides/context_guide.rst +++ b/docs/source/user_guides/context_guide.rst @@ -10,5 +10,17 @@ information that is relevant to the current context of a dialog. The Context provides a convenient interface for working with data, allowing developers to easily add, retrieve, and manipulate data as the conversation progresses. -Another important feature of the context is data serialization. -The context can be easily serialized to a format that can be stored or transmitted, such as JSON. \ No newline at end of file +**Serialization** + +The fact that the `Context` class is a Pydantic model makes it easily convertible to other data formats, +such as JSON. For instance, as a developer, you don't need to implement instructions on how datetime fields +need to be marshalled, since this functionality is provided by Pydantic out of the box. +As a result, working with web interfaces and databases that require the transmitted data to be serialized +becomes as easy as calling the `model_dump_json` method: + +.. code:: python + + context = Context() + serialized_context = context.model_dump_json() + +Knowing that, you can easily extend DFF to work with storages like Memcache or web APIs of your liking. \ No newline at end of file From ceab3634b23fc6b4cfc9dc396384867db5e2ec74 Mon Sep 17 00:00:00 2001 From: ruthenian8 Date: Wed, 4 Oct 2023 12:24:53 +0300 Subject: [PATCH 03/15] add attribute and method information --- docs/source/user_guides/context_guide.rst | 72 ++++++++++++++++++++--- 1 file changed, 65 insertions(+), 7 deletions(-) diff --git a/docs/source/user_guides/context_guide.rst b/docs/source/user_guides/context_guide.rst index 9f8912456..4f9400203 100644 --- a/docs/source/user_guides/context_guide.rst +++ b/docs/source/user_guides/context_guide.rst @@ -4,13 +4,71 @@ Context guide Introduction ~~~~~~~~~~~~ -A Context is a data structure that is used to store information about the current state of a conversation. -It is used to keep track of the user's input, the current stage of the conversation, and any other -information that is relevant to the current context of a dialog. -The Context provides a convenient interface for working with data, allowing developers to easily add, -retrieve, and manipulate data as the conversation progresses. +The `Context` class is a backbone component of the DFF API. +Like the name suggests, this data structure is used to store information +about the current state of a particular conversation, +i.e. each individual user has their own context instance and can be identified by it. -**Serialization** +`Context` is used to keep track of the user's requests, bot's replies, +user-related and request-related annotations, and any other information +that is relevant to the conversation with the user. + +.. warning:: + + Since most callback functions used in DFF script and DFF pipeline (see the `basic guide <./basic_conceptions>`_) + need to either read or update the current dialog state, + the framework-level convention is that all functions of this kind + use `Context` as their first parameter. This dependency is being + injected by the pipeline during its run. Thus, understanding + the `Context` class is essential for developing custom conversation logic + that is mostly made up by the said functions. + +`Context` provides a convenient interface for working with data, +allowing developers to easily add, retrieve, +and manipulate data as the conversation progresses. + +Attributes +~~~~~~~~~~~ + +* `id`: This attribute represents the unique context identifier. By default, it is randomly generated using uuid4. The id can be used for tracing user behavior and collecting statistical data. + +* `labels`: The labels attribute stores the history of all passed labels within the conversation. It maps turn IDs to labels. + +* `requests`: The requests attribute maintains the history of all received requests by the agent. It also maps turn IDs to requests. + +* `responses`: This attribute keeps a record of all agent responses, mapping turn IDs to responses. + +* `misc`: The misc attribute is a dictionary for storing custom data. The scripting in DFF doesn't use this dictionary by default. + +* `validation`: A flag that signals whether validation of the script is required during pipeline initialization. Some functions that may produce invalid data must consider this flag for successful validation. + +* `framework_states`: This attribute is used for storing addon or pipeline states. Pipeline records all its intermediate conditions into framework_states, and after context processing is completed, it resets the framework_states and returns the context. + +Key Methods +~~~~~~~~~~~~ + +The `Context` class provides essential methods for working with data: + +* **`add_request(request: Message)`**: Adds a request to the context for the next turn, where `request` is the request message to be added. It updates the `requests` dictionary. + +* **`add_response(response: Message)`**: Adds a response to the context for the next turn, where `response` is the response message to be added. It updates the `responses` dictionary. + +* **`add_label(label: NodeLabel2Type)`**: Adds a label to the context for the next turn, where `label` is the label to be added. It updates the `labels` dictionary. + +* **`clear(hold_last_n_indices: int, field_names: Union[Set[str], List[str]])`**: Clears all recordings from the context, except for the last `hold_last_n_indices` turns. You can specify which fields to clear using the `field_names` parameter. + +* **`last_label`**: Returns the last label of the context, or `None` if the `labels` dictionary is empty. + +* **`last_response`**: Returns the last response of the context, or `None` if the `responses` dictionary is empty. + +* **`last_response` (setter) and `last_request` (setter)**: These methods allow you to set the last response or request for the current context, which is useful for working with response and request wrappers. + +* **`current_node`**: Returns the current node of the context. This is particularly useful for tracking the node during the conversation flow. + +* **`overwrite_current_node_in_processing(processed_node: Node)`**: This method allows you to overwrite the current node with a processed node, but it can only be used within processing functions. + +Serialization +~~~~~~~~~~~~~ The fact that the `Context` class is a Pydantic model makes it easily convertible to other data formats, such as JSON. For instance, as a developer, you don't need to implement instructions on how datetime fields @@ -18,7 +76,7 @@ need to be marshalled, since this functionality is provided by Pydantic out of t As a result, working with web interfaces and databases that require the transmitted data to be serialized becomes as easy as calling the `model_dump_json` method: -.. code:: python +.. code-block:: python context = Context() serialized_context = context.model_dump_json() From 558400f6c8469934cb7a61029fda70c4010b09b9 Mon Sep 17 00:00:00 2001 From: ruthenian8 Date: Wed, 4 Oct 2023 16:59:07 +0300 Subject: [PATCH 04/15] add code-block to introduction --- docs/source/user_guides/context_guide.rst | 86 +++++++++++++++++------ 1 file changed, 63 insertions(+), 23 deletions(-) diff --git a/docs/source/user_guides/context_guide.rst b/docs/source/user_guides/context_guide.rst index 4f9400203..1f4b72af8 100644 --- a/docs/source/user_guides/context_guide.rst +++ b/docs/source/user_guides/context_guide.rst @@ -6,62 +6,102 @@ Introduction The `Context` class is a backbone component of the DFF API. Like the name suggests, this data structure is used to store information -about the current state of a particular conversation, -i.e. each individual user has their own context instance and can be identified by it. +about the current state, or context, of a particular conversation. +Each individual user has their own `Context` instance and can be identified by it. `Context` is used to keep track of the user's requests, bot's replies, user-related and request-related annotations, and any other information that is relevant to the conversation with the user. -.. warning:: +.. note:: Since most callback functions used in DFF script and DFF pipeline (see the `basic guide <./basic_conceptions>`_) need to either read or update the current dialog state, the framework-level convention is that all functions of this kind use `Context` as their first parameter. This dependency is being - injected by the pipeline during its run. Thus, understanding - the `Context` class is essential for developing custom conversation logic - that is mostly made up by the said functions. + injected by the pipeline during its run. + Thus, understanding the `Context` class is essential for developing custom conversation logic + which is mostly made up by the said functions. -`Context` provides a convenient interface for working with data, +As a callback parameter, `Context` provides a convenient interface for working with data, allowing developers to easily add, retrieve, and manipulate data as the conversation progresses. +Let's consider some of the builtin callback instances to see how the context can be leveraged: + +.. code-block:: python + :linenos: + + pattern = re.compile("[a-zA-Z]+") + + def regexp_condition_handler(ctx: Context, pipeline: Pipeline, *args, **kwargs) -> bool: + # retrieve the current request + request = ctx.last_request + if request.text is None: + return False + return bool(pattern.search(request.text)) + +The code above is a condition (see the `basic guide <./basic_conceptions>`_) +that belongs to the `TRANSITIONS` section of the script and returns `True` or `False` +depending on whether the current user request matches the given pattern. +As can be seen from the code block, the current +request can be easily retrieved as one of the attributes of the `Context` object. +Likewise, the `last_response` (bot's current reply) or the `last_label` +(the name of the currently visited node) attributes can be used in the same manner. + Attributes ~~~~~~~~~~~ -* `id`: This attribute represents the unique context identifier. By default, it is randomly generated using uuid4. The id can be used for tracing user behavior and collecting statistical data. +* `id`: This attribute represents the unique context identifier. By default, it is randomly generated using uuid4. + In most cases, this attribute will be used to identify a user -* `labels`: The labels attribute stores the history of all passed labels within the conversation. It maps turn IDs to labels. +* `labels`: The labels attribute stores the history of all passed labels within the conversation. + It maps turn IDs to labels. The collection is ordered, so getting the last item of the mapping + always shows the last visited node. -* `requests`: The requests attribute maintains the history of all received requests by the agent. It also maps turn IDs to requests. +* `requests`: The requests attribute maintains the history of all received requests by the agent. + It also maps turn IDs to requests. Like labels, it stores the requests in-order. -* `responses`: This attribute keeps a record of all agent responses, mapping turn IDs to responses. +* `responses`: This attribute keeps a record of all agent responses, mapping turn IDs to responses + stores the responses in-order. -* `misc`: The misc attribute is a dictionary for storing custom data. The scripting in DFF doesn't use this dictionary by default. +* `misc`: The misc attribute is a dictionary for storing custom data. This field is not used by any of the + built-in DFF classes or functions, so the values that you write there are guaranteed to persist + throughout the lifetime of the `Context` object. -* `validation`: A flag that signals whether validation of the script is required during pipeline initialization. Some functions that may produce invalid data must consider this flag for successful validation. +* `validation`: A flag that signals whether validation of the script is required during pipeline initialization. + It's important to validate custom scripts to ensure that no synthax errors have been made. -* `framework_states`: This attribute is used for storing addon or pipeline states. Pipeline records all its intermediate conditions into framework_states, and after context processing is completed, it resets the framework_states and returns the context. +* `framework_states`: This attribute is used for storing addon or pipeline states. + Each turn, the DFF pipeline records the intermediary states of its components into this field, + clearing it at the end of the turn. -Key Methods -~~~~~~~~~~~~ +Methods +~~~~~~~ -The `Context` class provides essential methods for working with data: +The methods of the `Context` class can be divided into two categories: -* **`add_request(request: Message)`**: Adds a request to the context for the next turn, where `request` is the request message to be added. It updates the `requests` dictionary. +* Public methods that get called manually in custom callbacks and in functions that depend on the context. +* Methods that are not designed for manual calls and get called automatically during pipeline runs, + i.e. quasi-private methods. You may still need them when developing extensions or heavily modifying DFF. -* **`add_response(response: Message)`**: Adds a response to the context for the next turn, where `response` is the response message to be added. It updates the `responses` dictionary. +**Public methods** -* **`add_label(label: NodeLabel2Type)`**: Adds a label to the context for the next turn, where `label` is the label to be added. It updates the `labels` dictionary. +* **`last_label`**: Returns the last label of the context, or `None` if the `labels` dictionary is empty. + +* **`last_response`**: Returns the last response of the context, or `None` if the `responses` dictionary is empty. + +* **`set_last_response` and `set_last_request`**: These methods allow you to set the last response or request for the current context, which is useful for working with response and request wrappers. * **`clear(hold_last_n_indices: int, field_names: Union[Set[str], List[str]])`**: Clears all recordings from the context, except for the last `hold_last_n_indices` turns. You can specify which fields to clear using the `field_names` parameter. -* **`last_label`**: Returns the last label of the context, or `None` if the `labels` dictionary is empty. +**Private methods** -* **`last_response`**: Returns the last response of the context, or `None` if the `responses` dictionary is empty. +* **`add_request(request: Message)`**: Adds a request to the context for the next turn, where `request` is the request message to be added. It updates the `requests` dictionary. -* **`last_response` (setter) and `last_request` (setter)**: These methods allow you to set the last response or request for the current context, which is useful for working with response and request wrappers. +* **`add_response(response: Message)`**: Adds a response to the context for the next turn, where `response` is the response message to be added. It updates the `responses` dictionary. + +* **`add_label(label: NodeLabel2Type)`**: Adds a label to the context for the next turn, where `label` is the label to be added. It updates the `labels` dictionary. * **`current_node`**: Returns the current node of the context. This is particularly useful for tracking the node during the conversation flow. From 5ddf7e128bbfb40cbec5171c0094fb35035b03f8 Mon Sep 17 00:00:00 2001 From: ruthenian8 Date: Thu, 5 Oct 2023 13:42:26 +0300 Subject: [PATCH 05/15] update context guide --- docs/source/user_guides/context_guide.rst | 28 +++++++++++++++++++---- 1 file changed, 24 insertions(+), 4 deletions(-) diff --git a/docs/source/user_guides/context_guide.rst b/docs/source/user_guides/context_guide.rst index 1f4b72af8..2cd752dd0 100644 --- a/docs/source/user_guides/context_guide.rst +++ b/docs/source/user_guides/context_guide.rst @@ -49,6 +49,21 @@ request can be easily retrieved as one of the attributes of the `Context` object Likewise, the `last_response` (bot's current reply) or the `last_label` (the name of the currently visited node) attributes can be used in the same manner. +Another use case is leveraging the `misc` field (see below for a detailed description): +pipeline functions or ``PROCESSING`` callbacks can write arbitrary values to the misc field, +making those available for other context-dependent functions +(see the `pre transitions processing tutorial <../tutorials/tutorials.script.core.9_pre_transitions_processing.py>`_). + +.. code-block:: python + :linenos: + + def save_previous_node_response_to_ctx_processing( + ctx: Context, _: Pipeline, *args, **kwargs + ) -> Context: + processed_node = ctx.current_node + ctx.misc["previous_node_response"] = processed_node.response + return ctx + Attributes ~~~~~~~~~~~ @@ -91,12 +106,19 @@ The methods of the `Context` class can be divided into two categories: * **`last_response`**: Returns the last response of the context, or `None` if the `responses` dictionary is empty. -* **`set_last_response` and `set_last_request`**: These methods allow you to set the last response or request for the current context, which is useful for working with response and request wrappers. +* **`clear(hold_last_n_indices: int, field_names: Union[Set[str], List[str]])`**: Clears all items from context fields, optionally keeping the data from `hold_last_n_indices` turns. + You can specify which fields to clear using the `field_names` parameter. This method is designed for cases + when contexts are shared over high latency networks. -* **`clear(hold_last_n_indices: int, field_names: Union[Set[str], List[str]])`**: Clears all recordings from the context, except for the last `hold_last_n_indices` turns. You can specify which fields to clear using the `field_names` parameter. +* **`overwrite_current_node_in_processing(processed_node: Node)`**: This method allows you to overwrite the current node with a processed node, + but it can only be used within processing functions. This may be required when you need to temporarily substitute the current node: + see `preprocessing tutorial <../tutorials/tutorials.script.core.7_pre_response_processing.py>`_ **Private methods** +* **`set_last_response` and `set_last_request`**: These methods allow you to set the last response or request for the current context. + This functionality can prove useful if you want to create a middleware component that overrides the pipeline functionality. + * **`add_request(request: Message)`**: Adds a request to the context for the next turn, where `request` is the request message to be added. It updates the `requests` dictionary. * **`add_response(response: Message)`**: Adds a response to the context for the next turn, where `response` is the response message to be added. It updates the `responses` dictionary. @@ -105,8 +127,6 @@ The methods of the `Context` class can be divided into two categories: * **`current_node`**: Returns the current node of the context. This is particularly useful for tracking the node during the conversation flow. -* **`overwrite_current_node_in_processing(processed_node: Node)`**: This method allows you to overwrite the current node with a processed node, but it can only be used within processing functions. - Serialization ~~~~~~~~~~~~~ From b966e2616fadfe75ece25a86dd85d13e0c20ad60 Mon Sep 17 00:00:00 2001 From: ruthenian8 Date: Thu, 5 Oct 2023 17:44:07 +0300 Subject: [PATCH 06/15] update references --- docs/source/user_guides/context_guide.rst | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/docs/source/user_guides/context_guide.rst b/docs/source/user_guides/context_guide.rst index 2cd752dd0..fbdc8bf2a 100644 --- a/docs/source/user_guides/context_guide.rst +++ b/docs/source/user_guides/context_guide.rst @@ -41,7 +41,7 @@ Let's consider some of the builtin callback instances to see how the context can return False return bool(pattern.search(request.text)) -The code above is a condition (see the `basic guide <./basic_conceptions>`_) +The code above is a condition function (see the `basic guide <./basic_conceptions>`_) that belongs to the `TRANSITIONS` section of the script and returns `True` or `False` depending on whether the current user request matches the given pattern. As can be seen from the code block, the current @@ -51,8 +51,7 @@ Likewise, the `last_response` (bot's current reply) or the `last_label` Another use case is leveraging the `misc` field (see below for a detailed description): pipeline functions or ``PROCESSING`` callbacks can write arbitrary values to the misc field, -making those available for other context-dependent functions -(see the `pre transitions processing tutorial <../tutorials/tutorials.script.core.9_pre_transitions_processing.py>`_). +making those available for other context-dependent functions. .. code-block:: python :linenos: @@ -64,6 +63,10 @@ making those available for other context-dependent functions ctx.misc["previous_node_response"] = processed_node.response return ctx +.. note:: + + See more in the `pre transitions processing tutorial <../tutorials/tutorials.script.core.9_pre_transitions_processing.py>`_. + Attributes ~~~~~~~~~~~ @@ -111,8 +114,10 @@ The methods of the `Context` class can be divided into two categories: when contexts are shared over high latency networks. * **`overwrite_current_node_in_processing(processed_node: Node)`**: This method allows you to overwrite the current node with a processed node, - but it can only be used within processing functions. This may be required when you need to temporarily substitute the current node: - see `preprocessing tutorial <../tutorials/tutorials.script.core.7_pre_response_processing.py>`_ + but it can only be used within processing functions. This may be required when you need to temporarily substitute the current node. + .. note:: + + see `preprocessing tutorial <../tutorials/tutorials.script.core.7_pre_response_processing.py>`_. **Private methods** From abcb595432dcd367e6ce883525a5c764299b8d8b Mon Sep 17 00:00:00 2001 From: ruthenian8 Date: Fri, 6 Oct 2023 12:07:04 +0300 Subject: [PATCH 07/15] Update field descriptions --- docs/source/user_guides/context_guide.rst | 76 +++++++++++++---------- 1 file changed, 44 insertions(+), 32 deletions(-) diff --git a/docs/source/user_guides/context_guide.rst b/docs/source/user_guides/context_guide.rst index fbdc8bf2a..27798d7c9 100644 --- a/docs/source/user_guides/context_guide.rst +++ b/docs/source/user_guides/context_guide.rst @@ -4,12 +4,12 @@ Context guide Introduction ~~~~~~~~~~~~ -The `Context` class is a backbone component of the DFF API. +The ``Context`` class is a backbone component of the DFF API. Like the name suggests, this data structure is used to store information about the current state, or context, of a particular conversation. -Each individual user has their own `Context` instance and can be identified by it. +Each individual user has their own ``Context`` instance and can be identified by it. -`Context` is used to keep track of the user's requests, bot's replies, +``Context`` is used to keep track of the user's requests, bot's replies, user-related and request-related annotations, and any other information that is relevant to the conversation with the user. @@ -18,12 +18,12 @@ that is relevant to the conversation with the user. Since most callback functions used in DFF script and DFF pipeline (see the `basic guide <./basic_conceptions>`_) need to either read or update the current dialog state, the framework-level convention is that all functions of this kind - use `Context` as their first parameter. This dependency is being + use ``Context`` as their first parameter. This dependency is being injected by the pipeline during its run. - Thus, understanding the `Context` class is essential for developing custom conversation logic + Thus, understanding the ``Context`` class is essential for developing custom conversation logic which is mostly made up by the said functions. -As a callback parameter, `Context` provides a convenient interface for working with data, +As a callback parameter, ``Context`` provides a convenient interface for working with data, allowing developers to easily add, retrieve, and manipulate data as the conversation progresses. @@ -42,14 +42,14 @@ Let's consider some of the builtin callback instances to see how the context can return bool(pattern.search(request.text)) The code above is a condition function (see the `basic guide <./basic_conceptions>`_) -that belongs to the `TRANSITIONS` section of the script and returns `True` or `False` +that belongs to the ``TRANSITIONS`` section of the script and returns `True` or `False` depending on whether the current user request matches the given pattern. As can be seen from the code block, the current -request can be easily retrieved as one of the attributes of the `Context` object. -Likewise, the `last_response` (bot's current reply) or the `last_label` +request (``last_request``) can be easily retrieved as one of the attributes of the ``Context`` object. +Likewise, the ``last_response`` (bot's current reply) or the ``last_label`` (the name of the currently visited node) attributes can be used in the same manner. -Another use case is leveraging the `misc` field (see below for a detailed description): +Another use case is leveraging the ``misc`` field (see below for a detailed description): pipeline functions or ``PROCESSING`` callbacks can write arbitrary values to the misc field, making those available for other context-dependent functions. @@ -70,34 +70,35 @@ making those available for other context-dependent functions. Attributes ~~~~~~~~~~~ -* `id`: This attribute represents the unique context identifier. By default, it is randomly generated using uuid4. +* **id**: This attribute represents the unique context identifier. By default, it is randomly generated using uuid4. In most cases, this attribute will be used to identify a user -* `labels`: The labels attribute stores the history of all passed labels within the conversation. +* **labels**: The labels attribute stores the history of all passed labels within the conversation. It maps turn IDs to labels. The collection is ordered, so getting the last item of the mapping always shows the last visited node. -* `requests`: The requests attribute maintains the history of all received requests by the agent. +* **requests**: The requests attribute maintains the history of all received requests by the agent. It also maps turn IDs to requests. Like labels, it stores the requests in-order. -* `responses`: This attribute keeps a record of all agent responses, mapping turn IDs to responses +* **responses**: This attribute keeps a record of all agent responses, mapping turn IDs to responses stores the responses in-order. -* `misc`: The misc attribute is a dictionary for storing custom data. This field is not used by any of the +* **misc**: The misc attribute is a dictionary for storing custom data. This field is not used by any of the built-in DFF classes or functions, so the values that you write there are guaranteed to persist - throughout the lifetime of the `Context` object. + throughout the lifetime of the ``Context`` object. -* `validation`: A flag that signals whether validation of the script is required during pipeline initialization. +* **validation**: A flag that signals whether validation of the script is required during pipeline initialization. It's important to validate custom scripts to ensure that no synthax errors have been made. -* `framework_states`: This attribute is used for storing addon or pipeline states. +* **framework_states**: This attribute is used for storing addon or pipeline states. Each turn, the DFF pipeline records the intermediary states of its components into this field, - clearing it at the end of the turn. + and clears it at the end of the turn. For this reason, developers are discouraged from storing + their own data in this field. Methods ~~~~~~~ -The methods of the `Context` class can be divided into two categories: +The methods of the ``Context`` class can be divided into two categories: * Public methods that get called manually in custom callbacks and in functions that depend on the context. * Methods that are not designed for manual calls and get called automatically during pipeline runs, @@ -105,37 +106,48 @@ The methods of the `Context` class can be divided into two categories: **Public methods** -* **`last_label`**: Returns the last label of the context, or `None` if the `labels` dictionary is empty. +* **last_request**: Returns the last label of the context, or `None` if the ``requests`` field is empty. -* **`last_response`**: Returns the last response of the context, or `None` if the `responses` dictionary is empty. +* **last_response**: Returns the last response of the context, or `None` if the ``responses`` field is empty. -* **`clear(hold_last_n_indices: int, field_names: Union[Set[str], List[str]])`**: Clears all items from context fields, optionally keeping the data from `hold_last_n_indices` turns. +* **last_label**: Returns the last label of the context, or `None` if the ``labels`` field is empty. + +* **clear**: Clears all items from context fields, optionally keeping the data from `hold_last_n_indices` turns. You can specify which fields to clear using the `field_names` parameter. This method is designed for cases when contexts are shared over high latency networks. -* **`overwrite_current_node_in_processing(processed_node: Node)`**: This method allows you to overwrite the current node with a processed node, +* **overwrite_current_node_in_processing**: This method allows you to overwrite the current node with a processed node, but it can only be used within processing functions. This may be required when you need to temporarily substitute the current node. - .. note:: - see `preprocessing tutorial <../tutorials/tutorials.script.core.7_pre_response_processing.py>`_. +.. note:: + + see the `preprocessing tutorial <../tutorials/tutorials.script.core.7_pre_response_processing.py>`_. **Private methods** -* **`set_last_response` and `set_last_request`**: These methods allow you to set the last response or request for the current context. +* **set_last_response, set_last_request**: These methods allow you to set the last response or request for the current context. This functionality can prove useful if you want to create a middleware component that overrides the pipeline functionality. -* **`add_request(request: Message)`**: Adds a request to the context for the next turn, where `request` is the request message to be added. It updates the `requests` dictionary. +* **add_request**: Adds a request to the context for the next turn, where `request` is the request message to be added. + It updates the `requests` dictionary. This method is called by the `Pipeline` component + before any of the `pipeline services <../tutorials/tutorials.pipeline.3_pipeline_dict_with_services_basic.py>`_ are executed, + including `Actor <../apiref/dff.pipeline.pipeline.actor.html>`_. -* **`add_response(response: Message)`**: Adds a response to the context for the next turn, where `response` is the response message to be added. It updates the `responses` dictionary. +* **add_response**: Adds a response to the context for the next turn, where `response` is the response message to be added. + It updates the `responses` dictionary. This function is run by the `Actor <../apiref/dff.pipeline.pipeline.actor.html>`_ pipeline component at the end of the turn, after it has run + the `PRE_RESPONSE_PROCESSING <../tutorials/tutorials.script.core.7_pre_response_processing.py>`_ functions. -* **`add_label(label: NodeLabel2Type)`**: Adds a label to the context for the next turn, where `label` is the label to be added. It updates the `labels` dictionary. +* **add_label**: Adds a label to the context for the next turn, where `label` is the label to be added. + It updates the `labels` field. This method is called by the `Actor <../apiref/dff.pipeline.pipeline.actor.html>`_ component when transition conditions + have been resolved, and when `PRE_TRANSITIONS_PROCESSING <../tutorials/tutorials.script.core.9_pre_transitions_processing.py>`_ callbacks have been run. -* **`current_node`**: Returns the current node of the context. This is particularly useful for tracking the node during the conversation flow. +* **current_node**: Returns the current node of the context. This is particularly useful for tracking the node during the conversation flow. + This method only returns a node inside ``PROCESSING`` callbacks yielding ``None`` in other contexts. Serialization ~~~~~~~~~~~~~ -The fact that the `Context` class is a Pydantic model makes it easily convertible to other data formats, +The fact that the ``Context`` class is a Pydantic model makes it easily convertible to other data formats, such as JSON. For instance, as a developer, you don't need to implement instructions on how datetime fields need to be marshalled, since this functionality is provided by Pydantic out of the box. As a result, working with web interfaces and databases that require the transmitted data to be serialized From a09ae121486b0d16250dfac2b46caa55d0d16836 Mon Sep 17 00:00:00 2001 From: Roman Zlobin Date: Mon, 16 Oct 2023 01:55:07 +0300 Subject: [PATCH 08/15] remove unused function in a referenced tutorial --- tutorials/script/core/7_pre_response_processing.py | 7 ------- 1 file changed, 7 deletions(-) diff --git a/tutorials/script/core/7_pre_response_processing.py b/tutorials/script/core/7_pre_response_processing.py index d43e9c17d..233d62f75 100644 --- a/tutorials/script/core/7_pre_response_processing.py +++ b/tutorials/script/core/7_pre_response_processing.py @@ -37,13 +37,6 @@ # %% -def add_label_processing(ctx: Context, _: Pipeline, *args, **kwargs) -> Context: - processed_node = ctx.current_node - processed_node.response = Message(text=f"{ctx.last_label}: {processed_node.response.text}") - ctx.overwrite_current_node_in_processing(processed_node) - return ctx - - def add_prefix(prefix): def add_prefix_processing(ctx: Context, _: Pipeline, *args, **kwargs) -> Context: processed_node = ctx.current_node From 2ccfbb8bd06b427f0cf66b14717ddfce2cf28219 Mon Sep 17 00:00:00 2001 From: Roman Zlobin Date: Mon, 16 Oct 2023 03:33:54 +0300 Subject: [PATCH 09/15] fix context doc --- dff/script/core/context.py | 97 +++++++++++++++++++++----------------- 1 file changed, 54 insertions(+), 43 deletions(-) diff --git a/dff/script/core/context.py b/dff/script/core/context.py index 6658c346f..78ee18072 100644 --- a/dff/script/core/context.py +++ b/dff/script/core/context.py @@ -32,7 +32,7 @@ def get_last_index(dictionary: dict) -> int: """ - Obtaining the last index from the `dictionary`. Functions returns `-1` if the `dict` is empty. + Obtain the last index from the `dictionary`. Return `-1` if the `dict` is empty. :param dictionary: Dictionary with unsorted keys. :return: Last index from the `dictionary`. @@ -44,6 +44,9 @@ def get_last_index(dictionary: dict) -> int: class Context(BaseModel): """ A structure that is used to store data about the context of a dialog. + + Avoid storing unserializable data in the fields of this class in order for + context storages to work. """ id: Union[UUID, int, str] = Field(default_factory=uuid4) @@ -77,13 +80,15 @@ class Context(BaseModel): `misc` stores any custom data. The scripting doesn't use this dictionary by default, so storage of any data won't reflect on the work on the internal Dialog Flow Scripting functions. + Avoid storing unserializable data in order for context storages to work. + - key - Arbitrary data name. - value - Arbitrary data. """ validation: bool = False """ - `validation` is a flag that signals that :py:class:`~dff.script.Pipeline`, - while being initialized, checks the :py:class:`~dff.script.Script`. + `validation` is a flag that signals that :py:class:`~dff.pipeline.pipeline.pipeline.Pipeline`, + while being initialized, checks the :py:class:`~dff.script.core.script.Script`. The functions that can give not valid data while being validated must use this flag to take the validation mode into account. Otherwise the validation will not be passed. @@ -91,12 +96,12 @@ class Context(BaseModel): framework_states: Dict[ModuleName, Dict[str, Any]] = {} """ `framework_states` is used for addons states or for - :py:class:`~dff.script.Pipeline`'s states. - :py:class:`~dff.script.Pipeline` + :py:class:`~dff.pipeline.pipeline.pipeline.Pipeline`'s states. + :py:class:`~dff.pipeline.pipeline.pipeline.Pipeline` records all its intermediate conditions into the `framework_states`. - After :py:class:`~dff.script.Context` processing is finished, - :py:class:`~dff.script.Pipeline` resets `framework_states` and - returns :py:class:`~dff.script.Context`. + After :py:class:`~.Context` processing is finished, + :py:class:`~dff.pipeline.pipeline.pipeline.Pipeline` resets `framework_states` and + returns :py:class:`~.Context`. - key - Temporary variable name. - value - Temporary variable data. @@ -106,7 +111,7 @@ class Context(BaseModel): @classmethod def sort_dict_keys(cls, dictionary: dict) -> dict: """ - Sorting the keys in the `dictionary`. This needs to be done after deserialization, + Sort the keys in the `dictionary`. This needs to be done after deserialization, since the keys are deserialized in a random order. :param dictionary: Dictionary with unsorted keys. @@ -117,16 +122,15 @@ def sort_dict_keys(cls, dictionary: dict) -> dict: @classmethod def cast(cls, ctx: Optional[Union["Context", dict, str]] = None, *args, **kwargs) -> "Context": """ - Transforms different data types to the objects of - :py:class:`~dff.script.Context` class. - Returns an object of :py:class:`~dff.script.Context` + Transform different data types to the objects of the + :py:class:`~.Context` class. + Return an object of the :py:class:`~.Context` type that is initialized by the input data. - :param ctx: Different data types, that are used to initialize object of - :py:class:`~dff.script.Context` type. - The empty object of :py:class:`~dff.script.Context` - type is created if no data are given. - :return: Object of :py:class:`~dff.script.Context` + :param ctx: Data that is used to initialize an object of the + :py:class:`~.Context` type. + An empty :py:class:`~.Context` object is returned if no data is given. + :return: Object of the :py:class:`~.Context` type that is initialized by the input data. """ if not ctx: @@ -137,14 +141,15 @@ def cast(cls, ctx: Optional[Union["Context", dict, str]] = None, *args, **kwargs ctx = Context.model_validate_json(ctx) elif not issubclass(type(ctx), Context): raise ValueError( - f"context expected as sub class of Context class or object of dict/str(json) type, but got {ctx}" + f"Context expected to be an instance of the Context class " + f"or an instance of the dict/str(json) type. Got: {type(ctx)}" ) return ctx def add_request(self, request: Message): """ - Adds to the context the next `request` corresponding to the next turn. - The addition takes place in the `requests` and `new_index = last_index + 1`. + Add a new `request` to the context. + The new `request` is added with the index of `last_index + 1`. :param request: `request` to be added to the context. """ @@ -154,8 +159,8 @@ def add_request(self, request: Message): def add_response(self, response: Message): """ - Adds to the context the next `response` corresponding to the next turn. - The addition takes place in the `responses`, and `new_index = last_index + 1`. + Add a new `response` to the context. + The new `response` is added with the index of `last_index + 1`. :param response: `response` to be added to the context. """ @@ -165,9 +170,8 @@ def add_response(self, response: Message): def add_label(self, label: NodeLabel2Type): """ - Adds to the context the next :py:const:`label `, - corresponding to the next turn. - The addition takes place in the `labels`, and `new_index = last_index + 1`. + Add a new :py:data:`~.NodeLabel2Type` to the context. + The new `label` is added with the index of `last_index + 1`. :param label: `label` that we need to add to the context. """ @@ -180,12 +184,12 @@ def clear( field_names: Union[Set[str], List[str]] = {"requests", "responses", "labels"}, ): """ - Deletes all recordings from the `requests`/`responses`/`labels` except for + Delete all records from the `requests`/`responses`/`labels` except for the last `hold_last_n_indices` turns. If `field_names` contains `misc` field, `misc` field is fully cleared. - :param hold_last_n_indices: Number of last turns that remain under clearing. - :param field_names: Properties of :py:class:`~dff.script.Context` we need to clear. + :param hold_last_n_indices: Number of last turns to keep. + :param field_names: Properties of :py:class:`~.Context` to clear. Defaults to {"requests", "responses", "labels"} """ field_names = field_names if isinstance(field_names, set) else set(field_names) @@ -206,9 +210,12 @@ def clear( @property def last_label(self) -> Optional[NodeLabel2Type]: """ - Returns the last :py:const:`~dff.script.NodeLabel2Type` of - the :py:class:`~dff.script.Context`. - Returns `None` if `labels` is empty. + Return the last :py:data:`~.NodeLabel2Type` of + the :py:class:`~.Context`. + Return `None` if `labels` is empty. + + Since `start_label` is not added to the `labels` field, + empty `labels` usually indicates that the current node is the `start_node`. """ last_index = get_last_index(self.labels) return self.labels.get(last_index) @@ -216,8 +223,8 @@ def last_label(self) -> Optional[NodeLabel2Type]: @property def last_response(self) -> Optional[Message]: """ - Returns the last `response` of the current :py:class:`~dff.script.Context`. - Returns `None` if `responses` is empty. + Return the last `response` of the current :py:class:`~.Context`. + Return `None` if `responses` is empty. """ last_index = get_last_index(self.responses) return self.responses.get(last_index) @@ -225,7 +232,7 @@ def last_response(self) -> Optional[Message]: @last_response.setter def last_response(self, response: Optional[Message]): """ - Sets the last `response` of the current :py:class:`~dff.core.engine.core.context.Context`. + Set the last `response` of the current :py:class:`~.Context`. Required for use with various response wrappers. """ last_index = get_last_index(self.responses) @@ -234,8 +241,8 @@ def last_response(self, response: Optional[Message]): @property def last_request(self) -> Optional[Message]: """ - Returns the last `request` of the current :py:class:`~dff.script.Context`. - Returns `None` if `requests` is empty. + Return the last `request` of the current :py:class:`~.Context`. + Return `None` if `requests` is empty. """ last_index = get_last_index(self.requests) return self.requests.get(last_index) @@ -243,7 +250,7 @@ def last_request(self) -> Optional[Message]: @last_request.setter def last_request(self, request: Optional[Message]): """ - Sets the last `request` of the current :py:class:`~dff.core.engine.core.context.Context`. + Set the last `request` of the current :py:class:`~.Context`. Required for use with various request wrappers. """ last_index = get_last_index(self.requests) @@ -252,7 +259,7 @@ def last_request(self, request: Optional[Message]): @property def current_node(self) -> Optional[Node]: """ - Returns current :py:class:`~dff.script.Node`. + Return current :py:class:`~dff.script.core.script.Node`. """ actor = self.framework_states.get("actor", {}) node = ( @@ -264,17 +271,21 @@ def current_node(self) -> Optional[Node]: ) if node is None: logger.warning( - "The `current_node` exists when an actor is running between `ActorStage.GET_PREVIOUS_NODE`" - " and `ActorStage.FINISH_TURN`" + "The `current_node` method should be called " + "when an actor is running between the " + "`ActorStage.GET_PREVIOUS_NODE` and `ActorStage.FINISH_TURN` stages." ) return node def overwrite_current_node_in_processing(self, processed_node: Node): """ - Overwrites the current node with a processed node. This method only works in processing functions. + Set the current node to be `processed_node`. + This method only works in processing functions (pre-response and pre-transition). + + The actual current node is not changed. - :param processed_node: `node` that we need to overwrite current node. + :param processed_node: `node` to set as the current node. """ is_processing = self.framework_states.get("actor", {}).get("processed_node") if is_processing: @@ -282,7 +293,7 @@ def overwrite_current_node_in_processing(self, processed_node: Node): else: logger.warning( f"The `{self.overwrite_current_node_in_processing.__name__}` " - "function can only be run during processing functions." + "method can only be called from processing functions (either pre-response or pre-transition)." ) From d9f825004f64c6a766437a9a1dd25e939619623f Mon Sep 17 00:00:00 2001 From: Roman Zlobin Date: Mon, 16 Oct 2023 03:37:06 +0300 Subject: [PATCH 10/15] add links to the context guide in the basic guide --- docs/source/user_guides/basic_conceptions.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/source/user_guides/basic_conceptions.rst b/docs/source/user_guides/basic_conceptions.rst index 9f81a8610..6cd4d2964 100644 --- a/docs/source/user_guides/basic_conceptions.rst +++ b/docs/source/user_guides/basic_conceptions.rst @@ -241,7 +241,7 @@ That's what we've changed: .. note:: - See `documentation of Context object`_. + See `guide on Context objects`_. * Transitions were changed: transitions to next, previous and current node were replaced with special standard transitions. @@ -268,7 +268,7 @@ For example: * You can serialize context (available on every transition and response) to json or dictionary in order to debug it or extract some values. - See `tutorial on context serialization`_. + See `guide on context serialization`_. * You can alter user input and modify generated responses. User input can be altered with ``PRE_RESPONSE_PROCESSING`` and will happen **before** response generation. @@ -293,11 +293,11 @@ Happy building! .. _tutorial on basic dialog structure: https://deeppavlov.github.io/dialog_flow_framework/tutorials/tutorials.script.core.1_basics.html .. _tutorial on response functions: https://deeppavlov.github.io/dialog_flow_framework/tutorials/tutorials.script.core.3_responses.html -.. _documentation of Context object: https://deeppavlov.github.io/dialog_flow_framework/apiref/dff.script.core.context.html +.. _guide on Context objects: ../user_guides/context_guide.html .. _tutorial on transitions: https://deeppavlov.github.io/dialog_flow_framework/tutorials/tutorials.script.core.4_transitions.html .. _tutorial on conditions: https://deeppavlov.github.io/dialog_flow_framework/tutorials/tutorials.script.core.2_conditions.html .. _tutorial on global transitions: https://deeppavlov.github.io/dialog_flow_framework/tutorials/tutorials.script.core.5_global_transitions.html -.. _tutorial on context serialization: https://deeppavlov.github.io/dialog_flow_framework/tutorials/tutorials.script.core.6_context_serialization.html +.. _guide on context serialization: ../user_guides/context_guide.html#serialization .. _tutorial on pre-response processing: https://deeppavlov.github.io/dialog_flow_framework/tutorials/tutorials.script.core.7_pre_response_processing.html .. _tutorial on pre-transition processing: https://deeppavlov.github.io/dialog_flow_framework/tutorials/tutorials.script.core.9_pre_transitions_processing.html .. _tutorial on script MISC: https://deeppavlov.github.io/dialog_flow_framework/tutorials/tutorials.script.core.8_misc.html From 802309fb1af4c22a20d43c938cee31560b6178a5 Mon Sep 17 00:00:00 2001 From: Roman Zlobin Date: Mon, 16 Oct 2023 03:49:07 +0300 Subject: [PATCH 11/15] fix context guide Notable changes: - make links anonymous - Replace misc function - Add more information about fields --- docs/source/user_guides.rst | 2 +- docs/source/user_guides/context_guide.rst | 94 +++++++++++++++-------- 2 files changed, 65 insertions(+), 31 deletions(-) diff --git a/docs/source/user_guides.rst b/docs/source/user_guides.rst index 866d5b45e..fa274db88 100644 --- a/docs/source/user_guides.rst +++ b/docs/source/user_guides.rst @@ -12,7 +12,7 @@ about current script execution. :doc:`Context guide <./user_guides/context_guide>` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -The ``context tutorial`` walks you through the details of working with the +The ``context guide`` walks you through the details of working with the ``Context`` object, the backbone of the DFF API, including most of the relevant fields and methods. :doc:`Superset guide <./user_guides/superset_guide>` diff --git a/docs/source/user_guides/context_guide.rst b/docs/source/user_guides/context_guide.rst index 27798d7c9..8e878704f 100644 --- a/docs/source/user_guides/context_guide.rst +++ b/docs/source/user_guides/context_guide.rst @@ -15,7 +15,7 @@ that is relevant to the conversation with the user. .. note:: - Since most callback functions used in DFF script and DFF pipeline (see the `basic guide <./basic_conceptions>`_) + Since most callback functions used in DFF script and DFF pipeline (see the `basic guide <./basic_conceptions.rst>`__) need to either read or update the current dialog state, the framework-level convention is that all functions of this kind use ``Context`` as their first parameter. This dependency is being @@ -27,21 +27,23 @@ As a callback parameter, ``Context`` provides a convenient interface for working allowing developers to easily add, retrieve, and manipulate data as the conversation progresses. -Let's consider some of the builtin callback instances to see how the context can be leveraged: +Let's consider some of the built-in callback instances to see how the context can be leveraged: .. code-block:: python :linenos: pattern = re.compile("[a-zA-Z]+") - def regexp_condition_handler(ctx: Context, pipeline: Pipeline, *args, **kwargs) -> bool: + def regexp_condition_handler( + ctx: Context, pipeline: Pipeline, *args, **kwargs + ) -> bool: # retrieve the current request request = ctx.last_request if request.text is None: return False return bool(pattern.search(request.text)) -The code above is a condition function (see the `basic guide <./basic_conceptions>`_) +The code above is a condition function (see the `basic guide <./basic_conceptions.rst>`__) that belongs to the ``TRANSITIONS`` section of the script and returns `True` or `False` depending on whether the current user request matches the given pattern. As can be seen from the code block, the current @@ -49,39 +51,54 @@ request (``last_request``) can be easily retrieved as one of the attributes of t Likewise, the ``last_response`` (bot's current reply) or the ``last_label`` (the name of the currently visited node) attributes can be used in the same manner. -Another use case is leveraging the ``misc`` field (see below for a detailed description): +Another common use case is leveraging the ``misc`` field (see below for a detailed description): pipeline functions or ``PROCESSING`` callbacks can write arbitrary values to the misc field, making those available for other context-dependent functions. .. code-block:: python :linenos: - def save_previous_node_response_to_ctx_processing( - ctx: Context, _: Pipeline, *args, **kwargs - ) -> Context: - processed_node = ctx.current_node - ctx.misc["previous_node_response"] = processed_node.response - return ctx + import urllib.request + import urllib.error -.. note:: + def ping_example_com( + ctx: Context, *_, **__ + ): + try: + with urllib.request.urlopen("https://example.com/") as webpage: + web_content = webpage.read().decode( + webpage.headers.get_content_charset() + ) + result = "Example Domain" in web_content + except urllib.error.URLError: + result = False + ctx.misc["can_ping_example_com"] = result + +.. + todo: link to the user defined functions tutorial + + .. note:: + For more information about user-defined functions see the `user functions guide <./user_functions.rst>`__. - See more in the `pre transitions processing tutorial <../tutorials/tutorials.script.core.9_pre_transitions_processing.py>`_. Attributes ~~~~~~~~~~~ * **id**: This attribute represents the unique context identifier. By default, it is randomly generated using uuid4. - In most cases, this attribute will be used to identify a user + In most cases, this attribute will be used to identify a user. * **labels**: The labels attribute stores the history of all passed labels within the conversation. It maps turn IDs to labels. The collection is ordered, so getting the last item of the mapping always shows the last visited node. + Note that `labels` only stores the nodes that were transitioned to + so `start_label` will not be in this attribute. + * **requests**: The requests attribute maintains the history of all received requests by the agent. It also maps turn IDs to requests. Like labels, it stores the requests in-order. -* **responses**: This attribute keeps a record of all agent responses, mapping turn IDs to responses - stores the responses in-order. +* **responses**: This attribute keeps a record of all agent responses, mapping turn IDs to responses. + Stores the responses in-order. * **misc**: The misc attribute is a dictionary for storing custom data. This field is not used by any of the built-in DFF classes or functions, so the values that you write there are guaranteed to persist @@ -106,13 +123,24 @@ The methods of the ``Context`` class can be divided into two categories: **Public methods** -* **last_request**: Returns the last label of the context, or `None` if the ``requests`` field is empty. +* **last_request**: Return the last request of the context, or `None` if the ``requests`` field is empty. + + Note that a request is added right after the context is created/retrieved from db, + so an empty ``requests`` field usually indicates an issue with the messenger interface. -* **last_response**: Returns the last response of the context, or `None` if the ``responses`` field is empty. +* **last_response**: Return the last response of the context, or `None` if the ``responses`` field is empty. -* **last_label**: Returns the last label of the context, or `None` if the ``labels`` field is empty. + Responses are added at the end of each turn, so an empty ``response`` field is something you should definitely consider. -* **clear**: Clears all items from context fields, optionally keeping the data from `hold_last_n_indices` turns. +* **last_label**: Return the last label of the context, or `None` if the ``labels`` field is empty. + Last label is always the name of the current node but not vice versa: + + Since ``start_label`` is not added to the ``labels`` field, + empty ``labels`` usually indicates that the current node is the `start_node`. + After a transition is made from the `start_node` + the label of that transition is added to the field. + +* **clear**: Clear all items from context fields, optionally keeping the data from `hold_last_n_indices` turns. You can specify which fields to clear using the `field_names` parameter. This method is designed for cases when contexts are shared over high latency networks. @@ -121,27 +149,33 @@ The methods of the ``Context`` class can be divided into two categories: .. note:: - see the `preprocessing tutorial <../tutorials/tutorials.script.core.7_pre_response_processing.py>`_. + See the `preprocessing tutorial <../tutorials/tutorials.script.core.7_pre_response_processing.py>`__. **Private methods** * **set_last_response, set_last_request**: These methods allow you to set the last response or request for the current context. This functionality can prove useful if you want to create a middleware component that overrides the pipeline functionality. -* **add_request**: Adds a request to the context for the next turn, where `request` is the request message to be added. +* **add_request**: Add a request to the context. It updates the `requests` dictionary. This method is called by the `Pipeline` component - before any of the `pipeline services <../tutorials/tutorials.pipeline.3_pipeline_dict_with_services_basic.py>`_ are executed, - including `Actor <../apiref/dff.pipeline.pipeline.actor.html>`_. + before any of the `pipeline services <../tutorials/tutorials.pipeline.3_pipeline_dict_with_services_basic.py>`__ are executed, + including `Actor <../apiref/dff.pipeline.pipeline.actor.html>`__. -* **add_response**: Adds a response to the context for the next turn, where `response` is the response message to be added. - It updates the `responses` dictionary. This function is run by the `Actor <../apiref/dff.pipeline.pipeline.actor.html>`_ pipeline component at the end of the turn, after it has run - the `PRE_RESPONSE_PROCESSING <../tutorials/tutorials.script.core.7_pre_response_processing.py>`_ functions. +* **add_response**: Add a response to the context. + It updates the `responses` dictionary. This function is run by the `Actor <../apiref/dff.pipeline.pipeline.actor.html>`__ pipeline component at the end of the turn, after it has run + the `PRE_RESPONSE_PROCESSING <../tutorials/tutorials.script.core.7_pre_response_processing.py>`__ functions. -* **add_label**: Adds a label to the context for the next turn, where `label` is the label to be added. + To be more precise, this method is called between the ``CREATE_RESPONSE`` and ``FINISH_TURN`` stages. + For more information about stages, see `ActorStages <../apiref/dff.script.core.types.html#dff.script.core.types.ActorStage>`__. + +* **add_label**: Add a label to the context. It updates the `labels` field. This method is called by the `Actor <../apiref/dff.pipeline.pipeline.actor.html>`_ component when transition conditions - have been resolved, and when `PRE_TRANSITIONS_PROCESSING <../tutorials/tutorials.script.core.9_pre_transitions_processing.py>`_ callbacks have been run. + have been resolved, and when `PRE_TRANSITIONS_PROCESSING <../tutorials/tutorials.script.core.9_pre_transitions_processing.py>`__ callbacks have been run. + + To be more precise, this method is called between the ``GET_NEXT_NODE`` and ``REWRITE_NEXT_NODE`` stages. + For more information about stages, see `ActorStages <../apiref/dff.script.core.types.html#dff.script.core.types.ActorStage>`__. -* **current_node**: Returns the current node of the context. This is particularly useful for tracking the node during the conversation flow. +* **current_node**: Return the current node of the context. This is particularly useful for tracking the node during the conversation flow. This method only returns a node inside ``PROCESSING`` callbacks yielding ``None`` in other contexts. Serialization From 2766cc63fef59171bc019144c5bb03533e9598ca Mon Sep 17 00:00:00 2001 From: ruthenian8 Date: Mon, 16 Oct 2023 12:30:48 +0300 Subject: [PATCH 12/15] add context storage section --- docs/source/user_guides/context_guide.rst | 49 +++++++++++++++++++++++ 1 file changed, 49 insertions(+) diff --git a/docs/source/user_guides/context_guide.rst b/docs/source/user_guides/context_guide.rst index 8e878704f..21c81ddeb 100644 --- a/docs/source/user_guides/context_guide.rst +++ b/docs/source/user_guides/context_guide.rst @@ -178,6 +178,55 @@ The methods of the ``Context`` class can be divided into two categories: * **current_node**: Return the current node of the context. This is particularly useful for tracking the node during the conversation flow. This method only returns a node inside ``PROCESSING`` callbacks yielding ``None`` in other contexts. +Context storages +~~~~~~~~~~~~~~~~ + +Since context instances contain all the information, relevant for a particular user, there needs to be a way +to persistently store that information and to make it accessible in different user sessions. +This functionality is implemented by the ``context storages`` module that provides +the uniform ``DBContextStorage`` interface as well as child classes thereof, that integrate +various database types (see the +`api reference <../apiref/dff.context_storages.database.html#dff.context_storages.database.DBContextStorage>`_). + +The supported storage options and are as follows: + +* `JSON `_ +* `pickle `_ +* `shelve `_ +* `SQLite `_ +* `PostgreSQL `_ +* `MySQL `_ +* `MongoDB `_ +* `Redis `_ +* `Yandex DataBase `_ + +``DBContextStorage`` instances can be uniformly constructed using the ``context_storage_factory`` function. +The function's only parameter is a connection string that specifies both the database type +and the connection parameters, as in *mongodb://admin:pass@localhost:27016/admin*. +(`see the reference <../apiref/dff.context_storages.database.html#dff.context_storages.database.DBContextStorage>`_) + +The GitHub-based distribution of DFF includes Docker images for each of the supported database types. +Therefore, the easiest way to deploy your service together with a database is to clone the GitHub +distribution and to take advantage of the packaged +`docker-compose file `_. + +.. code-block:: shell + :linenos: + + git clone https://github.com/deeppavlov/dialog_flow_framework.git + cd dialog_flow_framework + # assuming we need to deploy mongodb + docker-compose up mongo + +The images can be configured using the docker-compose file or the +`environment file `_, +also available in the distribution. Consult these files for more options. + +.. warning:: + + The data transmission protocols require the data to be JSON-serializeable. DFF tackles this problem + through utilization of ``pydantic`` as described in the next section. + Serialization ~~~~~~~~~~~~~ From af837b47174e41e47c902e30dd3b0c48acb157dc Mon Sep 17 00:00:00 2001 From: Roman Zlobin Date: Mon, 16 Oct 2023 18:59:12 +0300 Subject: [PATCH 13/15] remove comments about to-be-deprecated methods --- docs/source/user_guides/context_guide.rst | 6 ------ 1 file changed, 6 deletions(-) diff --git a/docs/source/user_guides/context_guide.rst b/docs/source/user_guides/context_guide.rst index 21c81ddeb..29dcd44d7 100644 --- a/docs/source/user_guides/context_guide.rst +++ b/docs/source/user_guides/context_guide.rst @@ -104,9 +104,6 @@ Attributes built-in DFF classes or functions, so the values that you write there are guaranteed to persist throughout the lifetime of the ``Context`` object. -* **validation**: A flag that signals whether validation of the script is required during pipeline initialization. - It's important to validate custom scripts to ensure that no synthax errors have been made. - * **framework_states**: This attribute is used for storing addon or pipeline states. Each turn, the DFF pipeline records the intermediary states of its components into this field, and clears it at the end of the turn. For this reason, developers are discouraged from storing @@ -144,9 +141,6 @@ The methods of the ``Context`` class can be divided into two categories: You can specify which fields to clear using the `field_names` parameter. This method is designed for cases when contexts are shared over high latency networks. -* **overwrite_current_node_in_processing**: This method allows you to overwrite the current node with a processed node, - but it can only be used within processing functions. This may be required when you need to temporarily substitute the current node. - .. note:: See the `preprocessing tutorial <../tutorials/tutorials.script.core.7_pre_response_processing.py>`__. From 559ac70d7c5751aa69e650e45b4a0df06e48c47d Mon Sep 17 00:00:00 2001 From: Roman Zlobin Date: Mon, 16 Oct 2023 23:31:48 +0300 Subject: [PATCH 14/15] merge attributes and methods into one section --- docs/source/user_guides/context_guide.rst | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/docs/source/user_guides/context_guide.rst b/docs/source/user_guides/context_guide.rst index 29dcd44d7..7a54db5d1 100644 --- a/docs/source/user_guides/context_guide.rst +++ b/docs/source/user_guides/context_guide.rst @@ -80,9 +80,16 @@ making those available for other context-dependent functions. .. note:: For more information about user-defined functions see the `user functions guide <./user_functions.rst>`__. +API +~~~ + +This sections describes the API of the ``Context`` class. + +For more information, such as method signatures, see +`API reference <./apiref/dff.script.core.context.html#dff.script.core.context.Context>`__. Attributes -~~~~~~~~~~~ +========== * **id**: This attribute represents the unique context identifier. By default, it is randomly generated using uuid4. In most cases, this attribute will be used to identify a user. @@ -110,7 +117,7 @@ Attributes their own data in this field. Methods -~~~~~~~ +======= The methods of the ``Context`` class can be divided into two categories: @@ -118,7 +125,8 @@ The methods of the ``Context`` class can be divided into two categories: * Methods that are not designed for manual calls and get called automatically during pipeline runs, i.e. quasi-private methods. You may still need them when developing extensions or heavily modifying DFF. -**Public methods** +Public methods +^^^^^^^^^^^^^^ * **last_request**: Return the last request of the context, or `None` if the ``requests`` field is empty. @@ -145,7 +153,8 @@ The methods of the ``Context`` class can be divided into two categories: See the `preprocessing tutorial <../tutorials/tutorials.script.core.7_pre_response_processing.py>`__. -**Private methods** +Private methods +^^^^^^^^^^^^^^^ * **set_last_response, set_last_request**: These methods allow you to set the last response or request for the current context. This functionality can prove useful if you want to create a middleware component that overrides the pipeline functionality. From af9a64945257b3052f1939d2bb5a00b22963b387 Mon Sep 17 00:00:00 2001 From: Roman Zlobin Date: Mon, 16 Oct 2023 23:32:09 +0300 Subject: [PATCH 15/15] fixes to the context storage section --- docs/source/user_guides/context_guide.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/source/user_guides/context_guide.rst b/docs/source/user_guides/context_guide.rst index 7a54db5d1..1dc47712c 100644 --- a/docs/source/user_guides/context_guide.rst +++ b/docs/source/user_guides/context_guide.rst @@ -187,11 +187,11 @@ Context storages Since context instances contain all the information, relevant for a particular user, there needs to be a way to persistently store that information and to make it accessible in different user sessions. This functionality is implemented by the ``context storages`` module that provides -the uniform ``DBContextStorage`` interface as well as child classes thereof, that integrate +the uniform ``DBContextStorage`` interface as well as child classes thereof that integrate various database types (see the `api reference <../apiref/dff.context_storages.database.html#dff.context_storages.database.DBContextStorage>`_). -The supported storage options and are as follows: +The supported storage options are as follows: * `JSON `_ * `pickle `_ @@ -205,8 +205,8 @@ The supported storage options and are as follows: ``DBContextStorage`` instances can be uniformly constructed using the ``context_storage_factory`` function. The function's only parameter is a connection string that specifies both the database type -and the connection parameters, as in *mongodb://admin:pass@localhost:27016/admin*. -(`see the reference <../apiref/dff.context_storages.database.html#dff.context_storages.database.DBContextStorage>`_) +and the connection parameters, for example, *mongodb://admin:pass@localhost:27016/admin*. +(`see the reference <../apiref/dff.context_storages.database.html#dff.context_storages.database.context_storage_factory>`_) The GitHub-based distribution of DFF includes Docker images for each of the supported database types. Therefore, the easiest way to deploy your service together with a database is to clone the GitHub @@ -227,7 +227,7 @@ also available in the distribution. Consult these files for more options. .. warning:: - The data transmission protocols require the data to be JSON-serializeable. DFF tackles this problem + The data transmission protocols require the data to be JSON-serializable. DFF tackles this problem through utilization of ``pydantic`` as described in the next section. Serialization