Add step / task / workflow run / observer metrics as logs #1691
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
<!-- ELLIPSIS_HIDDEN --> > [!IMPORTANT] > Add logging for step, task, workflow run, and observer metrics, and introduce `prompt_name` parameter for LLM API calls. > > - **Logging Enhancements**: > - Add logging for step duration in `update_step()` in `agent.py`. > - Add logging for task duration in `update_task()` in `agent.py`. > - Add logging for workflow run duration in `execute_workflow()` in `service.py`. > - Add logging for observer cruise duration in `mark_observer_task_as_completed()` in `observer_service.py`. > - Add logging for LLM API handler duration in `llm_api_handler_with_router_and_fallback()` and `llm_api_handler()` in `api_handler_factory.py`. > - **LLM API Changes**: > - Add `prompt_name` parameter to LLM API calls in `api_handler_factory.py` and various other files for better tracking. > - Update LLM API calls in `action_click.py`, `action_complete.py`, `action_input.py`, `agent.py`, `agent_functions.py`, `agent_protocol.py`, `totp.py`, `observer_service.py`, `block.py`, `service.py`, `caching.py`, `handler.py`, and `test_prompts.py` to include `prompt_name`. > - **Miscellaneous**: > - Set `include_in_schema=False` for `eval_router` in `app.py` to exclude it from the schema. > > <sup>This description was created by </sup>[<img alt="Ellipsis" src="https://img.shields.io/badge/Ellipsis-blue?color=175173">](https://www.ellipsis.dev?ref=Skyvern-AI%2Fskyvern-cloud&utm_source=github&utm_medium=referral)<sup> for 1645c7ad2ac1b6ddb4ae7f835e1e278503afa5ac. It will automatically update as commits are pushed.</sup> <!-- ELLIPSIS_HIDDEN -->
![ellipsis-dev[bot]](https://avatars.githubusercontent.com/in/64358?s=60&v=4){kind=small}
There was a problem hiding this comment.
👍 Looks good to me! Incremental review on a27f64f in 46 seconds
More details
- Looked at
554lines of code in11files - Skipped
0files when reviewing. - Skipped posting
1drafted comments based on config settings.
1. skyvern/forge/sdk/workflow/service.py:472
- Draft comment:
Ensure that theprompt_nameparameter is consistently passed to theLLM_API_HANDLERcalls to maintain consistency in logging and tracking. This is missing in some instances. - Reason this comment was not posted:
Comment did not seem useful.
Workflow ID: wflow_WJ76D52d3T5bcwxE
You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.
There was a problem hiding this comment.
👍 Looks good to me! Reviewed everything up to a27f64f in 1 minute and 22 seconds
More details
- Looked at
554lines of code in11files - Skipped
0files when reviewing. - Skipped posting
1drafted comments based on config settings.
1. skyvern/forge/sdk/workflow/service.py:473
- Draft comment:
Ensure that theprompt_nameparameter is consistently passed to allLLM_API_HANDLERcalls for consistent logging and tracking. This is missing in some instances. - Reason this comment was not posted:
Comment did not seem useful.
Workflow ID: wflow_agxSI5SAD09H68us
You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.
suchintan
approved these changes
Feb 1, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Important
Add logging for step, task, workflow run, and observer metrics, and enhance LLM API calls with
prompt_namefor better tracking.update_step()inagent.pywhen step is completed or failed.update_task()inagent.pywhen task is completed, failed, or terminated.mark_observer_task_as_completed()inobserver_service.pywhen completed.execute_workflow()inservice.pywhen completed.prompt_nameparameter to LLM API calls inagent.py,agent_functions.py,api_handler_factory.py,agent_protocol.py,totp.py,observer_service.py,block.py,caching.py,handler.pyfor better tracking.UTCfromdatetimeinagent.pyandobserver_service.py.This description was created by
for a27f64f. It will automatically update as commits are pushed.