You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently when an exception occurs that leads to an aborted episode, no logs are stored whatsoever. This makes it impossible to debug and handle issues that are dependent on model outputs.
The logs - which are properly created at runtime - should be stored in the log files regardless of any exceptions occuring, and not just be ephemeral and lost should an exception occur.
This should preferably be done at the time the log actions are actually happening, continuously writing to the log files. This might require an intermediate log format to preserve the existing JSON format log files and their handling for evaluation.
I currently have one such exception occuring with my project, and can only guess as to the model output that leads to it due to the lack of logging. I am tempted to log absolutely everything at the clembench.log level, just to be able to debug properly, but that would bloat clembench.log even more than it already is (my current clembench.log is 65mb just from an afternoon of experimenting).
The text was updated successfully, but these errors were encountered:
Currently when an exception occurs that leads to an aborted episode, no logs are stored whatsoever. This makes it impossible to debug and handle issues that are dependent on model outputs.
The logs - which are properly created at runtime - should be stored in the log files regardless of any exceptions occuring, and not just be ephemeral and lost should an exception occur.
This should preferably be done at the time the log actions are actually happening, continuously writing to the log files. This might require an intermediate log format to preserve the existing JSON format log files and their handling for evaluation.
I currently have one such exception occuring with my project, and can only guess as to the model output that leads to it due to the lack of logging. I am tempted to log absolutely everything at the clembench.log level, just to be able to debug properly, but that would bloat clembench.log even more than it already is (my current clembench.log is 65mb just from an afternoon of experimenting).
The text was updated successfully, but these errors were encountered: