Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Missing logs for episodes with exceptions #110

Open
Gnurro opened this issue Jul 31, 2024 · 1 comment
Open

Missing logs for episodes with exceptions #110

Gnurro opened this issue Jul 31, 2024 · 1 comment
Labels
enhancement New feature or request

Comments

@Gnurro
Copy link
Collaborator

Gnurro commented Jul 31, 2024

Currently when an exception occurs that leads to an aborted episode, no logs are stored whatsoever. This makes it impossible to debug and handle issues that are dependent on model outputs.
The logs - which are properly created at runtime - should be stored in the log files regardless of any exceptions occuring, and not just be ephemeral and lost should an exception occur.
This should preferably be done at the time the log actions are actually happening, continuously writing to the log files. This might require an intermediate log format to preserve the existing JSON format log files and their handling for evaluation.

I currently have one such exception occuring with my project, and can only guess as to the model output that leads to it due to the lack of logging. I am tempted to log absolutely everything at the clembench.log level, just to be able to debug properly, but that would bloat clembench.log even more than it already is (my current clembench.log is 65mb just from an afternoon of experimenting).

@davidschlangen davidschlangen added the enhancement New feature or request label Oct 8, 2024
@davidschlangen
Copy link
Contributor

Continue discussion on what best way is to enable selective logging.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants