Codecarbon to file

[davisethan]

Problem: MOABB prevents detailed CodeCarbon compute profiling metrics from being saved to file.

Proposed Solution: Make CodeCarbon fully configurable outside the script level; e.g. environment variables, configuration files ¹². When CodeCarbon is installed, the MOABB tabular results have an additional column codecarbon_task_name that is a unique UUID4 that can be joined with related rows from CodeCarbon tabular results by the column task_name. CodeCarbon writes multiple files, requiring the programmer to combine relevant CodeCarbon tables to join with MOABB tables to see detailed compute profiling metrics per cross-validation.

Google Colab: https://colab.research.google.com/drive/1YOUe47Easrj-FVbVrpsLfMmHsmcQGE2_?usp=sharing

Additional Changes: Python's time.time is wall clock time possibly relying on NTP server synchronization and is unreliable for tracking benchmark duration ³. A better alternative for benchmarking is to use a performance timer: time.perf_counter ⁴.

Footnotes

1. https://mlco2.github.io/codecarbon/usage.html#configuration ↩

2. https://mlco2.github.io/codecarbon/parameters.html ↩

3. https://peps.python.org/pep-0418/#time-time ↩

4. https://peps.python.org/pep-0418/#time-perf-counter ↩

[bruAristimunha]

Small adjustments, fill the what's new and can you also please adjust the tutorial?

[bruAristimunha]

Here, we usually prefer to put things within mne data folder.

[davisethan]

Currently, CodeCarbon only uses a hierarchical search for configurations ¹²:

1. (Lowest priority) Search the $HOME directory for a .codecarbon.config file, if found, apply its configuration parameters.
2. Search the $CWD directory, same steps as above.
3. Search environment variables, apply any found parameters.
4. (Highest priority) Use script level parameters.

Configurations cannot be found otherwise; e.g. In the $MNE_DATA directory. This .codecarbon.config configuration file that I'm auto-generating simply maintains current MOABB default configurations of CodeCarbon: save_to_file=false and log_level=error. Alternatively, this auto-generating code can be remove, in which case any MOABB user who has installed CodeCarbon will use its default configurations; e.g. save_to_file=true. This approach could be alright, though, CodeCarbon may generate many emissions files in $CWD.

A few options, summary:

1. (Current) Auto-generate a .codecarbon.config in $CWD if relevant configurations don't already exist.
2. Remove auto-generation of .codecarbon.config in $CWD, assume default behavior of CodeCarbon.
3. Auto-define CodeCarbon environment variables rather than a configuration file, however, this approach could be considered more dangerous since this is level visible, and takes higher priority than a configuration file.
4. CodeCarbon feature request: Introduce an EmissionsTracker parameter codecarbon_config that points to a configuration file that overrides/supplements hierarchical search for configurations. MOABB could then add a BaseEvaluation parameter codecarbon_config that points CodeCarbon to $MNE_DATA for its configuration search by default, e.g. WithinSessionEvaluation(codecarbon_config=MNE_DATA), and gives users control of alternative locations otherwise, .e.g. WithinSessionEvaluation(codecarbon_config=MY_DIR).

Generally, this PR creates a new MOABB feature: Users can join rows of MOABB predictive performance scores, and CodeCarbon compute profiling metrics from training. Currently, there isn't a tutorial for this in MOABB documentation (closest existing tutorial here ³). Though, I could create a tutorial for MOABB's website documentation.

Footnotes

1. https://mlco2.github.io/codecarbon/usage.html#configuration ↩

2. https://github.com/mlco2/codecarbon/blob/master/codecarbon/core/config.py#L76-L124 ↩

3. https://colab.research.google.com/drive/1YOUe47Easrj-FVbVrpsLfMmHsmcQGE2_?usp=sharing ↩

[bruAristimunha]

Here, it is better to have codecarbon_kargws arguments as optional arguments within the base of the evaluation.

This also allows you to comment in more detail on what is possible to do in our Code Carbon tutorial. Once you define the parameter, I can help/commit with you to adjust the tutorial within the CI, since it's a static tutorial at the moment.

[bruAristimunha]

Please proceed with option 4 and remove this code here.

[davisethan]

I can open a feature request for option 4. I was hesitant to add multiple new BaseEvaluation parameters for script level CodeCarbon configurations. I've added one new parameter codecarbon_config that is a dictionary of CodeCarbon parameters. This allows developers full breadth of use with configuration files, env vars, and script level parameters for CodeCarbon.

[davisethan]

The latest method of defining CodeCarbon configurations is good: 1) No more than one parameter added to BaseEvaluation for all CodeCarbon configurations, 2) Allows all methods of configuring CodeCarbon, 3) Isn't a compromise, no workaround needed to use CodeCarbon as already currently intended. Taking option 4 shouldn't be necessary, this PR is not blocked by the latest method of configuring CodeCarbon.

[bruAristimunha]

The line swapping change the replicability of the pipeline, but I agree with you that it is better isolated the loggers from the model fit. Can you log this as API Break within the what's new file?

[bruAristimunha]

To make the tutorial run within the documentation, you will need to rename the file name to plot or tutorial in the beginning of the file name.

[davisethan]

This PR does not contain the changes made in #870. If both PRs are accepted, I can add the changes from #870 into this PR, or alternatively, I can handle any merge conflicts that may occur between both PRs being merged.

[davisethan]

I've written a tutorial that can be included in MOABB documentation. Currently it clones the feature branch of this PR and installs MOABB in development mode. The tutorial can be made final and added in a separate PR, given the features in this PR are present in the MOABB pypi.

Tutorial: https://colab.research.google.com/drive/1XfHtbDqtEIcS4SlBNy_4cYU13eI4PPud?usp=sharing