Automatically detect restart files and add tests for restarts

[Sbozzolo]

This commit adds a new option, detect_restart_file. When detect_restart_file is true, ClimaAtmos looks at the previous output_XXXX output folder and tries to see if there's a restart file in it. If yes, it uses it. This behavior is overridden when a restart file is explicitely passed. This commit uses and re-implments the same logic implemented in ClimaUtilities.OutputPathGenerator. (We might want to move that logic to ClimaUtilities at some point.)

This PR also adds a test to ensure that restarted simulations are identical to non-restarted ones.

This is accomplished by running a simulation for three steps and comparing it to two other cases:

1. A simulation restarted from the checkpoint produced at step 3
2. A simulation restarted from the checkpoint produced at step 2 and solved

This is a variation of #3031
Closes #3031

This is the first step in improving compatibility with restarts, and more work is needed. There is a little bit of commented out code to loop over different configurations. I am working on it now, but I would like to start getting this test merged in, as I add more complexity and fix things.

GPU tests are commented out because of ClimaCore bug,
MPI tests are commented out because of process hanging.

I will work on this next

[charleskawczynski]

In terms of the source code, I'm fine with most of the changes. But I think we can improve a few things:

• get_restart_file
• compare
• format of test/restart.jl

I'll comment locally to go into details.

[Sbozzolo]

This is how the output looks like in case of failure:

config = sphere equil 0M NoWarp
solve!:: 0.554812 seconds (2.24 k allocations: 1.059 MiB, 1.74% compilation time)
    just reading data
integrator.p.precomputed.ᶜlinear_buoygrad error: 0.00020609275
integrator.p.precomputed.ᶜgradᵥ_θ_virt.components.data.1 error: 0.00023572937
integrator.p.precomputed.ᶜgradᵥ_θ_liq_ice.components.data.1 error: 0.00023572937
    reading and simulating
solve!:: 0.000005 seconds (1 allocation: 400 bytes)
integrator.p.precomputed.ᶜlinear_buoygrad error: 0.00018343895
integrator.p.precomputed.ᶜgradᵥ_θ_virt.components.data.1 error: 0.00018323408
integrator.p.precomputed.ᶜgradᵥ_θ_liq_ice.components.data.1 error: 0.00018323408
config = box equil 0M NoWarp
solve!:: 0.135827 seconds (2.60 k allocations: 493.500 KiB, 10.25% compilation time)
    just reading data
integrator.p.precomputed.ᶜlinear_buoygrad error: 0.00043861143
integrator.p.precomputed.ᶜgradᵥ_θ_virt.components.data.1 error: 0.0004185408
integrator.p.precomputed.ᶜgradᵥ_θ_liq_ice.components.data.1 error: 0.0004185408
    reading and simulating
solve!:: 0.000002 seconds (1 allocation: 400 bytes)
integrator.p.precomputed.ᶜlinear_buoygrad error: 0.00035853966
integrator.p.precomputed.ᶜgradᵥ_θ_virt.components.data.1 error: 0.0003376229
integrator.p.precomputed.ᶜgradᵥ_θ_liq_ice.components.data.1 error: 0.0003376229
config = column equil 0M NoWarp
┌ Warning: perturb_initstate flag is ignored for single column configuration
└ @ ClimaAtmos /central/scratch/esm/slurm-buildkite/climaatmos-ci/20797/climaatmos-ci/src/solver/type_getters.jl:193
┌ Warning: Bubble correction not compatible with single column configuration. It will be switched off.
└ @ ClimaAtmos /central/scratch/esm/slurm-buildkite/climaatmos-ci/20797/climaatmos-ci/src/solver/type_getters.jl:204
solve!:: 0.064094 seconds (2.45 k allocations: 343.078 KiB, 28.11% compilation time)
    just reading data
integrator.p.precomputed.ᶜlinear_buoygrad error: 0.0002045927
integrator.p.precomputed.ᶜgradᵥ_θ_virt.components.data.1 error: 0.0002344887
integrator.p.precomputed.ᶜgradᵥ_θ_liq_ice.components.data.1 error: 0.0002344887
    reading and simulating
solve!:: 0.000003 seconds (1 allocation: 400 bytes)
integrator.p.precomputed.ᶜlinear_buoygrad error: 9.4087634e-5
integrator.p.precomputed.ᶜgradᵥ_θ_virt.components.data.1 error: 0.0001073189
integrator.p.precomputed.ᶜgradᵥ_θ_liq_ice.components.data.1 error: 0.0001073189