Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Entire parameter space won't get explored #8

Open
capnspacehook opened this issue Oct 24, 2024 · 0 comments
Open

Entire parameter space won't get explored #8

capnspacehook opened this issue Oct 24, 2024 · 0 comments

Comments

@capnspacehook
Copy link

Hello, I've been using carbs with decent success but noticed some parameter spaces were not getting explored very well. Here are my parameters:

params = [
        Param(
            name="total_timesteps",
            space=LinearSpace(min=10_000_000, max=50_000_000, scale=10_000_000, is_integer=True),
            search_center=20_000_000,
        ),
        Param(name="learning_rate", space=LogSpace(min=1e-5, max=1e-1), search_center=3e-05),
        Param(name="gamma", space=LogitSpace(min=0.8, max=0.9999), search_center=0.995),
        Param(name="gae_lambda", space=LogitSpace(min=0.8, max=1.0), search_center=0.98),
        Param(
            name="update_epochs", space=LinearSpace(min=1, max=15, scale=5, is_integer=True), search_center=5
        ),
        Param(name="clip_coef", space=LogitSpace(min=0.1, max=0.4), search_center=0.2),
        Param(name="ent-coef", space=LogSpace(min=1e-5, max=1e-1), search_center=7e-03),
        Param(name="vf_coef", space=LogitSpace(min=0.0, max=1.0), search_center=0.5),
        Param(name="vf_clip_coef", space=LogitSpace(min=0.1, max=0.4), search_center=0.1),
        Param(name="max_grad_norm", space=LinearSpace(min=0, max=5), search_center=1),
        Param(
            name="batch_size",
            space=LinearSpace(min=65_536, max=1_048_576, scale=270_000, is_integer=True),
            search_center=98_304,
        ),
        Param(
            name="minibatch_size",
            space=LinearSpace(min=128, max=65_536, scale=20_000, is_integer=True),
            search_center=512,
        ),
    ]

I left the initial_search_radius at the default of 0.3. I saw in the README and in a warning that I needed to adjust the scale of Linear spaces so I started performing some tests to try and figure out a good scale value. I would initialize carbs with 1 parameter space, set num_random_samples to 1000, gather 1000 suggestions and take note of the min, max, mean, and standard deviation. I did this with the initial_search_radius at 1.0 because as I understand it, carbs will increase the search radius as more observations suggestions are made. My thought was to find a scale value that will allow carbs to search the entire Linear space.

This worked OK in practice, for example, the minimum suggestions for the update_epochs parameter was 2 and the max was 8 after 24 suggestions (12 of which were random). I was surprised to see several Log and Logit parameter spaces not explored very much though. gamma for example had a min suggested value of 0.9821 and a max of 0.9987. I'm assuming this has to do with the search center being set so close to the max.

All this to say, I don't entirely understand how scale is supposed to be set and how it interacts with initial_search_radius. How should I set scale so that the min and max values of a space can be reached, in all types of parameter spaces? 'Set it >3' wasn't enough information for me to find good scale values for Linear spaces, and I have no idea how to set the scale for Log and Logit spaces, or if I even need to, perhaps in these cases carbs was working as expected.

Thanks for carbs, despite my confusion with parameter space scaling it's been very useful!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant