Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DRL solvers using hash as state value #4

Open
rlipkis opened this issue Jan 14, 2021 · 2 comments
Open

DRL solvers using hash as state value #4

rlipkis opened this issue Jan 14, 2021 · 2 comments

Comments

@rlipkis
Copy link

rlipkis commented Jan 14, 2021

Hi! I've been using your package, and I've run into an issue with the DRL solvers.

The function get_action passes the state into the neural network policy.μ, but this state is computed in convert_s to be the hash of the ASTState. My impression is that a hash value is not really a meaningful input to a NN, and it seems like it would invalidate much of the learning, effectively reducing the DRL algorithms to a random search.

If the GrayBox interface is extended to allow a Vector{Float64} state to be specified and stored in the ASTState at each update, this can be extracted in convert_s and passed into the NN. I've made those changes in my local copy to get things working, but perhaps there's a solution that's more in line with your vision for the package, in terms of genericity, etc. If you'd like, I can submit a pull request.

@mossr
Copy link
Member

mossr commented Feb 19, 2021

You're absolutely correct. This has been an extension to the GrayBox interface I've been planning on for a while. I'll take a look at your approach and let you know if that's the generic design that fits within this package. Nevertheless, we'll converge to allow some explicit state in this framework.

Glad to see you're using this package, and I remember you from CS238 👋

@mossr
Copy link
Member

mossr commented May 26, 2021

I recently pushed changes based on your fork to include explicit GrayBox.state information (instead of the sim.hash).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants