How to get latest minimizer (u) instead of the best one for optimization?

[ThummeTo]

Question❓

Hi,

I really like the quality of the documentation, however I struggle to find a functionality I am pretty sure that it exists:
If I get this correct, Optimization.jl always returns the minimizer u for the best (minimum/maximum) f evaluation - this is nice and often what I want.
However in some cases one is more interested in the latest u - so the last try to find a new optimum (think e.g. of having some stochastic effects in your f). Is there a keyword or similar?

I would assume to find this on this page of the docu:
Common-Solver-Options-(Solve-Keyword-Arguments)

Thanks in advance!

[Vaibhavdixit02]

I really like the quality of the documentation,

I am genuinely surprised 🤣

You can use a callback to store the us and pass it to the solve call -

ustore = []
function my_storing_callback(state, l)
    push!(ustore, state.u)
    return false
end

[ThummeTo]

I am genuinely surprised 🤣

Yeah, I actually found the things that I'd been looking for, so 👍 (except the one above)

Thank you! Ok, this is actually the way I am doing it right now... maybe this is something for a feature request (e.g. adding u_latest to the result struct) or does this feel like an exotic requirement?

[Vaibhavdixit02]

Since some solvers already store this in their results I haven't seen a reason to do this and increase the memory usage even more for the results (and you as the user can do it with the callback)

[ChrisRackauckas]

If some solvers already store this, then you just use the same reference and there's no extra memory cost?

[ThummeTo]

I have another example where this is very common: Stochastic batching when optimizing ML models... without the option to pick the latest instead of the best minimizer, the optimization will return the minimizer of the best batch element loss - which might be a completely outdated parameter set ...

Also: The callback gets called one last time for the best element after optimization finished, so with the code above:

ustore = []
function my_storing_callback(state, l)
    push!(ustore, copy(state.u)) # copy required because inplace modification of `state.u`
    return false
end
ustore[end] # the best value
ustore[end-1] # actually the latest value during optimization

I didn't check the implementation, but wouldn't it be possible to let a keyword decide if either the state best or latest gets stored without additional memory required?

Thanks!

[Vaibhavdixit02]

I am not sure if this was specific for Optimisers.jl solvers since you mention stochastic methods, if it is you can pass save_best=false as kwarg in the solve call to disable saving the best and then it'll just give you the latest set of parameters and loss