Docs: Explain problem types (`AggregationLevel`) in the docs

[hamogu]

In the docs, the How to write objective functions section has this intriguing sentence:

To use least-squares optimizers in optimagic, you need to mark your function with a decorator and return the least-squares residuals instead of the aggregated function value.

However, there is no more detail what this does and what it's good for. So, I decided to start reading the code and found that it's an AggregationLevel, but, going back to the docs (as a user, I try to avoid spending too much time to read the code to find out what it does), AggregationLevel is only mentioned in passing in an example with the note "the type of problem this optimizer can solve -> scalar problems; Other optimizers solve likelihood or least_squares problems."

At this point, at least I found that there are in fact three Aggregationlevels: scalar, likelihood, and least-squares.

Some of my objective functions will have a least-square structure, others are log-likehoods, so I wonder if I should mark those. I don't think this needs separate examples, a short (1-3 sentence each) explanation in the docs about the following would suffice:

• What is the purpose of setting those levels / use the decorators to mark objective functions (as far as I've found out so far, they change the return format expected from the objective function (scalar vs. array) and also the penalty function)?
• E.g. for the likelihood, do you expect the objective function to return a number or an array? The summed log-likelihood? A positive or negative number (in my field people commonly use the negative log-likelihood, because then you can call common minimizers and don't have to worry about "maximizers")?
• Similar 1-2 sentences for scalar and least-square.

A good place for that might be in the "How to write an objective function" guide, but it could also live in the API docs, if it's called out in the objective function guide.

[janosg]

Hi @hamogu, thanks for creating the issue.

I agree that we need more information on this. The only information we have is right above the sentence you mentioned:

Many estimation problems have a least-squares structure. If so, specialized optimizers that exploit this structure can be much faster than standard optimizers. The sphere function from above is the simplest possible least-squarse problem you could imagine: the least-squares residuals are just the params.

We'll improve the documentation but right now I have little time. So here just a very brief answer to your question. Let me know if you need more:

What is the purpose of setting those levels

This is all about solving your problem more efficiently. Least-squares optimizers like nag_dfols and scipy_ls_lm work only for least-squares problem because they need access to the vector of least-squares residuals. By using the information from that vector (and not just the aggregated objective value) they can be more efficient. A good introduction is this paper.

However, every least-squares problem can also be solved with a scalar optimizer. In that case you just discard the extra information.

There is only one optimizer that is specialized for likelihood functions. It is the bhhh algorithm but our implementation of that is still experimental.

E.g. for the likelihood, do you expect the objective function to return a number or an array?

• If you mark the function as least-squares it needs to return the vector of least-squares residuals (see the paper linked above for the definition)
• If you mark the function as likelihood it needs to return the vector of log likelihoods
• If you mark the function as scalar (or not at all) it needs to return a scalar value.

As I said, this is all about giving the optimizer more information. You can still solve any likelihood or least-squares problem with standard optimizers. Optimagic takes care of aggregating the function output for you.

in my field people commonly use the negative log-likelihood, because then you can call common minimizers and don't have to worry about "maximizers"

Optimagic has a maximize function that works with all minimizers so you don't have to flip the sign and - more importantly - you don't have to remember to flip it back.

[janosg]

I think the best place would be an "Explanation" article we can link to in the various tutorials and how-to-guides.