Should we return or give a warning when providing bounds/constraints to a problem that cannot handle them?

[nbelakovski]

We discussed this in #134 but I'd like to raise the discussion again. In the current code, if someone is merely curious as to how UOBYQA performs on their objective function, even if it ignores their constraints, they will be blocked from doing so until they modify their code to stop providing constraints. Once their curiosity is satisfied, they will need to both change the algorithm back to their original one and uncomment their constraints. This seems rather unfriendly to the user. It might even be difficult for them to comment out all of the constraints - maybe they call prima_minimize in many places with many different objective functions, and it could be very tedious and time consuming to comment out the constraints in all those places.

I see no harm in running one of these algorithms even if constraints are provided that it cannot handle. I do think we should issue a warning, because it's strange to call an unconstrained algorithm and provide constraints, but I don't think it makes sense to refuse to continue until the user does some extra actions. In real life if somebody came to me with a problem they wanted me to solve, and they provided some extraneous information, I certainly wouldn't refuse to solve the problem until they told me again but without the extraneous information, although I might inform them that some of the information they provided is unlikely to impact the solution.

In addition to the above, I was also thinking about what we should do in the event that someone calls an algorithm with accepts constraints, but doesn't provide any. I think this might also be a good case to issue a warning. If we refuse to run the algorithm, the user can always override this by providing a constraint function which does nothing, but again this is very unfriendly to the user.

We've talked about how objective and constraint functions can be expensive, so we do not want to run them unnecessarily, but if a user starts a program and sees a warning about constraints not being used or not being provided, they can always act on the warning and press Ctrl+C and investigate.

The main changes this would lead to in the code would be in prima_check_problem in prima.c, which currently returns an error code if an algorithm that doesn't take constraints is provided along with constraints. We would need to modify it to produce a warning and continue. We may also need to modify it depending on the answer to the question about running algorithms which take constraints but are not provided with any, I would propose we add a warning in these cases as well.

Edit: After writing this I went back to #134 and reviewed your comments there. I think that if the user does not specify an algorithm, we should refine the problem as we see fit and select the appropriate algorithm, and we'll never have the issue of calling an algorithm with extra information or with not enough constraints. However if the user explicitly asks us to use a particular method by overriding the method argument in Python, then it would be appropriate to issue a warning if we cannot refine the problem to the point that the bounds/constraints are appropraite for the method, but I do not see why we should refuse to run, especially when the user can modify their code and sidestep our restriction.

[zaikunzhang]

Thank you @nbelakovski for raising this.

However, I think the following is crystal clear and commonly accepted.

1. If an unconstrained solver is called to solve a contained problem, something is wrong. In this case, the best strategy is to tell the user as soon as possible what is wrong.

2. A constrained solver should be capable of handling unconstrained problems as well. It is no problem to do so (which is the current behavior).

You are right that a warning (a message) should be printed if the user invokes a solver on a problem it cannot handle, so that the user knows what goes wrong. But the solver should exit with an error code afterward.

Thanks.

[nbelakovski]

Commonly accepted by whom?

If an unconstrained solver is called to solve a contained problem, something is wrong.

Maybe, but maybe not. The only way an unconstrained solver can be called to solve a constrained problem is if the user explicitly asks for that solver. I'm ignoring any possibility of errors in any sort of preprocessing/problem relaxation routines because I hope we would test them well, and in any case we're not talking about bugs in this conversation we're talking about intended behavior.

If a user has explicitly asked for a particular solver, then may not be aware that this solver cannot handle constrained problems, but then again it's possible they are aware of this fact. In some cases running an unconstrained solver on a problem might help to gain insight on a problem. It might help the user figure out where to place x0. Or maybe the user is just curious about where the optimal point would be if constraints were removed.

In that case, why should we refuse to run the algorithm? They've explicitly asked for it. We should warn them, in case they are uninformed or made a typo/mistake, but why should we also refuse to honor their request? What is the harm in continuing? Surely the users who are uninformed or who made a mistake can kill the program upon seeing the warning and re-evaluate their code before trying again.

[zaikunzhang]

Commonly accepted by whom?

If an unconstrained solver is called to solve a contained problem, something is wrong.

You can see what happens if you use scipy.minimize to call an unconstrained solver to solve a constrained problem.

[nbelakovski]

The following code produces the following output:

Code:

from scipy.optimize import minimize, NonlinearConstraint as NLC
nlc = NLC(lambda x: x**2, lb=25, ub=100)
minimize(lambda x: x**2, 7, method='bfgs', constraints=nlc)

Output:

/Users/nickolai/Library/Python/3.11/lib/python/site-packages/scipy/optimize/_minimize.py:576: RuntimeWarning: Method bfgs cannot handle constraints.
  warn('Method %s cannot handle constraints.' % method,
  message: Optimization terminated successfully.
  success: True
   status: 0
      fun: 7.451631553407712e-17
        x: [-8.632e-09]
      nit: 3
      jac: [-2.363e-09]
 hess_inv: [[ 5.000e-01]]
     nfev: 8
     njev: 4

The same warning occurs with other unconstrained algorithms. Naturally asking for COBYLA produces no warning message. But the point is that it produces a warning and still runs the algorithm.

One more thing that I think i worth pointing out: in the past software was often written to produce an error message in case anything was wrong and leave it up to the user to figure out what's wrong how to fix it. The modern trend is towards helpful error messages (see here and here and here). In that spirit, an error message in this situation should give the user a hint as to what to do. Since we don't know whether the user wants to see the unconstrained solution to their problem or whether they selected the wrong algorithm we could give them a rather long error message suggesting both options. It seems more straightforward to simply issue a warning and continue, don't you think?

[zaikunzhang]

The correct behavior is to print a warning and select a correct solver. This is the behavior of PDFO and the MATLAB interface of PRIMA.

However, do you really want to jump into this rabbit hole when talking about the C interface? Recall that different solvers have different requirements on the parameters. I suggest you have a look at https://github.com/libprima/prima/blob/main/matlab/interfaces/private/preprima.m. Let us keep the C interface simple and do the sophisticated things in the Python/MATLAB/Julia/R interfaces.

thanks.

[nbelakovski]

The correct behavior is to print a warning and select a correct solver. This is the behavior of PDFO and the MATLAB interface of PRIMA.

Even if the user has explicitly selected a solver? I don't understand, why is an option to select a solver provided if the software is going to pick its own solver anyway? If the only way for a user to invoke a different solver is to change their problem, then does it even make sense to offer them a way to override the method*? To me, the whole point of a manual override is for the user to say "I know this is wrong, but I want you to do it anyway," and of course we should issue a warning if it's inappropriate, but to offer the user the ability to override without actually overriding seems disingenuous.

* - The only exception would be NEWUOA/UOBYQA since they can both be applied to the same type of problem.

It doesn't matter too much to me where this logic goes, probably Python is best because I imagine SciPy may want to configure this logic to be closer to what I'm suggesting (since that's their current logic as evidenced by the above example).

[zaikunzhang]

Even if the user has explicitly selected a solver? I don't understand, why is an option to select a solver provided if the software is going to pick its own solver anyway?

If the user specifies a wrong option, then we raise a warning and take the default. This has always been the way that PRIMA handles all wrongly specified options.

There may be other ways to handle wrongly specified options (honestly, I do not think there is a strictly better way), but the way mentioned above has been adopted (together with Tom @ragonneau ) since 2019 when we were working on PDFO, and it has been working well. I do not see any reason to change it. I do not want to make an exception here. I do not like exceptions.

However, as mentioned in my last comment, it will be too complicated to implement the C interface in this way. It should be implemented in the Python/Julia/R interface, as has been done in the MATLAB interface. Let us keep the C interface simple --- returning an error code after raising a warning is a reasonable compromise.

The current behavior of scipy.optimize.minimize you observed above (disregarding the constraints and declaring that the problem is solved "successfully") is a bug. At least, it should not claim "success" in this situation. I respect the SciPy developers, but I have enough confidence in my knowledge of numerical optimization to disagree with them on this particular issue.

Thank you for this discussion, but I do not really think this particular topic is an urgent issue. Thank you.