HTTPCLIENT-1625 Completely overhaul GSS-API-based authentication backend #577

stoty · 2024-09-06T06:14:25Z

No description provided.

michael-o

First, incomplete review.

httpclient5/src/main/java/org/apache/hc/client5/http/auth/AuthScheme.java

httpclient5/src/main/java/org/apache/hc/client5/http/auth/AuthSchemeV2.java

michael-o · 2024-09-06T11:57:00Z

httpclient5/src/main/java/org/apache/hc/client5/http/auth/KerberosConfig.java

-    private final Option stripPort;
-    private final Option useCanonicalHostname;
-    private final Option requestDelegCreds;
+    private final Option stripPort; //Effective default is ENABLE


I would invert this. For the past 15 years, I haven't seen a single deployment using the port with HTTP service class, especially because that major clients do not even support this.

That's certainly possible, but that would introduce an incompatibility with the 5.2 behaviour.

michael-o · 2024-09-06T11:57:30Z

httpclient5/src/main/java/org/apache/hc/client5/http/auth/KerberosConfig.java

+    private final Option stripPort; //Effective default is ENABLE
+    private final Option useCanonicalHostname; //Effective default is ENABLE
+    private final Option requestDelegCreds; //Effective default is DISABLE
+    private final Option requestMutualAuth; //Effective default is DISABLE


I would like to see a usecase why this should not be on by default without an option at all.

All I can came up with is:

GSS does not default to mutual auth

Backwards compatibility

I would at least leave an option to disable this, so that there is an out for very broken cases.
Of course there is also a case for forcing users to clean up their potentially broken setup.

I have looked at this, and the current API is really built around using the gss library default settings.

Since we are making incompatible changes anyway, if we want to set non-default gss options by default, (and keep them configurable), then we should remove the default options, and use simple booleans with explicit true/false defaults.

httpclient5/src/main/java/org/apache/hc/client5/http/auth/StandardAuthScheme.java

michael-o · 2024-09-06T12:04:39Z

httpclient5/src/main/java/org/apache/hc/client5/http/impl/auth/GGSSchemeBase.java

+                throw new AuthenticationException(gsse.getMessage(), gsse);
+            }
+            // other error
+            throw new AuthenticationException(gsse.getMessage(), gsse);


Won't those duplicate the message?

Not really.
But the Auth. execption would have the same message as the cause.
We can write some new message for the Auth. exception, I just really hate when a library swallows the original exception instead of wrapping it.

httpclient5/src/main/java/org/apache/hc/client5/http/impl/auth/GGSSchemeBase.java

httpclient5/src/main/java/org/apache/hc/client5/http/impl/auth/KerberosScheme.java

httpclient5/src/main/java/org/apache/hc/client5/http/impl/auth/SPNegoScheme.java

stoty · 2024-09-06T12:28:50Z

Thanks for the initial review @michael-o .

Regarding the renames and changing the defaults, I have specifically tried to touch as little as possible because much of the GSS/Kerberos code was not removed from codebase when disabling SPNEGO, which I assumed was for brackwards compatibility reasons.

I'm fine with your proposed changes, but those may theoretically cause issues with pre-5.3 apps, and will cause the api compare plugin to go wild.

michael-o · 2024-09-08T07:06:32Z

Thanks for the initial review @michael-o .

Regarding the renames and changing the defaults, I have specifically tried to touch as little as possible because much of the GSS/Kerberos code was not removed from codebase when disabling SPNEGO, which I assumed was for brackwards compatibility reasons.

I'm fine with your proposed changes, but those may theoretically cause issues with pre-5.3 apps, and will cause the api compare plugin to go wild.

Will engage next week again.

stoty · 2024-09-10T07:35:22Z

I believe I have addressed your comments which do not change the API.
I will work on your comments that introduce incompatible changes now.

stoty · 2024-09-17T07:41:36Z

Can you take another look at the current patch @michael-o ?
I have implemented most of your suggestions, but I had a few more questions comments regarding KerberosConfig.

michael-o · 2024-09-17T07:43:57Z

Can you take another look at the current patch @michael-o ? I have implemented most of your suggestions, but I had a few more questions comments regarding KerberosConfig.

Will pick end of this week or next week.

stoty · 2024-09-30T06:19:58Z

Rebased to current master

michael-o · 2024-09-30T07:47:00Z

@ok2c @stoty I gave this another conceptual thought: We said that we are going to remove the current code for good, but this PR reuses it and makes it partially undeprecated. For me, this is not straight forward. My gut feeling says that we should remove completely first and the @stoty can readd the code he thinks is required to satisfy this PR. WDYT?

ok2c · 2024-09-30T08:04:41Z

@michael-o We cannot remove old implementation unless we are willing to go to the major release cycle and release these changes as 6.0. I suggest that the old classes are left deprecated as there might still be users that rely on them and new implementation is added in parallel (with different names or in a different package).

michael-o · 2024-09-30T08:17:59Z

@michael-o We cannot remove old implementation unless we are willing to go to the major release cycle and release these changes as 6.0. I suggest that the old classes are left deprecated as there might still be users that rely on them and new implementation is added in parallel (with different names or in a different package).

I see, than we need new packages names. Agreed.

stoty · 2024-09-30T09:02:11Z

I have reverted the removals.

I don't see what alternative package name we could use instead of "org.apache.hc.client5.http.auth"
Using another package might also break package level visibility.

Maybe I could prefix the new classes with "Mutual" ?
i.e
MutualGssSchemeBase
MutualSpengoScheme

The other classes are strictly backwards compatible, like KerberosCredentials, and KerberosConfig.
Should I duplicate those as well ?

michael-o · 2024-09-30T09:06:02Z

I have reverted the removals.

I don't see what alternative package name we could use instead of "org.apache.hc.client5.http.auth" Using another package might also break package level visibility.

Maybe I could prefix the new classes with "Mutual" ? i.e MutualGssSchemeBase MutualSpengoScheme

The other classes are strictly backwards compatible, like KerberosCredentials, and KerberosConfig. Should I duplicate those as well ?

I'll get back to this question this week, have some serious trouble at $work I need to solve first.

stoty · 2024-09-30T09:06:05Z

I don't think that removing deprecation is problem, especially as these classes were deprecated without providing a replacement, not because they were replaced by something.

ok2c · 2024-09-30T09:48:11Z

I don't think that removing deprecation is problem, especially as these classes were deprecated without providing a replacement, not because they were replaced by something.

@stoty Un-deprecate those classes that can be kept fully backward compatible and create new classes for those that cannot

stoty · 2024-10-01T14:27:34Z

rebased

stoty · 2024-10-01T14:28:24Z

I have made the discussed changes.

ok2c · 2024-10-19T12:55:37Z

@stoty @michael-o Folks, what is the reason this PR is stuck?
@stoty Please rebase the latest off master to pick up Docker based compatibility tests. Feel free to disable japicmp but all integration must pass with your changes.

michael-o · 2024-10-19T13:12:15Z

@stoty @michael-o Folks, what is the reason this PR is stuck? @stoty Please rebase the latest off master to pick up Docker based compatibility tests. Feel free to disable japicmp but all integration must pass with your changes.

I am busy, can only peek once in a while. Hopefully the upcoming weeks will be better.

…names

and mark HttpAuthenticator @internal

ok2c · 2024-10-19T14:44:33Z

@stoty If you want to expedite the review process we could do the following. Once you are more or less happy with the state of things, submit changes to HttpAutheticator and AuthScheme in a separate PR and will happily review them. Kerberos specific code will have to wait until @michael-o has time to review it.

stoty · 2024-10-19T15:07:20Z

Thank you @ok2c.

I have rebased the patch and temporarily disabled japcmp.
I have also added and @internal annotation to HttpAuthenticator, I cannot imagine realistic user code calling that directly.
(though it does not seem to apply retroactively)

I will try to make the time to split the PR.

ok2c · 2024-10-19T17:31:16Z

I have also added and @internal annotation to HttpAuthenticator, I cannot imagine realistic user code calling that directly. (though it does not seem to apply retroactively)

@stoty There is still enough code in HC that goes back to version 3 and 2. I am fine with deprecating the present HttpAuthenticator and creating a new one marked as internal. However I still want to be able to understand what changes had to be made to the original code.

stoty · 2024-10-24T11:52:02Z

I don't think that duplicating HttpAuthenticator is a good idea.

For the authentication shemes, a case could be made for duplicating them, as those are pluggable components.

However, HttpAuthenticator is a purely internal class, and no sane external code is expected to call it.

The only way I could think that HttpAuthenticator would be called by external code, is if someone fully re-implemented a new (Closable...)Client class (and the corresponding ...Exec class ), which is not something that I think should be covered by backwards compatibility.

The end result would be replacing HttpAuthenticator with HttpAuthenticator2, and having a dangling HttpAuthenticator, which doesn't work with the other classes anyway (and it would possibly have to be modified anyway just to compile).

ok2c · 2024-10-24T12:02:23Z

@stoty Please disable japicmp for now and make changes to HttpAuthenticator you deem necessary. I will take it from there.

stoty · 2024-10-24T13:02:44Z

Thank you.
I have already done that.

stoty

I have add some comments to the most important changes

stoty · 2024-10-24T12:02:11Z

httpclient5/src/main/java/org/apache/hc/client5/http/auth/AuthScheme2.java

+     * @throws AuthenticationException in case the authentication process is unsuccessful.
+     * @since 5.5
+     */
+    void processChallenge(


When processing a response with mutual a auth token, we don't send a new request, hence we must be able to signal the authentication failiure to the caller, and MalformedChallengeException is not semantically correct for that.

We also need additional information, the HttpContext, and the challenged flag to be able to verify the received mutual auth token.

The old method did not verify anything, it just stored the token, an deferred processing to the generateAuthResponse() method.

stoty · 2024-10-24T12:08:20Z

httpclient5/src/main/java/org/apache/hc/client5/http/auth/KerberosConfig.java

@@ -151,11 +159,17 @@ public Builder setRequestDelegCreds(final Option requestDelegCreds) {
            return this;
        }

+        public Builder setRequestMutualAuth(final Option requestMutualAuth) {


This the new option to enable mutual auth.

Michal suggested changing this API, which is certainly possible, especially now, that we have duplicated the Scheme objects, and require changes in callers anyway.

However, this configuration API is indenpendent of the deeper changes.

stoty · 2024-10-24T12:09:21Z

httpclient5/src/main/java/org/apache/hc/client5/http/impl/DefaultAuthenticationStrategy.java

@@ -95,7 +99,7 @@ public List<AuthScheme> select(
        Collection<String> authPrefs = challengeType == ChallengeType.TARGET ?
                config.getTargetPreferredAuthSchemes() : config.getProxyPreferredAuthSchemes();
        if (authPrefs == null) {
-            authPrefs = DEFAULT_SCHEME_PRIORITY;
+            authPrefs = getSchemePriority();


This is a convenience change, to make it easier to re-add the SpnegoScheme without having to fully duplicate this class.

stoty · 2024-10-24T12:12:03Z

httpclient5/src/main/java/org/apache/hc/client5/http/impl/async/AsyncConnectExec.java

@@ -515,10 +517,11 @@ private boolean needAuthentication(
            final AuthExchange proxyAuthExchange,
            final HttpHost proxy,
            final HttpResponse response,
-            final HttpClientContext context) {
+            final HttpClientContext context) throws AuthenticationException, MalformedChallengeException {


This is corresponds to to the processChallenge() change in AuthScheme2, we need to be able to signal the authentication failure to the caller even when the server considers the authentication complete, and does not challenge us.

I am bit hazy on Connection oriented Auth schemes, but I think that in reality, there is no valid combination where a connection oriented scheme is used with Spengo, so this is only about conforming to the interface change.

stoty · 2024-10-24T12:21:41Z

httpclient5/src/main/java/org/apache/hc/client5/http/auth/AuthScheme2.java

+     * @return true if responses with non 401/407 response codes must be processed by the scheme.
+     * @since 5.5
+     */
+    boolean isChallengeExpected();


This is the other half of the mutual auth handling.
If mutual auth is requested, then the server MUST send a challenge with the mutual auth token (which is not really a "challenge", as a response is not expected).

If this returns true, but the server does not send a challenge, then mutual authentication failed, and an Exception is thrown to the user.

stoty · 2024-10-24T12:50:21Z