POST request to get token is hanging out for all UAA 77.x.x version

[vuquanganhnguyen]

What version of UAA are you running?

77.x.x versions

How are you deploying the UAA?

deployed UAA docker image in kubernetes

What did you do?

upgrading uaa version from 76.31.0 to 77.x.x version

What did you expect to see? What goal are you trying to achieve with the UAA?

Everything works fine as UAA version 76.31.0; POST request to get token is not hanging out

What did you see instead?

Faced with the issue that the first POST request to get token is success, the following requests are hanging without any errors in the logs.

The logs contains two POST request fetching tokens. The first POST request was made at:

[2024-10-15 08:41:13.549] uaa - 1 [http-nio-8080-exec-3] .... DEBUG — FilterChainProxy: Securing POST /oauth/token

and got response without delays:

[2024-10-15 08:41:14.208] uaa - 1 [http-nio-8080-exec-3] .... DEBUG — DispatcherServlet: Completed 200 OK

The second request was made at

[2024-10-15 08:42:18.699] uaa - 1 [http-nio-8080-exec-6] .... DEBUG — FilterChainProxy: Securing POST /oauth/token

and stuck at:
[2024-10-15 08:42:18.943] uaa - 1 [http-nio-8080-exec-6] .... DEBUG — DataSourceUtils: Fetching JDBC Connection from DataSource

The response was received at:
[2024-10-15 08:43:43.745] uaa - 1 [http-nio-8080-exec-6] .... DEBUG — DispatcherServlet: Completed 200 OK

[strehle]

A reason for "hanging" could be fips compliant version of bouncy castle.

Can you provide the docker file and how you setup UAA . You need to set with property
property("java.security.egd", "file:/dev/./urandom")

and we know from some platforms that even then dev/random is used with FIPS version. so you need to gather more random... if the issue is, what I assume that it could be, e.g. https://stackoverflow.com/questions/26021181/not-enough-entropy-to-support-dev-random-in-docker-containers-running-in-boot2d

[vuquanganhnguyen]

Hello @strehle ; following is docker file

FROM tomcat:9.0-jre17 AS uaa-https
RUN apt-get update && apt-get install -y netcat-openbsd && rm -rf /var/lib/apt/lists/*
ENV LOGIN_CONFIG_URL WEB-INF/classes/required_configuration.yml
ENV CLOUD_FOUNDRY_CONFIG_PATH /uaa
ENV JAVA_OPTS="-Djava.security.egd=file:/dev/./urandom"
COPY uaa/uaa/build/libs/cloudfoundry-identity-uaa-*.war /usr/local/tomcat/webapps/uaa.war
COPY log4j2.properties /usr/local/tomcat/conf/
COPY server_https.xml /usr/local/tomcat/conf/server.xml
COPY startup.sh /usr/local/bin/docker-entrypoint.sh
RUN chmod +x /usr/local/bin/docker-entrypoint.sh
HEALTHCHECK --interval=60s --timeout=15s --start-period=30s --retries=3
CMD curl http://127.0.0.1:8080/uaa/.well-known/openid-configuration
EXPOSE 8080
ENTRYPOINT ["docker-entrypoint.sh", "postgres:5432", "-t", "10", "--"]
CMD ["catalina.sh", "run"]

This property("java.security.egd", "file:/dev/./urandom") was added 2 years ago.
Any help would be greatly appreciated.

[strehle]

Please try tomcat:9-jdk17-openjdk-slim

The start of the UAA is normal ? I assume if you do "uaac token get client xxx " then you see the hanging.

Do you run your docker on a RHEL VM ? Ask because we have seen this behavior and this there is a work around, see https://issues.redhat.com/browse/RHEL-3478 , means -Dsecurerandom.strongAlgorithms=PKCS11:SunPKCS11-NSS-FIPS

[vuquanganhnguyen]

tomcat:9-jdk17-openjdk-slim did not help. Deploy new image on any k8s environment, port forward. Then try to get token via Postman. The issue was observed when I was trying to get token in 2nd/3rd time.
But it works perfectly when I rollback to 76.31.0.

[strehle]

But it works perfectly when I rollback to 76.31.

Yes, but then we can close this and you run an insecure UAA.

If you want have solved this with newer UAAs, then I need your help in try-out, because K8s is something which I have no option to test. I run UAA sometimes also in a docker env, but locally and as mentioned we have seen issues, if we run this on RHEL VMs.

With UAA 77.x the bc-fips replaces the bc-java for signature creation. This was not the only change for the major upgrade, but one reason why we moved from 76.x -> 77.x

If you are willing to test, then here is another test where you can setup system properties
https://downloads.bouncycastle.org/fips-java/docs/BCFipsDescription.pdf

From there I would give a try to

-Dorg.bouncycastle.rsa.allow_multi_use=true

So should decide whether you close this or we proceed in finding the reason. I mentioned this because you can decide on it.

[vuquanganhnguyen]

i replaced -Djava.security.egd=file:/dev/./urandom
with
-Dsecurerandom.strongAlgorithms=PKCS11:SunPKCS11-NSS-FIPS
It seems the issue has been gone.
Many thanks to @strehle !! You helped me.

[strehle]

ok, thanks for keeping on it.

and yesCentOS is a RedHad derivate, means it behaves similar...; so this here is the bug which is relevant for your case
https://issues.redhat.com/browse/RHEL-3478

I have noticed, that the combination of JDK17 and RHEL8/9 (dont know what it means on CentOS but the have relations) and this will be fixed with newer version.

The issue is here is only for your information.

A real solution now whould be to setup a deamon for your entropy, e.g. rngd or havedge
see:
https://developers.redhat.com/blog/2017/10/05/entropy-rhel-based-cloud-instances#check_available_entropy

Then it should work without extra system settings and best: you run your UAA with more secure signatures because your random is now good enough.

[strehle]

Close this now, but commented for later - simply.
Thanks

[vuquanganhnguyen]

Hello @strehle : Unfortunately it did not resolve the issue; it worked only one time.
I tried with following
-Dsecurerandom.strongAlgorithms=PKCS11:SunPKCS11-NSS-FIPS -Djava.security.egd=file:/dev/./urandom"
The issue was observed when I was trying to get token in 2nd/3rd time.
Any help would be greatly appreciated.

[strehle]

Hello @strehle : Unfortunately it did not resolve the issue; it worked only one time. I tried with following -Dsecurerandom.strongAlgorithms=PKCS11:SunPKCS11-NSS-FIPS -Djava.security.egd=file:/dev/./urandom" The issue was observed when I was trying to get token in 2nd/3rd time. Any help would be greatly appreciated.

on your host , can you just execute cat /proc/sys/kernel/random/entropy_avail , you can execute it selveral times..... and if you seen hanging UAA then this number may is very low. If this expectation is true, then you should install on your host haveged

[vuquanganhnguyen]

Hello @strehle : Following is the result when i execute cat /proc/sys/kernel/random/entropy_avail for uaa version 76.31.0 and uaa version 77.19.0
76.31.0

root@uaa-service-6777645647-88rb9:/usr/local/tomcat# cat /proc/sys
sys/           sysrq-trigger  sysvipc/
root@uaa-service-6777645647-88rb9:/usr/local/tomcat# cat /proc/sys/kernel/random/entropy_avail
3543
root@uaa-service-6777645647-88rb9:/usr/local/tomcat# cat /proc/sys/kernel/random/entropy_avail
3578
root@uaa-service-6777645647-88rb9:/usr/local/tomcat# cat /proc/sys/kernel/random/entropy_avail
3584
root@uaa-service-6777645647-88rb9:/usr/local/tomcat# cat /proc/sys/kernel/random/entropy_avail
3585
root@uaa-service-6777645647-88rb9:/usr/local/tomcat#

77.19.0

root@uaa-service-768dbd764d-4z5q6:/usr/local/tomcat# cat /proc/sys/kernel/random/entropy_avail
3577
root@uaa-service-768dbd764d-4z5q6:/usr/local/tomcat# cat /proc/sys/kernel/random/entropy_avail
2967
root@uaa-service-768dbd764d-4z5q6:/usr/local/tomcat# cat /proc/sys/kernel/random/entropy_avail
1377
root@uaa-service-768dbd764d-4z5q6:/usr/local/tomcat# cat /proc/sys/kernel/random/entropy_avail
28
root@uaa-service-768dbd764d-4z5q6:/usr/local/tomcat# cat /proc/sys/kernel/random/entropy_avail
40

The consensus is that numbers below 1000 will lead to processes blocking waiting for more entropy.
I am trying rngd to seed randomness from non-blocking source will let you know the result soon.

[strehle]

I re-opend the ticket/issue, because I think about adding a YAML options for your case
There is a work around before BC is initialized.

Your results are expected. The 77.x uses /dev/random not urandom and this random gets empty and then it waits for more :-) that is the reason why you are waiting...

The better solution would be to add a daemon ( havedge or rngd=) which fill-up your random again and then you dont have a problem. A workaround would be use change the algorithm. We had similar issue in a forked UAA and solved it with some extra settings. Therefore we should add this to community UAA, however the best solution is to have a stable random device

[vuquanganhnguyen]

I tried to install the rng-tools package and map it to use the /dev/urandom inside docker container.

  apt-get install -y rng-tools
  rngd -r /dev/urandom 

Unfortunately it did not help.

[strehle]

The docker container uses /dev/random, even if you setup /dev/urandom.
But I will add an option and send you a package for testing... need some days but we should solve it with a configuration...

[vuquanganhnguyen]

Hello @strehle : no problem with some days for preparing new packaging. I will test it and let you know the result. Many thanks in advance.

[vuquanganhnguyen]

Hello @strehle : Do you think I can have new packaging from you this week for testing ? 🙏

[strehle]

Hello @strehle : Do you think I can have new packaging from you this week for testing ? 🙏

will do it tomorrow. I have a RHEL 8 for validation and for me it worked with

-Dsecurerandom.strongAlgorithms=NativePRNGNonBlocking:SUN,NativePRNGBlocking:SUN,DRBG:SUN

see:
https://docs.parasoft.com/display/DOTTEST20241/Configuring+FIPS+Mode

If you have no success with it, I will send you a uaa.war (based on 77.19.0) with this setting at a early UAA start, then it should work for sure

[vuquanganhnguyen]

Tried with

"-Dsecurerandom.strongAlgorithms=NativePRNGNonBlocking:SUN,NativePRNGBlocking:SUN,DRBG:SUN -Djava.security.egd=file:/dev/./urandom"

Unfortunately it did not help, The issue was observed when I was trying to get token in 3rd time.

Following is the result when i execute cat /proc/sys/kernel/random/entropy_avail

root@uaa-service-6fd4cbccd6-2z9vc:/usr/local/tomcat# cat /proc/sys/kernel/random/entropy_avail
3838
root@uaa-service-6fd4cbccd6-2z9vc:/usr/local/tomcat# cat /proc/sys/kernel/random/entropy_avail
3213
root@uaa-service-6fd4cbccd6-2z9vc:/usr/local/tomcat# cat /proc/sys/kernel/random/entropy_avail
1646
root@uaa-service-6fd4cbccd6-2z9vc:/usr/local/tomcat# cat /proc/sys/kernel/random/entropy_avail
156
root@uaa-service-6fd4cbccd6-2z9vc:/usr/local/tomcat# cat /proc/sys/kernel/random/entropy_avail
55
root@uaa-service-6fd4cbccd6-2z9vc:/usr/local/tomcat#

[strehle]

Ok, the system does not allow you to set it.
However I tested and changed it in UAA runtime

The war is more than I can upload. Is a war (based on 77.19.0) enough for you ? if yes, can you provider a place where I can upload 60 MBs for you ?

Otherwise here is the patch
algorithm.patch

With the patch after start you see in uaa.log

ERROR --- YamlServletProfileInitializer: Current securerandom.strongAlgorithms: NativePRNGBlocking:SUN,DRBG:SUN
ERROR --- YamlServletProfileInitializer: New - not set - securerandom.strongAlgorithms: NativePRNGNonBlocking:SUN,NativePRNGBlocking:SUN,DRBG:SUN

So is only an error, because then you can better search for it, however the point is, if you see this, then you have the patched UAA running and then it will work.... as mentioned I can verify your issue on a RHEL8 with a simply loop around "uaac token client get admin -s adminsecret"

[vuquanganhnguyen]

Hi @strehle : is it possible commit your algorithm.patch in dedicated branch as https://github.com/cloudfoundry/uaa/tree/77.19.0-algorithm-patch or what ever name you wish that I can build war file easily ? I had this auto script for building it.

[strehle]

Hi @strehle : is it possible commit your algorithm.patch in dedicated branch as https://github.com/cloudfoundry/uaa/tree/77.19.0-algorithm-patch or what ever name you wish that I can build war file easily ? I had this auto script for building it.

#3180

[vuquanganhnguyen]

https://github.com/cloudfoundry/uaa/tree/77.19.0-algorithm-patch resolved this issue. 🙏 🙏
Following is the result when i execute cat /proc/sys/kernel/random/entropy_avail

root@uaa-service-858f6556cf-nkds4:/usr/local/tomcat# cat /proc/sys/kernel/random/entropy_avail
3645
root@uaa-service-858f6556cf-nkds4:/usr/local/tomcat# cat /proc/sys/kernel/random/entropy_avail
3661
root@uaa-service-858f6556cf-nkds4:/usr/local/tomcat# cat /proc/sys/kernel/random/entropy_avail
3664
root@uaa-service-858f6556cf-nkds4:/usr/local/tomcat# cat /proc/sys/kernel/random/entropy_avail
3667
root@uaa-service-858f6556cf-nkds4:/usr/local/tomcat# cat /proc/sys/kernel/random/entropy_avail
3670
root@uaa-service-858f6556cf-nkds4:/usr/local/tomcat# cat /proc/sys/kernel/random/entropy_avail
3673
root@uaa-service-858f6556cf-nkds4:/usr/local/tomcat# cat /proc/sys/kernel/random/entropy_avail

I can see 2 ERROR above relating to securerandom.strongAlgorithms in uaa logs

ESAPI: WARNING: System property [org.owasp.esapi.opsteam] is not set
ESAPI: WARNING: System property [org.owasp.esapi.devteam] is not set
ESAPI: SecurityConfiguration for Encryptor.CipherTransformation not found in ESAPI.properties. Using default: AES/CBC/PKCS5Padding
[2024-12-06 12:34:31.921] uaa/uaa - 1 [main] .... ERROR --- YamlServletProfileInitializer: Current securerandom.strongAlgorithms: NativePRNGBlocking:SUN,DRBG:SUN
[2024-12-06 12:34:31.927] uaa/uaa - 1 [main] .... ERROR --- YamlServletProfileInitializer: New set - securerandom.strongAlgorithms: NativePRNGNonBlocking:SUN,NativePRNGBlocking:SUN,DRBG:SUN
[2024-12-06 12:34:32.674] uaa/uaa - 1 [main] .... INFO --- SpringSecurityCoreVersion: You are running with Spring Security Core 5.8.16
[2024-12-06 12:34:32.677] uaa/uaa - 1 [main] .... INFO --- SecurityNamespaceHandler: Spring Security 'config' module version is 5.8.16
....

That's great work.
Hope the PR #3180 shall be merged in main branch as soon as possible.