Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

High CPU Consumption #25

Open
ym-corey opened this issue Feb 23, 2023 · 8 comments · May be fixed by #56
Open

High CPU Consumption #25

ym-corey opened this issue Feb 23, 2023 · 8 comments · May be fixed by #56

Comments

@ym-corey
Copy link

ym-corey commented Feb 23, 2023

Hi @chuff (maybe)?

We've started using the GPP library in our live environment to get a better idea of how it performs etc., and to ensure it works. While we're not receiving many GPP strings, we are receiving TcfEuV2/TCF2/GDPR strings in large volume. On our servers, we've noticed that CPU consumption for handling a TcfEuV2 (GDPR IAB TCF V2) string processing through the TcfEuV2 model eats up around 8% CPU, while the same strings processing through the older IAB TCF TCString library only consumes around 0.8% CPU.

Looking at the TcfEuV2 model, I see that the entire string is decoded upon creation rather than lazy decoding only the segments that are relevant/used. Would it be feasible to update the decryption/parsing/processing methodology to be lazy?

For performance stats, I used this tool

@travisbeale
Copy link

We are having the same problem. This library creates an additional 15-20% CPU usage in our ad server.

@patmmccann
Copy link

Was this solved by #34 ?

@ym-corey
Copy link
Author

Hi @patmmccann ,

I ran some more tests today. Since we aren't receiving GPP in high volume I substituted the usage of the TCString library with TcfEuV2 on GDPR strings to simulate what the CPU consumption/distribution within our Java process would be if we instead needed to process GPP at the same volume.

Our control traffic shows around 4% CPU (TCString.decode(consentString)), while the test shows around 18% (new TcfEuV2(consentString)). This is on a service where around 80% of inbound traffic requires processing of a consent string.

My understanding is that the TCString library is lazily decoding each section of GDPR, whereas the GPP library is lazily decoding each section of GPP. The key difference being that the GDPR string is not lazily decoded in the GPP library.

@patmmccann
Copy link

patmmccann commented Apr 22, 2024

@ym-corey I recommend you reach out to @Net-burst ; who made substantial contributions to make prebid-server-java performant at decoding gpp strings. I am sure he is much more knowledgable than I on this matter

@ym-corey
Copy link
Author

I've reviewed the prebid-server-java code and while there is some code in place to prevent re-encoding of strings (specifically the TcfEuV2 and UsPrivacyV1 sections) for outbound requests it otherwise doesn't resolve or work around the lazy decoding issue present in the GppModel. I suspect prebid-server-java will also notice a large cpu spike when CMPs switch from providing GDPR strings to GPP strings.

@Net-burst
Copy link

Hey, @ym-corey. Our team made a decision to wait for IAB fix/workaround and work from there instead of jumping the gun. We will re-evaluate the bottlenecks after we update the library and will open-source our work by making a PR here.
And yeah, we are already seeing an increase in CPU usage, so this will become a priority...

@yuzawa-san yuzawa-san linked a pull request Jul 23, 2024 that will close this issue
@yuzawa-san
Copy link

Please review #56
I was able to cut down on CPU and memory significantly.

@captify-adanilov
Copy link

Would be good to have this issue sorted out, as the library may cause a massive increase in CPU load at the moment. I eventually had to revert to using the older TCF library, keeping GPP only as a fallback.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants