KQL Function CRD #419
Merged
KQL Function CRD #419
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Due to a golang bug related to http2 connections not getting closed when they are broken at times, we disabled http2 and also disabled long-lived connections. While this resolved the issues we saw, ingestor and collector have much higher CPU due to the constant TLS connection setup and teardown. Since we understand the root problem better now, this re-enables persistent connections with http 1.1 with a gradual drain on the server side. The drain is used to when scaling out so that traffic eventually becomes balanced. This also add a mitigation for http2 (though still disabled) which would allow us to use http2 connections in the future. http2 is still disabled because even when we send a GOAWAY message from the server, we still get spurious client side errors. When the upstream bug is fixed, we can revisit http2 enablement.
This was a case that was missing a repair when we get an error while reading the block.
The segment flush interval defaults to 100ms which allows time for many small writes to fill up the buffer and be flushed to disk in larger batches. When there a lot of metrics or tables and not a lot of concurrent writes, this flush interval increases CPU utilization. This setting allows for increasing the flush interval in collector to reduce CPU usage since the number of writes is relatively infrequent. For ingestor, we leave the default value in place.
Until now bundle.sh would expect that the AKS and ADX clusters be in the same region. This PR updates the script to add support for these clusters to be in different regions.
jwilder
approved these changes
Oct 24, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Introduce a CRD for the management of KQL Functions and Views in Ingestor. The primary target for this PR is the handling of Views as a means to enable custom schema definitions, but since Views are just special Functions, we also handle them as well. Included is a
sampledirectory with an included Function and View.