-
Notifications
You must be signed in to change notification settings - Fork 55
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Label.key too long #49
Comments
seems like this sort of error should drop the tag vs kill the span?
…On Wed, Dec 20, 2017 at 9:51 PM, Jorrit Salverda ***@***.***> wrote:
I'm running istio 0.4.0 and send all traces to the zipkin-to-stackdriver
deployment running container gcr.io/stackdriver-trace-
docker/zipkin-collector:v0.3.0.
When I then run the bookinfo sample the zipkin-to-stackdriver application
keeps throwing errors like the following:
zipkin-to-stackdriver-74866c5cd9-s7b78 zipkin-to-stackdriver 2017-12-20 13:30:40.704 WARN 8 --- [driverStorage-1] zipkin.server.ZipkinHttpCollector : Cannot store spans [13b58682a43a952f.13b58682a43a952f<:13b58682a43a952f, 7c0d8a04309058d1.7f5ed293f891c6e9<:5f472c38c35e1e8b, 7c0d8a04309058d1.1965565a42a9b299<:7c0d8a04309058d1, 7c0d8a04309058d1.238eb23603b1992a<:7c0d8a04309058d1, 7c0d8a04309058d1.5f472c38c35e1e8b<:7c0d8a04309058d1, 7c0d8a04309058d1.7c0d8a04309058d1<:7c0d8a04309058d1, 5f5a0c322c213e20.31d4affd593c5ceb<:7f578ce3cfe96c6c, 5f5a0c322c213e20.386db219f31a9e00<:5f5a0c322c213e20, 5f5a0c322c213e20.7db2b757b548762f<:5f5a0c322c213e20, 5f5a0c322c213e20.7f578ce3cfe96c6c<:5f5a0c322c213e20, 5f5a0c322c213e20.5f5a0c322c213e20<:5f5a0c322c213e20, 223508843e90e31a.24b34b78807b3118<:223508843e90e31a, 223508843e90e31a.223508843e90e31a<:223508843e90e31a, 07e821b6551e692a.0583947468ee72a8<:07e821b6551e692a, 07e821b6551e692a.07e821b6551e692a<:07e821b6551e692a, 47a33573140bd7a7.755cec4744707cbf<:47a33573140bd7a7, 47a33573140bd7a7.47a33573140bd7a7<:47a33573140bd7a7, 7cde1c8e41f432fc.2a403e0949d626e9<:7cde1c8e41f432fc, 7cde1c8e41f432fc.7cde1c8e41f432fc<:7cde1c8e41f432fc, 2b2e6869b685f79d.6a59b7b637487e54<:302990ef143f2473, 2b2e6869b685f79d.173250b89df0307c<:2b2e6869b685f79d, 2b2e6869b685f79d.1ef844a83abff6fc<:2b2e6869b685f79d, 2b2e6869b685f79d.302990ef143f2473<:2b2e6869b685f79d, 2b2e6869b685f79d.035b3f9d8307c66b<:165d3637ba428305, 2b2e6869b685f79d.2ba0968f11eb9c6d<:2b2e6869b685f79d, 2b2e6869b685f79d.05d2db6e1ea50b44<:2b2e6869b685f79d, 2b2e6869b685f79d.165d3637ba428305<:2b2e6869b685f79d, 2b2e6869b685f79d.0512c6e5ef74083f<:719ddc4aa227995d, 2b2e6869b685f79d.1d813bf86cb57635<:2b2e6869b685f79d, 2b2e6869b685f79d.37d7217e907ac09b<:2b2e6869b685f79d, 2b2e6869b685f79d.719ddc4aa227995d<:2b2e6869b685f79d, 2b2e6869b685f79d.012ce8036813410c<:6e780125ea917718, 2b2e6869b685f79d.2b13555068fe0c18<:2b2e6869b685f79d, 2b2e6869b685f79d.4120e994016cb628<:2b2e6869b685f79d, 2b2e6869b685f79d.6e780125ea917718<:2b2e6869b685f79d, 2b2e6869b685f79d.2b2e6869b685f79d<:2b2e6869b685f79d, 086820b0c562fa14.405214a8613b8db9<:726b01ad4eb0d4c9, 086820b0c562fa14.1333daeacda8ab6d<:086820b0c562fa14, 086820b0c562fa14.0ffef8a18531f40f<:086820b0c562fa14, 086820b0c562fa14.726b01ad4eb0d4c9<:086820b0c562fa14, 5eea61f97d7e08c4.5fd1416de1b4ccad<:16c0253cac27df00, 5eea61f97d7e08c4.0df93364cf3a9e79<:5eea61f97d7e08c4, 5eea61f97d7e08c4.5c98e42860b5a95b<:5eea61f97d7e08c4, 5eea61f97d7e08c4.16c0253cac27df00<:5eea61f97d7e08c4, 5eea61f97d7e08c4.5eea61f97d7e08c4<:5eea61f97d7e08c4, 086820b0c562fa14.086820b0c562fa14<:086820b0c562fa14, 162335cb62ad1242.55266f4d1e8a8af5<:633c323e889a6cc8, 162335cb62ad1242.2f1e490db06154a8<:162335cb62ad1242, 162335cb62ad1242.179f42f0417f28b3<:162335cb62ad1242, 162335cb62ad1242.633c323e889a6cc8<:162335cb62ad1242, 162335cb62ad1242.162335cb62ad1242<:162335cb62ad1242, 083bd2a9143e6144.1578c7f27d3241ef<:546df37fc49ff6b1, 083bd2a9143e6144.78c3c170b92d5987<:083bd2a9143e6144, 083bd2a9143e6144.1060797712da5ef6<:083bd2a9143e6144, 083bd2a9143e6144.546df37fc49ff6b1<:083bd2a9143e6144, 083bd2a9143e6144.083bd2a9143e6144<:083bd2a9143e6144, 72f3388bae2c2dc1.5ca9c1eae6f07320<:5aab96fe8d3bc1eb, 72f3388bae2c2dc1.080710a528c4006d<:72f3388bae2c2dc1, 72f3388bae2c2dc1.4ef01eb624ccb4e8<:72f3388bae2c2dc1, 72f3388bae2c2dc1.5aab96fe8d3bc1eb<:72f3388bae2c2dc1, 72f3388bae2c2dc1.72f3388bae2c2dc1<:72f3388bae2c2dc1, 6adfbda545b64620.08f3824bd7a341c2<:46c0c3ad7a374112, 2e65a5f873cb13c3.0486cfb5c813309f<:416810f450c7a736, 2e65a5f873cb13c3.1f26e91705e6dbd2<:2e65a5f873cb13c3, 2e65a5f873cb13c3.7447d0cafa651281<:2e65a5f873cb13c3, 2e65a5f873cb13c3.416810f450c7a736<:2e65a5f873cb13c3, 2e65a5f873cb13c3.2e65a5f873cb13c3<:2e65a5f873cb13c3, 6adfbda545b64620.13aa8e63caabcc0a<:6adfbda545b64620, 6adfbda545b64620.67362a4bf25d6781<:6adfbda545b64620, 6adfbda545b64620.46c0c3ad7a374112<:6adfbda545b64620, 6adfbda545b64620.6adfbda545b64620<:6adfbda545b64620, 464b31471cd8db4b.746809e45bfcf2cc<:4792cbd9690ac695, 464b31471cd8db4b.574b7d31a048ae9f<:464b31471cd8db4b, 464b31471cd8db4b.006fa0ab35926d4f<:464b31471cd8db4b, 464b31471cd8db4b.4792cbd9690ac695<:464b31471cd8db4b, 464b31471cd8db4b.464b31471cd8db4b<:464b31471cd8db4b, 3c48f39fbf75d312.4626328f960d3dc7<:4c66fe25730044ab, 3c48f39fbf75d312.7ef663915dc80c23<:3c48f39fbf75d312, 3c48f39fbf75d312.00cfce66f86a0994<:3c48f39fbf75d312, 3c48f39fbf75d312.4c66fe25730044ab<:3c48f39fbf75d312, 3c48f39fbf75d312.3c48f39fbf75d312<:3c48f39fbf75d312] due to StatusRuntimeException(INVALID_ARGUMENT: Label.key in TraceSpan.span_id 397324419836310184 with length 171 exceeds maximum permitted length.)
zipkin-to-stackdriver-74866c5cd9-s7b78 zipkin-to-stackdriver
zipkin-to-stackdriver-74866c5cd9-s7b78 zipkin-to-stackdriver io.grpc.StatusRuntimeException: INVALID_ARGUMENT: Label.key in TraceSpan.span_id 397324419836310184 with length 171 exceeds maximum permitted length.
When sending it to zipkin instead all works fine. I this an issue in this
application?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#49>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAD618j6FbZUG8nXJ_3_jayE7C5urRM4ks5tCRDKgaJpZM4RIal5>
.
|
I'm no zipkin / tracing expert, so not sure what it should actually do :) |
sorry this note was for the code :) |
@JorritSalverda do you know if the issue goes away if we update to |
@JorritSalverda is this still a issue for you? I was able to collect tracing of sample bookinfo app through zipkin-to-stackdriver without any error logs. I ran it with istio-0.5.0 and zipkin-collector:v0.4.0. |
I had trouble to get istio and the bookinfo sample to work for a while, but with some modified settings in the istio configmap bookinfo itself works fine now. However now there are no traces showing up at all in Stackdriver Trace. When I point the zipkin service to zipkin itself I see them showing up immediately, but when pointing the zipkin service to the zipkin-to-stackdriver pods nothing happens. The logs don't show anything particular either:
I've tried both with and without setting GOOGLE_APPLICATION_CREDENTIALS and PROJECT_ID. |
can you try the new docker image?
https://github.com/openzipkin/docker-zipkin-gcp
|
Unfortunately that doesn't make a difference. Is there an easy way to make logging more verbose so I can see whether the application is called at all? |
I'm not in a good timezone to troubleshoot right now, but you can hop on
gitter https://gitter.im/openzipkin/zipkin
If you are using the zipkin provided image (not StackDriverCollector as was
mentioned in your log), than it is easier for us to support you as all the
settings are standard.
for example, at least via /metrics you should be able to see if you are
receiving spans at all. /health can verify your setup /configprops what
your configuration is
https://github.com/openzipkin/zipkin/tree/master/zipkin-server#metrics
At any rate github issues are not good for general troubleshooting so
please try gitter.
|
Probably what's happening here is StackDriver limits https://cloud.google.com/trace/docs/quotas Size of key per label/attribute | 128 bytes You can either reduce the size of your key to below that or someone can add code support for truncation. Note that the "TruncatableString" thing in the proto expects the cut to be UTF codepoint aware. This means the programming is a bit more interesting than substring. |
heres a hint static TruncatableString maybeTruncate(String string, int length) {
int sizeInBytes = utf8SizeInBytes(string);
if (sizeInBytes <= length) {
return TruncatableString.newBuilder().setValue(string).setTruncatedByteCount(0).build();
}
// not precise because this doesn't truncate codepoint aware, but maybe good enough..
return TruncatableString.newBuilder()
.setValueBytes(ByteString.copyFrom(string.getBytes(UTF_8), 0, length))
.setTruncatedByteCount(sizeInBytes - length).build();
} |
And here I was hoping since opencensus doesn't truncate, maybe the server does it... |
possible that it does now cc @mtwo
…On Fri, Jul 5, 2019, 7:18 PM Anuraag Agrawal ***@***.***> wrote:
And here I was hoping since opencensus doesn't truncate, maybe the server
does it...
https://github.com/census-instrumentation/opencensus-java/blob/master/exporters/trace/stackdriver/src/main/java/io/opencensus/exporter/trace/stackdriver/StackdriverV2ExporterHandler.java#L364
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#49?email_source=notifications&email_token=AAAPVV37O5RBSPRNTWA6SJ3P54UXJA5CNFSM4EJBVF42YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZJIW2Y#issuecomment-508726123>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAAPVV65K6SYROBKKM2IQZDP54UXJANCNFSM4EJBVF4Q>
.
|
Yep, I'm pretty sure that we truncate things server side if they're too large |
I'm running
istio 0.4.0
and send all traces to thezipkin-to-stackdriver
deployment running containergcr.io/stackdriver-trace-docker/zipkin-collector:v0.3.0
.When I then run the
bookinfo
sample the zipkin-to-stackdriver application keeps throwing errors like the following:When sending it to zipkin instead all works fine. I this an issue in this application?
The text was updated successfully, but these errors were encountered: