to_nsq: added async flag #1332
to_nsq: added async flag #1332
Conversation
| err := producer.Publish(*topic, line) | ||
| var err error | ||
| if *async { | ||
| err = producer.PublishAsync(*topic, line, nil, nil) |
There was a problem hiding this comment.
This is "fire and forget". Are you trying to get better throughput, or ignore errors (perhaps from some producers)? I think it would be better to address each thing more specifically (but just the one you currently need for now).
- for better throughput, you could use
PublishAsync()for all producers, or for a batch of messages, and then check the publish success of the batch - to ignore errors for some producers or messages, that should be an explicit option
There was a problem hiding this comment.
It is for throughput, I'm at that point where to_nsq slows the software piping through it.
While we can live with a few miss on our side it totally does miss error reporting and probably a way to limit the number of simultaneous async requests.
I will improve this one further, feel free to close until then!
There was a problem hiding this comment.
Alternatively, rather than using PublishAsync, build an option to use multiple producers per destination nsqd (tradeoff would be more connections).
Allows usage of async mode in
to_nsq