Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle "ReplyError: ERR Unknown worker" from faktory server #117

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

Laksh47
Copy link

@Laksh47 Laksh47 commented Mar 26, 2021

Issue:
Faktory returning ReplyError: ERR Unknown worker is not handled in client.ts resulting in the below exception,

(node:43609) UnhandledPromiseRejectionWarning: ReplyError: ERR Unknown worker 22dded38
    at parseError (/Users/laksh/Desktop/repos/kaiju/node_modules/redis-parser/lib/parser.js:179:12)
    at parseType (/Users/laksh/Desktop/repos/kaiju/node_modules/redis-parser/lib/parser.js:302:14)
node_modules/source-map-support/source-map-support.js:495
(node:43609) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 2)

Cause:
Currently, the issue is raised in this line, Client.ts #L239

Cause of the issue:

The unhandled error has been explained in detail in this open Issue #116

Error from faktory-server originates in Faktory commands.go#L223

The error was replicated and the fix was tested in a separate project with the following env:

node.js version: v12.16.2

npm/yarn version: 6.14.11

faktory-server version: 1.4.2

faktory-worker package version: 4.0.1

Added a test for this case in src/__tests__/client.test.ts @jbielick

Let me know if this is valid!

Otherwise, we can send a terminate signal back to the worker too if that's the proper fix, please advise!

References:

src/client.ts Outdated
@@ -236,11 +236,15 @@ export class Client {
*/
async beat(): Promise<string> {
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This this method is a bit low-level, I would prefer for it to reject the promise when it fails sending the beat to the server. Callers of Client#beat could then catch that error where necessary. As it's written, a caller of Client#beat would have no way of knowing whether it succeeded or not, because it returns a truthy value either way.

Since beat is already wrapped in a try/catch in Worker#work as of 4.1.0, do you think it will be okay if we don't try/catch here?

Copy link
Author

@Laksh47 Laksh47 Mar 29, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your inputs, I will upgrade the package and test this try/catch in Worker#work and provide an update

And I think stopping the worker during this error seems to be the best thing to do @jbielick

I can update the PR later today!

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey Josh, @jbielick

4.1.4 seems to handle the error signal in the catch block!

@jbielick
Copy link
Owner

What do you think about shutting the worker down when the heartbeat to the server fails unexpectedly? That could be done in the catch of the periodic heartbeat here:

} catch (error) {
this.emit(
"error",
new Error(`Couldn't send heartbeat to the server: ${error.stack}`)
);
}

If the server responded with a healthy response to the beat, no error would be thrown and things would work normally. The code should only ever enter the catch when the server responds to the heartbeat with ERR (unexpected) and something is very wrong (worker was evicted). We could do something like this.stop() in that case and the process will shut down.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants