[management/client] Integrate Job API with Job Stream and Client Engine #4428

aliamerj · 2025-09-01T16:08:52Z

Describe your changes

This PR connects the Job API with the peer job stream and updates both server and client code to work together.

What was done:

Integrated Job API with the JobManager stream channel.
Updated server logic to send and track jobs through the channel.
Implemented client-side handling in the engine for job requests and responses.
Adjusted existing code to align with the new unified job flow.

Issue ticket number and link #4354

Stack

feature/remote-debug - ⚠️ No PR associated with branch
- [management/client] Integrate Job API with Job Stream and Client Engine #4428 👈

Checklist

By submitting this pull request, you confirm that you have read and agree to the terms of the Contributor License Agreement.

Documentation

Select exactly one:

I added/updated documentation for this change
Documentation is not needed for this change (explain why)

Docs PR URL (required if "docs added" is checked)

Paste the PR link from https://github.com/netbirdio/docs here:

https://github.com/netbirdio/docs/pull/__

integrate api with stream and implement some client side

aliamerj · 2025-09-01T16:32:24Z

shared/management/client/grpc.go

@@ -298,7 +298,7 @@ func (c *GrpcClient) handleSyncStream(ctx context.Context, serverPubKey wgtypes.

 	// blocking until error
 	err = c.receiveUpdatesEvents(stream, serverPubKey, msgHandler)
-	if err != nil {
+	if err != nil && err != io.EOF{


I noticed that when sending a job, I was repeatedly seeing logs from the sync with EFO. To fix this, I updated this not ideal but for now,
the problem is when sending stream request for peer both sync and job triggered and one get valid request and the other get nothing , so we get this error and disconnect the peer , I think we need we to drop the io.EOF check in sync as i did for the job

pascal-fischer · 2025-09-04T14:30:32Z

client/internal/engine.go

+			case *mgmProto.JobRequest_Bundle:
+				uploadKey, err := e.handleBundle(params.Bundle)
+				if err != nil {
+					return &mgmProto.JobResponse{
+						ID:     msg.ID,
+						Status: mgmProto.JobStatus_failed,
+						Reason: []byte(err.Error()),
+					}
+				}
+				return &mgmProto.JobResponse{
+					ID:     msg.ID,
+					Status: mgmProto.JobStatus_succeeded,
+					WorkloadResults: &mgmProto.JobResponse_Bundle{
+						Bundle: &mgmProto.BundleResult{
+							UploadKey: uploadKey,
+						},


Here you are creating unnecessary code duplication. The only generic part is WorkloadResults. The rest is identical and can be build after the switch

The generic part are

Status could be succeeded or failed

Reason could be nil or error message

WorkloadResults could be nil or the result
so the identical part is just job id

pascal-fischer · 2025-09-04T14:37:34Z

management/server/store/sql_store.go

+	var job types.Job
+	if err := job.ApplyResponse(jobResponse); err != nil {
+		return status.Errorf(status.Internal, err.Error())
+	}


I would move this conversion to the GRPC layer to not drag proto sturct to deep into the code

pascal-fischer · 2025-09-05T09:01:46Z

client/internal/engine.go

+func (e *Engine) getPeerClient() (*grpc.ClientConn, error) {
+	conn, err := grpc.NewClient(
+		strings.TrimPrefix(e.daemonAddress, "tcp://"),
+		grpc.WithTransportCredentials(insecure.NewCredentials()),
+	)
+	if err != nil {
+		return nil, fmt.Errorf("failed to connect to daemon error: %v\n"+
+			"If the daemon is not running please run: "+
+			"\nnetbird service install \nnetbird service start\n", err)
+	}
+
+	return conn, nil
+}
+


I don't think this is the way to go. The engine is part of the daemon so we are basically connecting to ourself.

yes, i understand , that is why i needed help to understand what is the best why to call bundle ,
is passing grpc client and call it from it or passing address string and call the bundle or without the grpc and call the debugging directly ?
and also considering the other os

…4469)

…rview

aliamerj · 2025-09-15T13:17:46Z

client/android/client.go

@@ -112,7 +112,7 @@ func (c *Client) Run(urlOpener URLOpener, dns *DNSList, dnsReadyListener DnsRead

 	// todo do not throw error in case of cancelled context
 	ctx = internal.CtxInitState(ctx)
-	c.connectClient = internal.NewConnectClient(ctx, cfg, c.recorder)
+	c.connectClient = internal.NewConnectClient(ctx, cfg, c.recorder, "")


Do we need to pass the logFile here

aliamerj · 2025-09-15T13:18:28Z

client/cmd/up.go

@@ -196,7 +196,8 @@ func runInForegroundMode(ctx context.Context, cmd *cobra.Command, activeProf *pr
 	r := peer.NewRecorder(config.ManagementURL.String())
 	r.GetFullStatus()

-	connectClient := internal.NewConnectClient(ctx, config, r)
+	//todo: do we need to pass logFile here ?
+	connectClient := internal.NewConnectClient(ctx, config, r, "")


Do we need to pass the logFile here

aliamerj · 2025-09-15T13:19:05Z

client/embed/embed.go

-	client := internal.NewConnectClient(ctx, c.config, recorder)
+
+	//todo: do we need to pass logFile here ?
+	client := internal.NewConnectClient(ctx, c.config, recorder, "")


Do we need to pass the logFile here

aliamerj · 2025-09-15T13:20:37Z

client/ios/NetBirdSDK/client.go

@@ -127,7 +127,8 @@ func (c *Client) Run(fd int32, interfaceName string) error {
 	c.onHostDnsFn = func([]string) {}
 	cfg.WgIface = interfaceName

-	c.connectClient = internal.NewConnectClient(ctx, cfg, c.recorder)
+	//todo: do we need to pass logFile here ?
+	c.connectClient = internal.NewConnectClient(ctx, cfg, c.recorder, "")


Do we need to pass the logFile here

aliamerj · 2025-09-15T13:21:31Z

client/server/server.go

@@ -235,7 +233,7 @@ func (s *Server) connectWithRetryRuns(ctx context.Context, config *profilemanage

 	runOperation := func() error {
 		log.Tracef("running client connection")
-		s.connectClient = internal.NewConnectClient(ctx, config, statusRecorder)
+		s.connectClient = internal.NewConnectClient(ctx, config, statusRecorder, s.logFile)


here the only place we we pass the logFile

sonarqubecloud · 2025-09-16T18:07:54Z

Quality Gate failed

Failed conditions
3 New issues
3 New Code Smells (required ≤ 0)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

integrate api

08641b7

integrate api with stream and implement some client side

aliamerj requested review from pascal-fischer and lixmal September 1, 2025 16:29

aliamerj commented Sep 1, 2025

View reviewed changes

fix typo and fix validation

171925d

aliamerj mentioned this pull request Sep 3, 2025

Add Frontend Support for Peer Debug Bundle Trigger and History netbirdio/dashboard#485

Open

use real daemon address

e47e648

pascal-fischer reviewed Sep 5, 2025

View reviewed changes

redo the connect via address

dec307f

github-actions bot mentioned this pull request Sep 9, 2025

Refactor the debug bundle generator to be ready to use from engine #4469

Merged

7 tasks

pappz and others added 4 commits September 9, 2025 18:12

Refactor the debug bundle generator to be ready to use from engine (#…

f1cc9e4

…4469)

fix tests

9cb9fc4

fix lint

7fba07b

fix bug with stream

7c60ad6

aliamerj requested a review from pascal-fischer September 10, 2025 12:25

aliamerj added 5 commits September 12, 2025 11:06

try refactor status 1

36215b8

fix convert fullStatus to statusOutput & add logFile

426a6f2

fix tests

3ee24d7

fix tests

b1077aa

fix not enough arguments in call to nbstatus.ConvertToStatusOutputOve…

d82d56a

…rview

aliamerj commented Sep 15, 2025

View reviewed changes

fix status_test

9d5694f

aliamerj requested a review from pappz September 16, 2025 14:49

fix(engine): avoid deadlock when stopping engine during debug bundle

889c3c8

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

[management/client] Integrate Job API with Job Stream and Client Engine #4428

[management/client] Integrate Job API with Job Stream and Client Engine #4428

aliamerj commented Sep 1, 2025 •

edited by github-actions bot

Loading

Uh oh!

aliamerj Sep 1, 2025 •

edited

Loading

Uh oh!

pascal-fischer Sep 4, 2025

Uh oh!

aliamerj Sep 8, 2025

Uh oh!

pascal-fischer Sep 4, 2025

Uh oh!

pascal-fischer Sep 5, 2025

Uh oh!

aliamerj Sep 5, 2025

Uh oh!

aliamerj Sep 15, 2025 •

edited

Loading

Uh oh!

aliamerj Sep 15, 2025

Uh oh!

aliamerj Sep 15, 2025 •

edited

Loading

Uh oh!

aliamerj Sep 15, 2025

Uh oh!

aliamerj Sep 15, 2025

Uh oh!

sonarqubecloud bot commented Sep 16, 2025

Uh oh!

Uh oh!

Uh oh!

[management/client] Integrate Job API with Job Stream and Client Engine #4428

Are you sure you want to change the base?

[management/client] Integrate Job API with Job Stream and Client Engine #4428

Conversation

aliamerj commented Sep 1, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Describe your changes

What was done:

Issue ticket number and link #4354

Stack

Checklist

Documentation

Docs PR URL (required if "docs added" is checked)

Uh oh!

aliamerj Sep 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aliamerj Sep 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aliamerj Sep 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud bot commented Sep 16, 2025

Quality Gate failed

Uh oh!

Uh oh!

aliamerj commented Sep 1, 2025 •

edited by github-actions bot

Loading

aliamerj Sep 1, 2025 •

edited

Loading

aliamerj Sep 15, 2025 •

edited

Loading

aliamerj Sep 15, 2025 •

edited

Loading