Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

shadow desktop ceases to update when using --desktop-scaling < 100% #4323

Open
Lanchon opened this issue Aug 12, 2024 · 18 comments
Open

shadow desktop ceases to update when using --desktop-scaling < 100% #4323

Lanchon opened this issue Aug 12, 2024 · 18 comments
Labels
bug Something isn't working geometry X11

Comments

@Lanchon
Copy link

Lanchon commented Aug 12, 2024

Describe the bug
shadow desktop ceases to update a couple of seconds after starting when using --desktop-scaling < 100%. this happens regardless of whether OpenGL is used. mouse actions in the client to server direction keep on working. no text is displayed in the client console when the image ceases to update.

To Reproduce
Steps to reproduce the behavior:

  1. server command: none
  2. client command:
    • xpra shadow ssh:XXX:0 --desktop-scaling=70%
    • xpra shadow ssh:XXX:0 --desktop-scaling=70% --opengl=off
  3. specific action to trigger the bug: wait 2 seconds

System Information (please complete the following information):

  • Server OS: Linux Mint 22 Wilma base: Ubuntu 24.04 noble
  • Client OS: Linux Mint 22 Wilma base: Ubuntu 24.04 noble
  • Xpra Server Version: xpra v6.1.1-r0
  • Xpra Client Version: xpra v6.1.1-r0

Additional context
on both sides:
Desktop: Cinnamon v: 6.2.9 tk: GTK v: 3.24.41 wm: Muffin vt: 7 dm: LightDM v: 1.30.0
standard X11 session

@totaam
Copy link
Collaborator

totaam commented Aug 13, 2024

That's odd, is there anything in your client output at all?
Your server log perhaps?

I did hit some major painting issues with opengl, but none without.
The opengl issues are now fixed (and have already been backported to 6.1.x and will be included in the upcoming v6.1.2 release)
I've just tried it here with Fedora 40, xpra from git master and everything worked as expected - attaching locally with --no-mmap.

@Lanchon
Copy link
Author

Lanchon commented Aug 14, 2024

i downgraded only the client, then tested several client versions against 6.1.1-r0:

(freezes = stops updating the client windows a couple of seconds after launching)
(works = no freezing)

version opengl off opengl default
5.0.9-r0-1 works (bilinear interpolation) works (nearest neighbor)
6.0-r0-1 works works but with #4324 and significant dirty refresh issues
6.0.2-r0-1 freezes freezes and has #4324

the screen freezes exactly before the "xpra shadow server ready" notification is shown in the server. (note that client mouse and keyboard events continue to be forwarded to the sever normally.)

@totaam
Copy link
Collaborator

totaam commented Aug 14, 2024

6.0.x are not supported versions: https://github.com/Xpra-org/xpra/wiki/Versions

significant dirty refresh issues

Partial workaround: #4319 (comment)

@Lanchon
Copy link
Author

Lanchon commented Aug 14, 2024

well i just wanted to do a gross bisection regarding this freezing issue.

it was introduced between 6.0 and 6.0.2.

@totaam
Copy link
Collaborator

totaam commented Aug 14, 2024

Thanks, I'm not seeing any freezing here - but I'll try again.

@Lanchon
Copy link
Author

Lanchon commented Aug 14, 2024

try linux mint 22 cinnamon, default session. im using that on both sides.

@totaam
Copy link
Collaborator

totaam commented Aug 14, 2024

5.0.9-r0-1 works (bilinear interpolation) works (nearest neighbor)

Oh, and BTW, I don't see how that's possible unless your client display is also using some quite odd scaling settings.
The opengl renderer uses linear: #4324 (comment)

@Lanchon
Copy link
Author

Lanchon commented Aug 14, 2024

yes, technically it's actually called bilinear, given that it acts on a 2D surface and has to interpolate linearly in 2 axes, but we are talking about the same thing.

@totaam
Copy link
Collaborator

totaam commented Aug 14, 2024

technically it's actually called bilinear

What is surprising here is that you're saying that the non-opengl client uses linear and the opengl client does not.
Both should be using server-side downscaling, which means that the client-side rendering should be irrelevant - and identical.

@totaam
Copy link
Collaborator

totaam commented Aug 14, 2024

Found a major issue with opengl scaling, which I still don't fully understand but is finally fixed: 775d435

I believe that this is the same issue that you reported here as it requires desktop-scaling to trigger.
The updates are being sent to the client, but the opengl client was somehow failing to update the window front buffer.

Feel free to re-open if 6.1.2 doesn't fix things for you.
(there will be 6.2 builds in the beta area soon too)

@totaam totaam closed this as completed Aug 14, 2024
@Lanchon
Copy link
Author

Lanchon commented Aug 14, 2024

What is surprising here is that you're saying that the non-opengl client uses linear and the opengl client does not.
Both should be using server-side downscaling, which means that the client-side rendering should be irrelevant - and identical.

that is correct.

i hope you don't need to see this with your own eyes, i have tested it repeatedly. i also saw that opengl mode was more power efficient on the client so i wanted to use that, but i couldn't because scaled down text is poorly readable without proper filtering.

keep in mind that this is a 5.x client talking to a 6.x server, so it's probably a setup you don't care much to debug.

Both should be using server-side downscaling

how does the 6.x server do the downscaling? this is an operation that would benefit from GPU usage. even if today's CPUs can do it, it'd be much more efficient to do it in the GPU. this also applies to image/video encoding of course. does the server use the GPU? can i tell the server NOT to use the GPU? from the client? are you sure --opengl=off does not influence the server?

@Lanchon
Copy link
Author

Lanchon commented Aug 14, 2024

Feel free to re-open

i can't reopen if a team member closed the issue; only you can. (i could reopen it if i had closed it myself though.)

Found a major issue with opengl scaling, which I still don't fully understand but is finally fixed: 775d435

I believe that this is the same issue that you reported here as it requires desktop-scaling to trigger. The updates are being sent to the client, but the opengl client was somehow failing to update the window front buffer.

according to your description, it is not the same bug. unfortunately, as stated in the OP, this issue happens during downscaling even if opengl is disabled in the client. so you can go ahead and reopen this, sorry...

@totaam totaam reopened this Aug 15, 2024
@totaam
Copy link
Collaborator

totaam commented Aug 15, 2024

how does the 6.x server do the downscaling?

Typically via libyuv:

def scale_yuv_image(self, image: ImageWrapper) -> ImageWrapper:

this is an operation that would benefit from GPU usage

Very much so.

even if today's CPUs can do it, it'd be much more efficient to do it in the GPU.

Not always. The pixel data usually resides in CPU memory.
Also, setting up a GPU context can be expensive, it isn't worth it for small sizes.

this also applies to image/video encoding of course. does the server use the GPU?

Yes, if available.
In that case, it will upload (if needed) the pixel data and do colourspace conversion, downscaling and encoding on the GPU.

can i tell the server NOT to use the GPU?

Sure: --video-encoders=x264,vpx

from the client?

No.
The client has its own switches for GPU decoding.

are you sure --opengl=off does not influence the server?

Yes.


Anyway, that turned out to be interesting, as I found lots of small issues along the way:

And the bilinear vs nearest sampling issue is now tracked here: #4326

The bad news is that apart from these issues, I left my connection to the shadow server open for hours and did not experience any other glitches.
(and that's to and from Linux Mint 22 - except for the OpenGL accelerated client which cannot run in VirtualBox, so I used Fedora 40 to connect)

@Lanchon
Copy link
Author

Lanchon commented Aug 15, 2024

The bad news is that apart from these issues, I left my connection to the shadow server open for hours and did not experience any other glitches.

you didn't need to. this bug is extremely reliable, happening 100% of the time and before the "xpra shadow server ready" notification (so, about 2 seconds after starting showing the remote display).

EDIT: disregard all that comes next, i got mixed up :(

can i tell the server NOT to use the GPU?

Sure: --video-encoders=x264,vpx

from the client?

No.

so the difference between your case and mine seems to be that my server is using the integrated intel GPU and yours is not. which points to they issue probably being worked-around by turning off GPU use in the server.

  1. is this assessment compatible with the fact that changing the client from 6.x to 5.x also sidesteps the issue? (this requires that using a 5.x client alters the codepaths related to GPU use in the 6.x server.)

  2. what can i add to client command xpra shadow ssh:XXX:0 --desktop-scaling=70% --opengl=off to disable GPU usage in the server so i can test the hypothesis?

@totaam
Copy link
Collaborator

totaam commented Aug 16, 2024

so the difference between your case and mine seems to be that my server is using the integrated intel GPU and yours is not.

I have no idea where you got this from.
Did you try to force the video encoders as per the previous comment? Did it help?

which points to they issue probably being worked-around by turning off GPU use in the server.

How so?

is this assessment compatible with the fact that changing the client from 6.x to 5.x also sidesteps the issue?

No, it is not.

what can i add to client command

Start your server independently via an ssh session and use the --video-encoders=x264,vpx switch.
Then connect your client as a separate command.

@Lanchon
Copy link
Author

Lanchon commented Aug 18, 2024

so the difference between your case and mine seems to be that my server is using the integrated intel GPU and yours is not.

I have no idea where you got this from.

sorry my bad, in my head i mixed the the bug report regarding start-desktop and this one regarding shadow desktop. so please disregard the previous message.


anyway...

on both client and server running v6.1.1-r0 on mint 22, as per the OP, trying this command fails because the screen stops refreshing, as described in the OP:

xpra shadow --opengl=off --desktop-scaling=70% ssh:XXX:0

however if i do this, the screen keeps on refreshing! the bug never shows up... weird:

ssh XXX xpra shadow :0
xpra attach --opengl=off --desktop-scaling=70% ssh:XXX:0
...
xpra stop ssh:XXX:0

so... there seems to be a race condition. take this command:

ssh XXX xpra shadow :0; sleep N; xpra attach --opengl=off --desktop-scaling=70% ssh:XXX:0; xpra stop ssh:XXX:0

in my case, if N >= 5 then the connection works fine. but if N is smaller it tends to stop refreshing the display. the stopping happens exactly as the "xpra shadow server ready" notification is shown in the server, as said earlier. so it's just a question of not connecting before the notification happens.

furthermore, if you don't wait and the connection freezes, simply disconnecting and attaching again fixes the issue. (of course this doesn't work with the xpra shadow ssh:XXX:0-style commands, as they tear the server down immediately on disconnection.)

Did you try to force the video encoders as per the previous comment? Did it help?

so if you add --video-encoders=x264,vpx and remove the sleep, it works fine:

ssh XXX xpra shadow --video-encoders=x264,vpx :0; xpra attach --opengl=off --desktop-scaling=70% ssh:XXX:0; xpra stop ssh:XXX:0

no delay is needed which means the race condition is gone.


maybe consider adding something like --server-option="--video-encoders=x264,vpx" to allow passing arguments to the server.

@totaam
Copy link
Collaborator

totaam commented Aug 18, 2024

so if you add --video-encoders=x264,vpx and remove the sleep, it works fine:

So the problem is very likely to be related to gstreamer or some other hardware accelerated encoder.
v6.2.x has a --no-gstreamer command line option for this purpose: #4330

maybe consider adding something like --server-option="--video-encoders=x264,vpx" to allow passing arguments to the server.

Most command line options are already forwarded to the server automatically, without the need for a server-option, unless they are listed in:

CLIENT_OPTIONS: list[str] = [

The problem with encoding options (like video-encoders, etc) is that the command line parsing will replace the defaults (ie: all) with the actual values to be used (ie: h264, vp8, ...) but this substitution is only valid on the client, as the server may have completely different modules installed.
Changing this is very much non-trivial.

@Lanchon
Copy link
Author

Lanchon commented Oct 14, 2024

hi,

tested on v6.2.0-r0:

  • xpra shadow ssh:XXX:0 --desktop-scaling=70%
    now works, well done! but there is a different issue: below the local cursor (which correctly echoes the remote cursor shape), xpra also shows a time-lagged remote cursor, but this extra cursor is corrupted.

    • this happens even without desktop scaling.
    • adding --opengl=off makes the issue go away, but results in constantly high cpu usage, so i prefer the defective cursor.
  • ssh XXX xpra shadow --video-encoders=x264,vpx :0; xpra attach --desktop-scaling=0.7 ssh:XXX:0; xpra stop ssh:XXX:0
    now became completely unusable due to extreme artifacts. the corrupted remote cursor issue also appears here.

    • adding --opengl=off does not fix the artifacts, but fixes the cursor.
    • disabling desktop scaling fixes the artifacts.
      • but adding --opengl=off on top, though this fixes the cursor, it introduces large black rectangular defects.
    • EDIT: i also tested --encoding=stream, hoping a simple video stream would get rid of the defects, but it made no change.

in all, the original bug report is fixed when --video-encoders=x264,vpx is not used, so you can close this issue if you want.

thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working geometry X11
Projects
None yet
Development

No branches or pull requests

2 participants