Help with using Lailloken-UI with remote desktop application (Moonlight)

[Stethor]

Greetings.

I have a weird and very specific issue with not being able to use your UI properly with remote desktop applications.
For example when using "Moonlight" I'm not able to perform any hold-button actions in your UI: Mouse doesn't work for holding, keyboard too. Omni-key doesn't seem to work either. So the use of the tool is limited. I tested using UI from directly controlling the PC in question by plugging in all controls - and in this case everything works well.

Unfortunately I don't have an opportunity to not use this PC as a remote desktop, I'm really limited in this regard.

If I have to guess there is some AHK limitations and also how streaming remote desktop handles the holding actions, however I digged as much AHK forums as I could, while also digging Moonlight's GitHub for some clues to what even do about it, but this was completely unsuccessful.

I am a bit lost, but I really want to use your UI, so I decided to ask directly for help. I would understand if this issue is too specific and you wouldn't be able to help, but at least I want some tip where I can start on fixing it myself, or if I need to contact someone else (Devs from Moonlight or AHK)

[Lailloken]

Is Moonlight running on a Windows PC (EDIT: i.e. is the client a Windows PC), and does Moonlight forward clipboard contents when copying something on the host-PC? There's a potential workaround that comes to mind but it would only work if these two aspects apply.

[Stethor]

Ugh, butterfingers. Sorry about closing and reopening. Moonlight runs on Client PC that works on Win10, the Host PC uses "Sunshine" to stream contents to Client. Moonlight does copy the contents of clipboard of the client PC to host PC, but only with a special hotkey: Ctrl+Alt+Shift+V. That's what I know for now

[Lailloken]

Moonlight does copy the contents of clipboard of the client PC to host PC

I meant the other way around so that item-text copied from the game on the host is available on the client-side.

Off the top of my head, there are two possible workarounds (if long-presses cannot be fixed) that both involve running my tool on the client-PC instead of the host-PC:

• treat the moonlight-window as a streaming-client (GeForce Now), but you'd lose features that are reliant on clipboard-data and the client.txt log-file

• treat the moonlight-window as a game-client and hope there's no delay in accessing the host-clipboard: you'd also have to set up file-access over your local network so that the client can access the PoE game-folder and the client.txt log-file inside

[Stethor]

I can try second solution, I'm already have a setup for LAN file access, as for the clipboard from host PC this might be a problem, but I would try to dig some info about it. Thanks for tips, I will let you know if I got anything to work

[Stethor]

I can make clipboards of both host PC and client PC be completely connected by using Windows application "Mouse without borders", or something else similar with easy clipboard sharing. So this part ain't the issue, but what to do now?

[Lailloken]

Read all the steps first, then follow one by one:

• download and extract the tool on the client-PC

• get a copy from your host's PoE-config-file (usually X:\Documents\My Games\Path of Exile\Production_config.ini) and save it in the same location on the client-PC

  • the tool checks that to get the game's current window-settings and some other minor things

  • alternatively, if you also have network access to this file, you can skip this

• open Lailloken UI.ahk in an editor and search for this block:

GroupAdd, poe_window, ahk_class POEWindowClass
GroupAdd, poe_window, ahk_exe GeForceNOW.exe
GroupAdd, poe_window, ahk_exe boosteroid.exe

• add this line underneath: GroupAdd, poe_window, ahk_exe XYZ.exe (XYZ being moonlight's exe)

• search for this block:

If FileExist(SubStr(poe_log_file, 1, InStr(poe_log_file, "\",,,LLK_InStrCount(poe_log_file, "\"))) "logs\client.txt")
	poe_log_file := SubStr(poe_log_file, 1, InStr(poe_log_file, "\",,,LLK_InStrCount(poe_log_file, "\"))) "logs\client.txt"
Else poe_log_file := SubStr(poe_log_file, 1, InStr(poe_log_file, "\",,,LLK_InStrCount(poe_log_file, "\"))) "logs\kakaoclient.txt"

• add this line underneath: poe_log_file := "X:\Path of Exile\logs\client.txt" (you'll have to figure out the correct path for network locations, I don't know it myself)

• save the file, launch the script on client-PC when moonlight is already running

• if you didn't choose the config-copy option at the top, a Windows dialog should open, asking you to locate the games config.file (try to do that, but I don't know how well it handles network locations)

  • if the dialog doesn't work, you can alternatively add a manual override to ini\config.ini by adding the line PoE config-file="X:\Documents\My Games\Path of Exile\production_Config.ini" in the [settings] section

• if luck is on our side, it'll work and treat moonlight as a game-client and use the client.txt (and config) via the network

[Stethor]

Hey, I tried to make it work, but it seems it works only partially: the UI tool seems to hook to shared files in local network, but I can't properly make UI to work and can't test it to be sure - the script works at first and I can change settings, but closing settings or trying to use any opened UI windows for some reason completely hides all UI, but it still runs as I have icon in system tray and I can make emergency script restart. Also for some reason it doesn't always wants to show itself when I switch to moonlight window, and sometimes it just pops in for a frame and disappears again. While UI was on screen I tried to use omni-key to use some functions, but seems this didn't really make it to work. I tried to run Moonlight in fullscreen and in windowed fullscreen, but the result is the same. I think you almost made it to work, but something is off. It would be also nice to prove that UI actually reads the stuff from shared files in local network.

[Lailloken]

Oh, sorry. I forgot one line that also needs to be added after the first block you searched for:

GroupAdd, poe_ahk_window, ahk_exe XYZ.exe

This missing line is why it every window just hides immediately. Once you can actually open the settings menu, you can test the "current char" field and type in your recent characters to see if the tool finds them in the client.txt.

[Stethor]

Added new line into the script. I tried to enter current char and it's actually seems to work. I will test the UI more in depth tomorrow and report the results here. Thank you for you hard work and time spent.

[Stethor]

I tried to do some screen checks and also text recognition in "TLDR-Tooltips" feature, but none of them work reliably: screen checks often fail after proper calibration, and only rarely and once they return positive, but after that they don't work. And text recognition from "TLDR-Tooltips" works very poorly and often eats parts of words. But now it doesn't work at all and just sends an error: "Screen cap failed". Also the moonlight app have 2 separate windows that both named as Moonlight.exe, but have different exe IDs. Problem is that sometimes UI hooks to wrong windows with host selection, instead of streaming window and overall perhaps that's why the screen checks seems to fail?

[Lailloken]

Hmm, instead of grouping by exe, you could try grouping by window-title of the correct window (but the title would have to be consistent):

GroupAdd, poe_window, WhatEverMoonlightUsesAsTitle
GroupAdd, poe_ahk_window, WhatEverMoonlightUsesAsTitle

Window-title refers to the name of a window (shown on taskbar, or when alt-tabbing). You don't have to specify the whole title, just make sure the snippet is spelled correctly.

[Stethor]

I would try that - correct window always have the host PC's name, which is constant. There shouldn't be any issues in that regard

[Stethor]

I did that and it works, however I still can't do proper screen checks - test returns positive only when I do test right after calibration, if I move my char even slightly then test starts to fail. Same with inventory check, which I found weird. And on text recognition I'm leaving screenshot of TLDR function and how it(poorly) tries to detect words.

[Lailloken]

Do you know whether Moonlight is applying filters to the image? Things like sharpening filters are incompatible with the tool because they alter the image too much and prevent screen reading/recognition.

What are your quality settings? Would be a shame if we got this far, only to be stopped by image quality.

[Stethor]

I thought about image quality in the first place, however I am located in local network, have 1080p resolution for stream decoder and 70 Mbit/s bitrate. As for filters I don't really know if Moonlight at it's own does something with like image compression, which may shift colors around. I also have NVidia App installed on host PC, but I just checked and no filters are enabled. Perhaps we can test if the issue with image recognition if we somehow compare images from my Moonlight stream and image that 100% works with UI? May you send such a screenshot? If the issue on the my end I would try to fix it myself, somehow..

[Lailloken]

I guess I overestimated the image quality via local streaming. I think you have to modify two lines in modules\screen-checks.ahk:

vars.pixelsearch.variation := 0

and

vars.imagesearch.variation := 15

These represent the strictness applied to screen-checks. Increase these by 15, save, then restart the tool and recalibrate and test the screen-checks. Repeat until image and pixel-checks work reliably and consistently. As for TLDR-tooltips, there is no way to modify the results. It reads whatever it can from the screenshot.

[Stethor]

No luck with that. I got up to 90 and 105 and checking become so loose that even stationary the success of checks was completely random. I still think that something may distort the screen image, and I'm not sure it's the stream quality problem, as I was using parsec for web streaming from host before, and in comparison Moonlight in local network just doesn't have any visible quality degradation and it at least look like source-quality image. I still think that image comparison from healthy screen-check and my failed screen check may say something about the issue

[Lailloken]

How consistent are the screen-checks if you run the tool on the host? Did you encounter any problems there when you double-checked your initial long-press issues?

Checks failing immediately after moving your character are a tell-tale sign for some form of (contrast-)adaptive sharpening/filtering. I've had many user reports of issues with screen-checks, and disabling ReShade or GeForce Experience always fixed it. So I'd say moonlight is doing something with the image.

You can enable the act-tracker, calibrate the skilltree screen-check, open the screen-check img-folder, rename the skilltree-file, move the skilltree around a bit, then recalibrate the check. You'll have two image-files to compare (there are tools online for that).

[Stethor]

Sorry for missing from this issue page. I tried to do a lot of stuff regarding trying to identify the problem. I actually ran your UI on host PC, and all screen checks were working correctly and well. So something is altering image ether on client PC or the moonlight itself. I tried and was digging for a lot of things, but for no real luck. There should be no filters whatsoever on my client PC and I tried to turn off everything that could enable them. I even tried to check pixels in paint by inserting 2 images with opened area info, and I found out that in different background for some reason pixel colors shift a little bit. Only thing I can think of is to install game on client PC and check how screencheck behaves there without moonlight to hopefully determine the true issue source. However even if I would find the source - I have no idea what to do, ether if the Moonlight is the issue, or the client.

[Stethor]

Even TLDR tooltip image calibration works perfectly.
Honestly I think it would be so much easier if I somehow made mouse holding work for streaming. Oh and also for some reason when I try to make manual screencheck on host and without connecting real mouse to host pc - the screen check completely freezes without being able to even press ESC to exit from it, and not even to alt-tab, ctrl-shift-esc or ctrl-alt-del. Only Win + Space for emergency script restart helps.

[Lailloken]

Do you have a spare set of mouse & keyboard for the host to test whether anything changes about inputs sent via Moonlight:

• run tool on host, with mouse and keyboard plugged in

• connect via Moonlight and test omni-key, etc.

Maybe AutoHotkey doesn't like it when the system doesn't have any input devices plugged in. Other than that, I don't know what Moonlight is doing with regards to inputs. The tool doesn't use anything exotic, it's simple hotkeys bound to ESC, middle mouse-button, etc.

[Stethor]

I checked connecting other mouse to host and this didn't work.
However I know what the main problem regarding holding the button: it's the overall issue with all streaming apps, as they can't actually transfer any holding action through, as this would eat too much bandwidth. Instead they "simulate" holding by sending the same key with a high frequency, which works fine for windows and most games on it's own, but for some reason AHK itself doesn't have this build in feature in "(hold) detect function", so AHK doesn't detect this as "holding". I heard about some kind of workarounds for that, but most involve a bit of rewriting the hold function in script, which is presumably too much to ask for just myself. Theoretically there also might be something for host PC that "smooths out" incoming inputs and simulates them as holding. I will dig into host-pc streaming settings and try to find something, but I already tried that once, so I'm not sure I would be able to find something new.