Remote monitoring configurable host

[artificial-aidan]

Feature Request

Configurable host for remote monitoring.

Currently the port can be configured when connecting to remote memray instances, but the host cannot be.

I have a POC done here and it seemed to be working. Was hoping to get some approval before documentation+test updates. And some guidance on versioning/release notes if approved.

[godlygeek]

Our initial version of this feature actually did include the ability to specify a remote host, but we wound up ripping it out. Our rationale was that it added a fair bit of complexity, that we imagined that in general people would be using the same host for visualizing the process as it's running on, and most importantly, that it could slow down the tracked process by quite a lot. When the socket send buffer fills up, the tracked process will block until the reader catches up. We've seen that slow down a tracked process by quite a lot even when using loopback addresses, and allowing traffic to remote machines would exacerbate that by quite a lot.

That said, it's not an unreasonable feature request. Can I ask about your setup? Why is it more useful for you to attach to a process from a different machine than it's running on, instead of the same one?

[artificial-aidan]

We are doing memory profiling on containers running in k8s. We can port forward to localhost and do it, but there is an added complication of most devs being on OSX, which isn't supported, so then you have to spin up a docker container, and getting to localhost on osx can be painful from docker, so you either run kubectl and portforward in the docker container, or just use host.docker.internal and you can connect to a port forwarded to your local machine.

So the setup is:

Developer laptop forwarding a k8s pod running memray
Docker container running memray locally
memray connected to host.docker.internal

There may be a better way to do the docker->localhost translation, but it was non-obvious when I was tinkering last night.

[godlygeek]

Hm. One notable thing about this setup is that it won't necessarily work for --native traces, because of the caveat documented here:

When generating reports for result files that contain native frames, the report needs to be generated on the same machine where the result file was generated. This is because the shared libraries that were loaded by the process need to be inspected by Memray to get the correct symbol names.

Much like interpreting loading a core file into a debugger, the reader side needs to go and hunt down .so files. If the ones it finds aren't identical to the ones that were used by the tracked process, we won't be able to correctly resolve native instruction pointers to file name and line number.

Admittedly, though, two containers running on one laptop is the most reasonable case of wanting to use a non-loopback address.

[artificial-aidan]

Yeah I ran into that when trying out the --native traces.

The changes I made to support it seemed quite minor, and I left default behavior the same.

[agronholm]

I just encountered this issue, trying to do live profiling in a Docker container, on the same host. Doesn't work because it binds to localhost.