Podman ssh connection lost after some time on Mac (Apple M4)

[aaronpliu]

Issue Description

Install Podman (v5.7.1) on MacOS M4 and installed Jenkins in it. Setup ssh connection to agent. It's OK to running.
After some time, all the Jenkins agents disconnect due to ssh connection refused.
When run "podman machine ssh" to step into VM, and run "ssh name@ip" to any one of agent, the connection refused (looks like the ssh port is not available)
If restart host and start podman VM again, then it resumed ssh connection. Not sure what's wrong for that

Steps to reproduce the issue

Steps to reproduce the issue

1. Install Podman (v5.7.1) and applehv
2. Install Jenkins (or just using ssh to remote)
3. check ssh connection after some time
4. if restart host or kill gvproxy and restart VM, the ssh connection resumed

Describe the results you received

all Jenkins agent offline

Describe the results you expected

ssh connection is good

podman info output

ssh: connect to host 192.168.0.20 port 22: Connection refused

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

Additional environment details

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting