Unable to start sshd on Windows 11, ssh_rsa_host_key access problem

[peterpc5j]

I have been trying to install the OpenSSH server on a Windows 11 machine. I am following the instructions on here (https://github.com/PowerShell/Win32-OpenSSH/wiki/Install-Win32-OpenSSH-Using-MSI) and https://github.com/PowerShell/Win32-OpenSSH/wiki/Troubleshooting-Steps for troubleshooting - but things start to go wrong at a very early stage. I suspect I'm having multiple problems really. I have also installed OpenSSH on another machine, Windows 10, and it seems to run cleanly there, so I went to compare the debug logs to see where things start to diverge, although the versions are different (OpenSSH 8.6 on Win10, 9.4 on Win11, but I don't think that's the issue).

First of all, I go in debug mode and manual startup because when I try to start sshd as a windows service with net start I get either an error message that the service is starting or terminating, or, when it does try to boot up, it says that the service cannot be started (together with a NET HELPMSG 3523, no useful information there). When I start it through Windows' control panel, it just keeps restarting in an endless loop. I don't find any log anywhere when I do this.
So, in order to find out where the problem is, I run sshd -ddd on both machines.
The first weird log message on Windows 11 relates to a problem to access the host_ssh_rsa_key file. I did run the FixHostFilePermissions.ps1 script and it now says that everything 'looks good'. However, I get messages in Win11 that I don't get in Win10 that seem related to file permissions:

debug2: load_server_config: filename __PROGRAMDATA__\\ssh/sshd_config
debug3: w32_fstat ERROR: bad fd: 3
debug2: load_server_config: done config len = 2241
debug2: parse_server_config_depth: config __PROGRAMDATA__\\ssh/sshd_config len 2241
debug3: __PROGRAMDATA__\\ssh/sshd_config:24 setting SyslogFacility LOCAL0
debug3: __PROGRAMDATA__\\ssh/sshd_config:26 setting LogLevel Debug3
debug3: __PROGRAMDATA__\\ssh/sshd_config:40 setting AuthorizedKeysFile .ssh/authorized_keys
debug3: __PROGRAMDATA__\\ssh/sshd_config:81 setting Subsystem sftp      sftp-server.exe
debug3: checking syntax for 'Match Group administrators'
debug1: sshd version OpenSSH_for_Windows_9.4, LibreSSL 3.7.3
debug1: get_passwd: lookup_sid() failed: 1332.
debug3: failed to open file:C:/ProgramData/ssh/ssh_host_rsa_key error:5
debug3: Failed to open file:C:/ProgramData/ssh/ssh_host_rsa_key error:13
debug1: Unable to load host key: __PROGRAMDATA__\\ssh/ssh_host_rsa_key

Then the log goes on and it does go to a point where it is listening to port 22. When I try to login on localhost afterwards, I get other problems, but I would like to start with the apparently problematic access. I would try to delete the host rsa key but even as Administrator I do not have the right permissions (nor am I the owner, I don't know what is) to remove or change it. I don't have this problem on Win10, but I didn't run the file permissions script.

So how do I fix the ssh_rsa_host_key, for starters?

[maertendMSFT]

Can you confirm that you are running sshd as an admin or system? If you are not running as an admin in debug mode it explains why you are unable to load the keys. If you are running as a standard user, then it is expected that you are unable load the keys.

If you are still having issues, take a look at the solution in this issue: #2161

[peterpc5j]

Dear Danny, thanks for helping.

Short answer: use this install script - https://github.com/PowerShell/openssh-portable/blob/latestw_all/contrib/win32/openssh/install-sshd.ps1

Here is the rest of the diagnostics of this problem, in case it is helpful to anyone.

One observation I failed to mention: other keys load apparently without error.
I run sshd, either manually or through the net start options, from an Admin Powershell.
I noticed that, on the machine it fails, the rsa key has other mode attributes than on the machine where it doesn't fail:

 Get-Item -path .\ssh_host_rsa_key

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-ar---        07/08/2023     16:08           2610 ssh_host_rsa_key

note the 'read' mode - versus, where it does work:

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a----        30/11/2023     09:57           2622 ssh_host_rsa_key

I found out about the Get-Acl command to display owner information. I get, in windows 11 (where it doesn't work):

Path   : Microsoft.PowerShell.Core\FileSystem::C:\ProgramData\ssh\ssh_host_rsa_key
Owner  : AUTORIDADE NT\SISTEMA
Group  : [Computer name]\None
Access :
Audit  :
Sddl   : [...] 

while where it does work it is like this:

Path   : Microsoft.PowerShell.Core\FileSystem::C:\ProgramData\ssh\ssh_host_rsa_key
Owner  : AUTORIDADE NT\SISTEMA
Group  : AUTORIDADE NT\SISTEMA
Access : AUTORIDADE NT\SISTEMA Allow  FullControl
         BUILTIN\Administradores Allow  FullControl
Audit  :
Sddl   :  [...]

I have a similar profile with the host key files that do not raise an error on my Windows 11 installation. AUTORIDADE NT is NT Authority in english-language versions of Windows.

This clearly suggests that the system identifier of the ssh_host_rsa_key is wrong, and that the script to fix host keys that maybe should detect this, doesn't:

[*] C:\ProgramData\ssh\ssh_host_rsa_key
      looks good

After running the install-sshd.ps1 script that was linked from the thread you referred to, a number of file permissions changed. user access to the logs and the public keys was removed, and access to the rsa key was changed:

  [*] C:\ProgramData\ssh\ssh_host_rsa_key
'AUTORIDADE NT\SISTEMA' now has FullControl to 'C:\ProgramData\ssh\ssh_host_rsa_key'.
'BUILTIN\Administradores' now has FullControl to 'C:\ProgramData\ssh\ssh_host_rsa_key'.
      Repaired permissions

Now I have:

PS C:\ProgramData\ssh> Get-Acl .\ssh_host_rsa_key |Format-List


Path   : Microsoft.PowerShell.Core\FileSystem::C:\ProgramData\ssh\ssh_host_rsa_key
Owner  : AUTORIDADE NT\SISTEMA
Group  : [computer name]\None
Access : AUTORIDADE NT\SISTEMA Allow  FullControl
         BUILTIN\Administradores Allow  FullControl
Audit  :
Sddl   :  [...]

The Group did not change, but Acess did. I have no problem anymore with the rsa key when starting the service manually. "net start sshd" and 'Start-Service -Name "sshd"' seem to run without issues now. I tested the connection through localhost and from outside the server.

It works!

Thank you very much!

I think the install script should be mentioned on the OpenSSH installation pages.