Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MeshAgent creating powershell.exe handles #256

Open
JasperE84 opened this issue Aug 29, 2024 · 34 comments
Open

MeshAgent creating powershell.exe handles #256

JasperE84 opened this issue Aug 29, 2024 · 34 comments
Labels

Comments

@JasperE84
Copy link

On a Win10 virtual machine (on ESX) I have an issue with the Mesh Agent creating many powershell.exe instances.
This will slowly fill up the VM's memory until no virtual memory is left. The memory clears when the meshcentral is restarted.

Any ideas what could be causing this? I do not see any remote sessions in meshcentrals log for this host since the 24th of august and the memory consumption has been gradually growing since that date because of the many powershell.exe instances (see screenshot below).

Meshcentral version: 1.1.27

Host info:
OS Name: Microsoft Windows 10 Pro
OS Version: 10.0.18362 N/A Build 18362

MeshAgent info:
image

image
image

@si458 si458 added the bug label Aug 29, 2024
@si458
Copy link
Collaborator

si458 commented Aug 29, 2024

ooo this is interesting!
can you check what command the powershell is running at all?
as yes it should run powershell to do a some command etc but should then close afterwards!
so might be a bug not closing the powershell session afterwards

@JasperE84
Copy link
Author

Sure no problem.
Instances have this cmd line: image

More properties:
image

@si458
Copy link
Collaborator

si458 commented Aug 29, 2024

ah frig! thats what i was worried about!
we have no idea what command its running because we pipe it into the powershell!
BUT im sure i remember seeing this issue in the past and it was something to do with Languages in the powershell?
what language is your OS?

@JasperE84
Copy link
Author

It's set to Dutch (The Netherlands)

@JasperE84
Copy link
Author

Uninstalled the agent from this vm for now (wasnt really required on it and it seems like a somewhat isolated issue). Let me know if I can do anything to help debug the issue. I'll be glad to run some tests to fix this.

@si458
Copy link
Collaborator

si458 commented Aug 30, 2024

i found the commits i was talking about and it was to do with the bitlocker
Ylianst/MeshCentral#5832 Ylianst/MeshCentral#5747
can you verify if the VM in question was using bitlocker and was it enabled at all?

edit: can you also share a screenshot of the details tab? does it show information missing at all?

@JasperE84
Copy link
Author

Bitlocker is actually not enabled on this VM:

PS C:\WINDOWS\system32> manage-bde -status
BitLocker Drive Encryption: Configuration Tool version 10.0.18362
Copyright (C) 2013 Microsoft Corporation. All rights reserved.

Disk volumes that can be protected with
BitLocker Drive Encryption:
Volume C: []
[OS Volume]

    Size:                 59,05 GB
    BitLocker Version:    None
    Conversion Status:    Fully Decrypted
    Percentage Encrypted: 0,0%
    Encryption Method:    None
    Protection Status:    Protection Off
    Lock Status:          Unlocked
    Identification Field: None
    Key Protectors:       None Found

I have reinstalled the agent and these show up as details:
image

@si458
Copy link
Collaborator

si458 commented Aug 31, 2024

All looks OK, very weird?
Does it now show multiple powershells at all after u reinstalled it?

@JasperE84
Copy link
Author

Yep. Same issue.

@si458
Copy link
Collaborator

si458 commented Aug 31, 2024

Does it happen straight away right after u install the meshagent Or does it appear after a period of time?

Also one thing to try is watch it in process explorer (Count number of powershells)

Then run sysinfo from the console tab of the device,
Wait for it to return data, and see if more powershells appear, if they do then we have tracked down what it is!

@JasperE84
Copy link
Author

Straight away immediately after launch I see multiple powershell.exe processes spawning, only one of which persists and presumably fails to exit. Thats consistent behavior both before and after I reinstalled it. Then, with patience (I did not time it at all but I think like every 10-30mins or so) another powershell.exe process gets added to the process list.

@si458
Copy link
Collaborator

si458 commented Aug 31, 2024

Yep, that's the sysinfo !

Every 30ish mins, it will do a refresh of ur specs of ur computer!

One way to trigger it manually is to run sysinfo in the console tab of the device in the Web ui

If u have a moment and happy too, email me the output from sysinfo and I can compare with a working system and see what's missing.

Then the missing information will be the culprit of the rogue powershell

@JasperE84
Copy link
Author

JasperE84 commented Aug 31, 2024

Here ya go, hope this helps and is what you meant (I redacted some serialno's/uid's with 000-000-0000 like strings).

> sysinfo
{
 "hardware": {
  "windows": {
   "memory": [
    {
     "BankLabel": "RAM slot #0",
     "Capacity": "17179869184",
     "Caption": "Fysiek geheugen",
     "CreationClassName": "Win32_PhysicalMemory",
     "DataWidth": 32,
     "Description": "Fysiek geheugen",
     "DeviceLocator": "RAM slot #0",
     "FormFactor": 8,
     "MemoryType": 2,
     "Name": "Fysiek geheugen",
     "SMBIOSMemoryType": 3,
     "Tag": "Physical Memory 0",
     "TotalWidth": 32,
     "TypeDetail": 512
    }
   ],
   "osinfo": {
    "BootDevice": "\\Device\\HarddiskVolume1",
    "BuildNumber": "18362",
    "BuildType": "Multiprocessor Free",
    "Caption": "Microsoft Windows 10 Pro",
    "CodeSet": "1252",
    "CountryCode": "31",
    "CreationClassName": "Win32_OperatingSystem",
    "CSCreationClassName": "Win32_ComputerSystem",
    "CSName": "BATCH",
    "CurrentTimeZone": 120,
    "DataExecutionPrevention_32BitApplications": true,
    "DataExecutionPrevention_Available": true,
    "DataExecutionPrevention_Drivers": true,
    "DataExecutionPrevention_SupportPolicy": 2,
    "Description": "batch",
    "EncryptionLevel": 256,
    "ForegroundApplicationBoost": 2,
    "InstallDate": "20240706174958.000000+120",
    "LastBootUpTime": "20240829155423.500000+120",
    "Locale": "0413",
    "Manufacturer": "Microsoft Corporation",
    "MaxNumberOfProcesses": -1,
    "Name": "Microsoft Windows 10 Pro|C:\\WINDOWS|\\Device\\Harddisk0\\Partition2",
    "NumberOfProcesses": 180,
    "NumberOfUsers": 9,
    "OperatingSystemSKU": 48,
    "OSArchitecture": "64 bits",
    "OSLanguage": 1043,
    "OSProductSuite": 256,
    "OSType": 18,
    "Primary": true,
    "ProductType": 1,
    "RegisteredUser": "Windows-gebruiker",
    "SerialNumber": "00000-00000-00000-00000",
    "SizeStoredInPagingFiles": "7740164",
    "Status": "OK",
    "SuiteMask": 272,
    "SystemDevice": "\\Device\\HarddiskVolume2",
    "SystemDirectory": "C:\\WINDOWS\\system32",
    "SystemDrive": "C:",
    "Version": "10.0.18362"
   },
   "partitions": [
    {
     "BlockSize": "512",
     "Bootable": true,
     "BootPartition": true,
     "Caption": "Schijfnr. 0, partitienr. 0",
     "CreationClassName": "Win32_DiskPartition",
     "Description": "Installable File System",
     "DeviceID": "Disk #0, Partition #0",
     "Name": "Schijfnr. 0, partitienr. 0",
     "NumberOfBlocks": "1024000",
     "PrimaryPartition": true,
     "Size": "524288000",
     "StartingOffset": "1048576",
     "SystemCreationClassName": "Win32_ComputerSystem",
     "SystemName": "BATCH"
    },
    {
     "BlockSize": "512",
     "Caption": "Schijfnr. 0, partitienr. 1",
     "CreationClassName": "Win32_DiskPartition",
     "Description": "Installable File System",
     "DeviceID": "Disk #0, Partition #1",
     "Index": 1,
     "Name": "Schijfnr. 0, partitienr. 1",
     "NumberOfBlocks": "123842678",
     "PrimaryPartition": true,
     "Size": "63407451136",
     "StartingOffset": "525336576",
     "SystemCreationClassName": "Win32_ComputerSystem",
     "SystemName": "BATCH"
    },
    {
     "BlockSize": "512",
     "Caption": "Schijfnr. 0, partitienr. 2",
     "CreationClassName": "Win32_DiskPartition",
     "Description": "Unknown",
     "DeviceID": "Disk #0, Partition #2",
     "Index": 2,
     "Name": "Schijfnr. 0, partitienr. 2",
     "NumberOfBlocks": "952320",
     "PrimaryPartition": true,
     "Size": "487587840",
     "StartingOffset": "63933775872",
     "SystemCreationClassName": "Win32_ComputerSystem",
     "SystemName": "BATCH"
    }
   ],
   "cpu": [
    {
     "Caption": "Intel64 Family 6 Model 85 Stepping 7",
     "DeviceID": "CPU0",
     "Manufacturer": "GenuineIntel",
     "MaxClockSpeed": 2993,
     "Name": "Intel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz",
     "SocketDesignation": "CPU #000"
    },
    {
     "Caption": "Intel64 Family 6 Model 85 Stepping 7",
     "DeviceID": "CPU1",
     "Manufacturer": "GenuineIntel",
     "MaxClockSpeed": 2993,
     "Name": "Intel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz",
     "SocketDesignation": "CPU #001"
    }
   ],
   "gpu": [
    {
     "Name": "VMware SVGA 3D",
     "CurrentHorizontalResolution": 2184,
     "CurrentVerticalResolution": 864
    }
   ],
   "drives": [
    {
     "Caption": "VMware Virtual disk SCSI Disk Device",
     "DeviceID": "\\\\.\\PHYSICALDRIVE0",
     "Model": "VMware Virtual disk SCSI Disk Device",
     "Partitions": 3,
     "Size": "64420392960",
     "Status": "OK"
    }
   ],
   "volumes": {
    "A": {
     "type": "Unknown",
     "size": 0,
     "sizeremaining": 0,
     "removable": true
    },
    "C": {
     "type": "NTFS",
     "size": 63407448064,
     "sizeremaining": 23507181568
    },
    "D": {
     "type": "Unknown",
     "size": 0,
     "sizeremaining": 0,
     "cdrom": true
    }
   }
  },
  "identifiers": {
   "bios_date": "20201112000000.000000+000",
   "bios_vendor": "Phoenix Technologies LTD",
   "bios_version": "6.00",
   "bios_serial": "VMware-42 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00",
   "bios_mode": "Legacy",
   "board_name": "440BX Desktop Reference Platform",
   "board_vendor": "Intel Corporation",
   "product_uuid": "00000000-0000-0000-0000-000000000000",
   "product_name": "VMware Virtual Platform",
   "gpu_name": [
    "VMware SVGA 3D"
   ],
   "storage_devices": [
    {
     "Caption": "VMware Virtual disk SCSI Disk Device",
     "Model": "VMware Virtual disk SCSI Disk Device",
     "Size": "64420392960"
    }
   ],
   "cpu_name": "Intel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz"
  },
  "agentvers": {
   "openssl": "1.1.1s",
   "duktape": "v2.6.0",
   "commitDate": "2022-12-02T19:42:16.000Z",
   "commitHash": "1ac56a2aa26cc403efe758fc53386a3dc0e706b7",
   "compileTime": "12:12:34, Dec  9 2022"
  },
  "network": {
   "dns": [
    "10.1.1.1",
    "10.1.1.2"
   ]
  }
 },
 "pendingReboot": null,
 "hash": "A0810A47D335B7A2A57137B47EC96679819E711EBE3F6D11481E02604F3A45EB50A5F3A545230DABE502F72C2E94C2BF"
}

@JasperE84
Copy link
Author

Some more info;

  • After a manual sysinfo command another stale powershell.exe process does -not- get stuck and added to the process list. I verified this by running the powershell cmd @(get-process -ea silentlycontinue "powershell").count before and after sysinfo. During sysinfo it will go +1 and when the output is shown in the console tab it will go -1.
  • If I have just restarted the meshagent the sysinfo will output its result within under 5 seconds. When I tried running sysinfo on the agent process which I reinstalled and started just this morning, it would take much much longer to output results.

Not sure if this helps but I can imagine it is relevant info

@JasperE84
Copy link
Author

JasperE84 commented Aug 31, 2024

Also,

  • Starting either a new powershell-administrator session on the console tab or desktop session in meshcentral on this host took a long while (powershell terminal would just say "connected" and not show terminal output. desktop session would show "connected" and show a black screen for a while)
  • After restarting the agent on the host, everything was snappy and responsive again on starting new sessions.

By the way as this is the only host on my ~200 agent installs, (a few of which are on windows server vm's instead of win10 vm in the same ESX infrastructure) which seemingly is showing this behavior I also ran a sfc /scannow and DISM /Online /Cleanup-Image /ScanHealth command to rule out problems originating from invalid windows binaries.

@HuFlungDu
Copy link

I'm not able to replicate this issue, but I see that the windows_volumes uses a different line terminator than all other uses of powershell:

child.stdin.write('Get-Volume | Select-Object -Property DriveLetter,FileSystemLabel,FileSystemType,Size,DriveType | ConvertTo-Csv -NoTypeInformation\nexit\n');

It uses \n instead of \r\n. It looks like your volumes are returning, and it should only return if that powershell exits, but it does run on startup and is part of the sysinfo, so that might be a place to look for the issue, Simon.

Is the information returned by the volumes accurate for that device?

@si458
Copy link
Collaborator

si458 commented Sep 3, 2024

@HuFlungDu the volume stuff is now returned by the computer-identifiers.js on the meshcentral server itself
So would need to update there if the is a bug with volumes

@si458
Copy link
Collaborator

si458 commented Sep 3, 2024

@HuFlungDu ive just pushed the fix in the meshcentral repo as i do believe you are right! Ylianst/MeshCentral@cf23a3d
its starting a new line but never actually pressing the enter key as it where to exit the powershell!
we do need to also replace all the inline powershell commands at some point (on my to do list)
replacing them with rather then writing, they run the command directly using -Command
then when something like this happens,
we can see what command was actually running and blocking things

@si458
Copy link
Collaborator

si458 commented Sep 3, 2024

@JasperE84 if you are happy too,
replace the file agents/modules_meshcore/computer-identifiers.js with the master from the meshcentral repo
https://github.com/Ylianst/MeshCentral/blob/master/agents/modules_meshcore/computer-identifiers.js
then restart meshcentral
and see if the issue still happens!

@si458
Copy link
Collaborator

si458 commented Sep 9, 2024

@JasperE84 1.1.29 was just released which included the fix above,
can you try upgrading your server and then seeing if the issue is resolved?

@JasperE84
Copy link
Author

JasperE84 commented Sep 9, 2024

Thanks, I've just upgraded the docker container to see if it is fixed and will let you know soon. By the way the hanging powershell.exe instances seem to terminate when the meshserver (not meshagent) is restarted after a docker update.

@si458
Copy link
Collaborator

si458 commented Sep 9, 2024

@JasperE84 that's an interesting find...
Let us know if it's fixed with the latest 1.1.29 update

@JasperE84
Copy link
Author

Unfortunately 3 stale powershell.exe processes within 40mins.
Disregard my statement that the powershell.exe's clear when meshserver is restarted that is not the case.
In fact, each meshcentral restart another powershell.exe instance is created. (reproducable on this box).

@si458
Copy link
Collaborator

si458 commented Sep 9, 2024

can you share your config.json at all?
what happens if you run the sysinfo again from the console tab of the device in the web ui?
does a new powershell.exe appear then stay there?

@si458
Copy link
Collaborator

si458 commented Sep 9, 2024

can u also share you info from the server console tab in the web ui too?
has info like what version u running and nodejs etc.

@JasperE84
Copy link
Author

Interesting find, restarting meshserver results in extra powershell.exe hanging.
After meshagent restart:
image
After subsequent meshserver restart:
image

But running sysinfo does not
After meshagent restart
image
After subsequent sysinfo on agent console tab
image

> info
Current Core: Dec 9 2022, 100151277
Agent Time: 2024-09-09 16:07:10.812+02:00.
User Rights: 0xffffffff.
Platform: win32.
Capabilities: 15.
Server URL: wss://#########.#############:####/agent.ashx.
OS: Microsoft Windows 10 Pro - 18362.
Modules: amt-apfclient, amt-lme, amt-manage, amt-mei, computer-identifiers, monitor-border, smbios, sysinfo, util-agentlog, wifi-scanner-windows, wifi-scanner, win-console, win-deskutils, win-info, win-securitycenter, win-terminal, win-virtual-terminal, win-volumes.
Server Connection: true, State: 1.
Node ID: ############
Application Location: C:\Program Files\Mesh Agent\
> info
{
    "meshVersion": "v1.1.29",
    "nodeVersion": "v20.15.1",
    "runMode": "Hybrid (LAN + WAN) mode",
    "productionMode": true,
    "database": "MongoDB",
    "dbChangeStream": false,
    "dbBulkOperations": false,
    "platform": "linux",
    "arch": "x64",
    "pid": 17,
    "uptime": 446.138698989,
    "cpuUsage": {
        "user": 5246297,
        "system": 1031461
    },
    "memoryUsage": {
        "rss": 118198272,
        "heapTotal": 49451008,
        "heapUsed": 47296632,
        "external": 51325021,
        "arrayBuffers": 49075685
    },
    "warnings": [],
    "allDevGroupManagers": []
}

@JasperE84
Copy link
Author

JasperE84 commented Sep 9, 2024

$ cat config.json
{
  "$schema": "https://raw.githubusercontent.com/Ylianst/MeshCentral/master/meshcentral-config-schema.json",
  "settings": {
    "plugins":{"enabled": false},
    "mongoDb": "mongodb://mongodbadmin:#####@mongodb:27017",
    "cert": "mcentral.#####.##",
    "_WANonly": true,
    "_LANonly": true,
    "sessionKey": "#############",
    "port": 4430,
    "aliasPort": 443,
    "agentAliasPort": 4021,
    "agentAliasDNS": "mcentral-agent.#######.##",
    "redirPort": 800,
    "_redirAliasPort": 80,
    "AgentPong": 300,
    "TLSOffload": true,
    "SelfUpdate": false,
    "AllowFraming": false,
    "WebRTC": false
  },
  "domains": {
    "": {
      "title": "######",
      "_title2": "docker",
      "welcomeText": "Login credentials only available for  Sysadmins.<br />Are you a supplier or vendor? Please contact your  contact to retrieve a temporary link for support tasks.",
      "minify": true,
      "NewAccounts": false,
      "localSessionRecording": false,
      "_userNameIsEmail": true,
      "certUrl": "https://mcentral.########.##:443",
      "SessionRecording": {
        "onlySelectedDeviceGroups": false,
        "filepath": "/opt/meshcentral/meshcentral-data/session_recordings",
        "index": true,
        "maxRecordingSizeMegabytes": 2048,
        "protocols": [1,2]
      }
    }
  },
  "_letsencrypt": {
    "__comment__": "Requires NodeJS 8.x or better, Go to https://letsdebug.net/ first before>",
    "_email": "[email protected]",
    "_names": "myserver.mydomain.com",
    "production": false
  }
}


@si458
Copy link
Collaborator

si458 commented Sep 9, 2024

thank you @JasperE84
i just wanted to check

  1. wasnt anything stupid in your config.json (it wasnt 👍)
  2. wasnt anything stupid like old nodejs (it wasnt 👍)
  3. and you was 110% on the latest version (which you are 👍)

please can you now try these few things for me now as ur on the latest release

  1. connect to admin powershell in meshcentral for me (right click connect and click admin powershell)
  2. run whoami check its output (should say nt authority\system)
  3. run Get-Volume | Select-Object -Property DriveLetter,FileSystemLabel,FileSystemType,Size,SizeRemaining,DriveType | ConvertTo-Csv -NoTypeInformation and let me have its output
  4. run Get-BitLockerVolume | Select-Object -Property MountPoint,VolumeStatus,ProtectionStatus | ConvertTo-Csv -NoTypeInformation and let us have its output
  5. can you go into the console tab of the device in meshcentral and run av and let us have its reply?
    (it might take a min to return but no more)

@JasperE84
Copy link
Author

JasperE84 commented Sep 9, 2024

Sure, here you go!

Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved.

Try the new cross-platform PowerShell https://aka.ms/pscore6

PS C:\Program Files\Mesh Agent> whoami
nt authority\system
PS C:\Program Files\Mesh Agent> Get-Volume | Select-Object -Property DriveLetter,FileSystemLabel,FileSystemType,Size,SizeRemaining,DriveType | ConvertTo-Csv -NoTypeInformation
"DriveLetter","FileSystemLabel","FileSystemType","Size","SizeRemaining","DriveType"
,"Door systeem gereserveerd","NTFS","524283904","485818368","Fixed"
"C","","NTFS","63407448064","23255228416","Fixed"
,"","NTFS","487583744","70332416","Fixed"
"D","","Unknown","0","0","CD-ROM"
"A","","Unknown","0","0","Removable"
PS C:\Program Files\Mesh Agent> Get-BitLockerVolume | Select-Object -Property MountPoint,VolumeStatus,ProtectionStatus | ConvertTo-Csv -NoTypeInformation
"MountPoint","VolumeStatus","ProtectionStatus"
"C:","FullyDecrypted","Off"     
PS C:\Program Files\Mesh Agent>
> av
[
 {
  "product": "CrowdStrike Falcon Sensor",
  "updated": true,
  "enabled": true
 }
]

@si458
Copy link
Collaborator

si458 commented Sep 9, 2024

All looks OK, but just out of curiosity

What does this output?

U will need to use cmd not powershell in meshcentral web ui

manage-bde -protectors -get C: -Type recoverypassword

@JasperE84
Copy link
Author

C:\Program Files\Mesh Agent>manage-bde -protectors -get C: -Type recoverypassword
BitLocker Drive Encryption: Configuration Tool version 10.0.18362
Copyright (C) 2013 Microsoft Corporation. All rights reserved.

Volume C: []
Key Protectors of Type Numerical Password

ERROR: No key protectors found.

C:\Program Files\Mesh Agent>

@InternalISTeam
Copy link

Hello,
i'm posting here, as it is quiet related to powershell instances.
We have an issue on our EDR : a huge amount of alerts are triggered by these powershell script (-noprofile option).
So, is it possible (i cnnot find it if it exists) :

  1. to have an option to disable these telemetry scripts ?
  2. to identify clearly the powershell process with a meshcentral tag somewhere, to isolate windows events / alerts and tell EDR it is not a threat ?
  3. to set the powershell script runs frequency ?

Thanks a lot for this tool 👍
Pierre

@thinrope
Copy link

FYI, there is a way to log what commands get executed inside the PS session (even piped that is) by enabling the corresponding log on Windows. See https://www.socinvestigation.com/threat-hunting-using-powershell-and-fileless-malware-attacks/ for more info.

As the last post by @InternalISTeam can indeed be a nuisance in monitored environment and to improve general security design, I think at least the telemetry stuff should be hard-coded in a script file with a specific path and executed (hopefully without arguments). That will allow easier auditing and thus safe whitelisting.
@InternalISTeam as for now, you can look at the parent of the powershell.exe (PPID) and if it is MeshAgent.exe - whitelist. Of course that means you'll be "blind" to anyone genuinely attacking a host via meshcentral :-)

@InternalISTeam
Copy link

InternalISTeam commented Oct 22, 2024

Hello, thanks but we cannot find the parent from windows events (or hard to have it), no solution right now... so if one of the solution explained above could be developed it would be very nice !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

5 participants