Server.Utils.ImportCollection Bulk import

[Walishaha]

Hi Mike and team,

We have created a custom artifact using the Server.Utils.ImportCollection and the Glob plugin to import multiple zip files from a directory on the SFTP server (VR offline collection (JSON)) configured on the same VR server. It works perfectly and was able to import all the zip files currently available in that SFTP folder/directory.

Now, what we want is to update this custom server artifact in a way that whenever there is a new zip file collected in the SFTP, it automatically starts importing that new file as well. My current artifacts are not able to perform this job on the new files.

Can you please advise if there is some way like an infinite loop or any other way?

[scudette]

What you are looking for is an event query that just monitors the directory for new additions then imports them. We did a similar exercise in our training at blackhat - this is the gist of it

LET WatchDir = "C:/Share/*.zip"

SELECT * FROM foreach(row={
SELECT * FROM diff(query={
     SELECT OSPath  FROM  glob(globs=WatchDir)
}, period=10, key="OSPath")
WHERE Diff =~ "added"
}, query={
  SELECT * FROM Artifact.Server.Utils.ImportCollection(Path=OSPath)
})

Turn it into a server artifact and point it to the directory you want to watch then when new files appear it will automatically import them.

[Walishaha]

Hi Mike,

Thanks for the quick response. I tried to implement a server artifact as below..

name: artifact.Windows.Auto.Import_collection
description: |
This artifact will monitor the harcoded path for any new offline collection and will import is automatically.

Can be CLIENT, CLIENT_EVENT, SERVER, SERVER_EVENT
type: SERVER

sources:
- query: |
LET WatchDir = "C:/xxx/xxx/xxx/SFTP_Uploads/*.zip"
SELECT * FROM foreach(row={
SELECT * FROM diff(query={
SELECT OSPath, split(sep_string='.',string=Name)[0] AS Host FROM glob(globs=WatchDir)
}, period=10, key="OSPath")
WHERE Diff =~ "added"
}, query={
SELECT * FROM Artifact.Server.Utils.ImportCollection(Path=OSPath, Hostname=Host, ClientId=Auto)}
)

The above server artifact worked well, but for some reason, it went time out after 600 sec. I was expecting this to be running indefinitely.

Can you please review and advise if I am doing something wrong?

Many thanks

[scudette]

It needs to be a server event artifact because it needs to run indefinitely