Windows.Hayabusa.Rules Upload to elastic is broken

[D-V-W]

after the 01/10/24 update to the Windows.Hayabusa.Rules artifact that added timestamp(epoch=System.TimeCreated.SystemTime) AS Timestamp to every select and to the where clause query. the ability to upload the data to elastic(using a simplified version of Elastic.Flows.Upload) has been broken.
the error it returns is as such

"reason":"[1:1601] failed to parse field [_Event.System.Keywords] of type [long] in document with id '154611705006142065'. Preview of field's value: '9223372036854775808
"reason":"[1:1620] Numeric value (9223372036854775808) out of range of long (-9223372036854775808 - 9223372036854775807) at [Source: (byte[])"{"Timestamp":"2024-06-02T08:13:17.198417663Z","Computer":"XXXXX","Channel":"Microsoft-Windows-Windows Defender/Operational","EID":1117,"Level":"high","Title":"Windows Defender Threat Detected","RecordID":26241,"Details":"Microsoft Defender Antivirus has taken action to protect this machine from malware or other potentially unwanted software.\n For more information please see the following:\nhttps://go.microsoft.com/fwlink/?linkid=37020\u0026name=Exploit:Python/CVE-2022-47966.A!dha\u0026thre"[truncated 4340 bytes]; line: 1, column: 1620]"

[scudette]

Timestamps are now always strings in ISO format with a timezone. You might need to change your elastic schema or drop the index to have it recreate the schema.

[D-V-W]

@scudette
we dropped the index and we still have the same error, it looks to be a error based on the event-system-keywords as it is out of range,
the max range it allows in elastic is 9223372036854775807 our results have higher number

[scudette]

Yeah it looks like elastic does not like the keywords field - Elastic actually needs a proper schema (called a mapping) with proper types. It looks like if you rely on the automatic schema creation it makes the field "long" but it needs to be "long long" to handle this data type.

So the issue seems unrelated to the timestamp change. I would recommend to try to drop the index again - elastic will guess the mapping depending on the first object to be uploaded so maybe this time the first keyword looked like a long. If you drop it again then maybe the next one will look like a long long.

Generally elastic uploads should be tailored to the artifact they are uploading and have the data normalized to the schema. See for example https://docs.velociraptor.app/artifact_references/pages/elastic.eventlogs.sysmon/

You can fix the mapping currently in the index using the update remapping api
https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-put-mapping.html

First dump out the mapping then update keyword to long long instead of long