Multi Line Alerts

[atilaloise]

For cases where the alert query returns more than one line, put an option to render all lines on a single card, or send several cards containing each of the resulting lines.

[guilhemmarchand]

@atilaloise

This relies on the configuration of the alert in Splunk, not on the alert action level:

If you use "For each result" rather than the default behavior "Once", every result in the table (every line) will be become one message card in the channel automatically.

On the other hand, when the alert is configured to trigger "Once" only, Splunk will limit the scope of all alert actions to one result.

[atilaloise]

There are many cases where alerts are sent to other media, so the scope of the alert via teams generally does not fit.

For example, the same alert goes to e-mail, rendering all lines in a table in the body of the e-mail, while via teams, each line can generate a discussion via chat.

So it becomes very interesting that the action behaves differently within its scope.

In my tests, if I don't use the "for each result" option, only one card with the first line is generated, ignoring the others.

[guilhemmarchand]

@atilaloise

I get your point, besides the notable event creation action and email I do not have an example of an alert action that can access all events and iterate through these as long as the alert is not configured to trigger "For each result".

The TA is originally built via the Splunk Addon Builter, it uses the builtin helper function:

        events = helper.get_events()
        for event in events:
            helper.log_debug("event={}".format(event))

Which will get all the results when the alert action trigger for each result is enabled, but the scope of the results gets limited to the first result when the default "Once" option is selected.

I would like to get the answer too, this is a question I have tried to clear out by the past, so I will be trying again, I do not know if this is a limitation of the AoB function or if it behaves the way it should and notable / email are unique cases for instance

https://docs.splunk.com/Documentation/AddonBuilder/3.0.2/UserGuide/PythonHelperFunctions

[Hollerweger]

I suggest here to configure the alert twice. For e-mail with Trigger "Once". And for MS Teams with Trigger "For each result".

[Hollerweger]

Tested out the "Once" option now with this Plugin and i see that the Card only shows first result and not a table like expected.

[guilhemmarchand]

@Hollerweger

The Addon expects a ttigger "For each result" if each line of the alert resulting table would lead to a message card in the MS Team channel, if set to trigger Once, only the first result would generate a message in the channel.

To my knowledge, this behaviour is the expected behaviour of AoB (Splunk Addon Builder) which the TA is built on.

[Hollerweger]

Okay so probably the default E-mail notification uses a different API for that.
Could this maybe be solved when I aggregate the results into one result where each field has the values comma separated?
Eg.:
time="14:40:53","14:40:54","14:40:55"
level="ERROR","WARN","ERROR"
message="Error while, ...", "WARN while, ..","Error while, ..."
Or maybe there is also some sort of array type in Splunk that would be the better approach.

[Hollerweger]

Could it be that you already post and return on the first event without iterating through all of them?

TA-ms-teams-alert-action/TA-ms-teams-alert-action/bin/ta_ms_teams_alert_action/modalert_ms_teams_publish_to_channel_helper.py

Line 231 in a872ebf

for event in events:

TA-ms-teams-alert-action/TA-ms-teams-alert-action/bin/ta_ms_teams_alert_action/modalert_ms_teams_publish_to_channel_helper.py

Line 480 in a872ebf

return response.text

[guilhemmarchand]

@Hollerweger Would you mind developing a bit further?

I tried earlier to make sure I understood properly the helper function from AoB, should you take this part of the code in consideration:

        events = helper.get_events()
        for event in events:
            helper.log_debug("event={}".format(event))

The first line "events = helper.get_events()" retrieves the "entire" data set resulting from the search, however if the alert is set to trigger "Once" and if you add:

        events = helper.get_events()
        helper.log_debug("events={}".format(events))

Then you would observe from the debug log that the Python backend only gets access to the first table result.
However if you define to trigger for each result, then entire table result will be exposed.

The code actually iterates over each of the results by:

for event in events:

Unfortunately the limitation happens before that via the AoB helper function, I tried to get some more information with no luck on this.

If you take the example of Enterprise Security and notable events, a common design practice is to have a scheduled alert lookig at all the notable events, set to trigger for each result, and running your special alert action such as the MS Teams, rather than modifiying each correlation search to trigger the alert action and forcing it to run for each result.

However I agree the email action does not have this limitation by design, which is why I believe the limitation comes from the AoB framework itself.

Let me know if this makes sense, I didn't find a way to bypass this btu I would be more than happy to get one ;-)

[Hollerweger]

@guilhemmarchand I'll try to have a look. First need to understand how plugin development for Splunk works.
Is there any existing testcase available that triggers an alert and mocks the search query? Or do I need to have a local Splunk instance running?

[abochmann]

I just ran into a similar problem, and have resorted to use "Trigger: Once" and combine all my results into a single multiline Splunk event, using some of the hints from #18 :

[...some Splunk search that produces a list of "mymessage" results...]
| sort -_time
| eval timestamp=strftime(_time,"%Y-%m-%d %H:%M:%S") 
| eval Message=timestamp." | ".host." | ".mymessage."<br />"
| stats list(Message) AS Messages

...and then use only "Messages" in the fields list of the Teams action.

It's not exactly pretty in Teams (some way to use basic Markdown in the results would be great), but does the job for now.

[Monawwer]

Hi @guilhemmarchand

Do we have any update on this multi line alerts fix.

Thanks.