A custom ReportingCommand OrderedDict does not honor column order

[michaelsalisbury]

Describe the bug
In a minimal custom reporting command one can override the reduce definition. The "report" object, a generator of OrderedDict objects, which are the rows of data passed into the custom command, are consumed and one, all or more rows are yielded back to Splunk. I have found that when a custom reporting command follows a stats command the order of the OrderedDict that contains each row of data does not honor the order of the columns prior to the custom reporting command. Note, the commands.conf parameter, changes_colorder = false, will force the column order post the custom reporting command to match what it was prior but this parameter has no effect on the order of the OrderedDict.

To Reproduce
Steps to reproduce the behavior:

1. Create a dead simple reporting command.
   def reduce(self, report):
for row in report:
row['cols'] = list(row.keys())
yield row
continue

2. Setup command.conf as follows.
   [crc]
python.version = python3
filename = crc.py
   chunked = false local = true passauth = true maxinputs = 0 run_in_preview = false enableheader = true outputheader = true requires_srinfo = true supports_getinfo = true supports_rawargs = true supports_multivalues = true requires_preop = false changes_colorder = false`

3. Use this splunk query:
   index=_internal uri=*
| head 10
| eval random = random()
| fields source index sourcetype host uri random
| stats values(*) AS * by random
| table source index sourcetype host uri random
| foreach source index sourcetype host uri random [ eval audit = mvappend(audit,"<<FIELD>>") ]
| crc

4. The above query will return a column audit that has the order of columns before the custom reporting command. The custom reporting command will generate a column, cols, that has the order of the columns as passed into the python script. Note: commenting out the stats command and you'll find that the order of the multivalue field cols will then match audit.

Expected behavior
I expect that OrderedDict passed into the custom python to honor the order of the columns set by a preceding table command or any aggregation command like stats or chart for example. The order of the columns as a user sees them is very useful in many custom operations. Requiring a custom reporting command to have an input parameter that details the order is cumbersome and limiting.

Logs or Screenshots
none

Splunk (please complete the following information):

• Version: 9.0.1
• OS: centos 7
• Deployment: single-instance and search cluster

SDK (please complete the following information):

• Version: 7.7.2
• Language Runtime Version: Python 3.7.11
• OS: centos 7

Additional context
none

[ashah-splunk]

Hi @michaelsalisbury, as per our analysis we found that SDK is not changing the order of columns received from the Splunk. In the above mentioned use case, transforming command(i.e table) is being used. Transforming commands are recommended to be kept at the end of the search command, therefore in the above example it should be kept after the Custom Search Command(i.e crc).

Please refer Transforming Command in the docs.

Note: As you will see in the following examples, you always place your transforming commands after your search commands, linking them with a pipe operator ( | ).

[michaelsalisbury]

Note: Not sure if this is clear. In my tests the custom search command is NOT changing the order of the columns as seen on-screen. The issue at hand is the order of the columns in the orderedDict passed into the custom search command class.

Question1: are you saying that if the SDK if used in the exact way I have described with the exact query I have described it IS passing an orderedDict into the custom search command in the exact column order as seen in the UI? If so can you detail something about your test environment or send an app that I can use to replicate sa my app consistently proves otherwise.

Question2: custom search commands are often used at the end of a query to perform some non-native data manipulation or export that can only be done in the final stage after all data massaging has been done (and no an alert action is not always appropriate). I have confirmed the described behavior using both custom reporting commands and custom eventing commands. Is there a specific type of custom search command class I should be using at the end of a query so as to receive in my custom search command an orderedDict that reflects the column order seen on-screen? This must assume that any native splunk command was used prior including stats, table, etc.