chore: merge develop into main (#1456)

release 5.53.0
merge develop into main

.github/PULL_REQUEST_TEMPLATE.md

@@ -1,20 +1,29 @@
 **Issue number:**
 
+### PR Type
+
+**What kind of change does this PR introduce?**
+* [ ] Feature
+* [ ] Bug Fix
+* [ ] Refactoring (no functional or API changes)
+* [ ] Documentation Update
+* [ ] Maintenance (dependency updates, CI, etc.)
+
 ## Summary
 
 ### Changes
 
-> Please provide a summary of what's being changed
+Please provide a summary of the changes.
 
 ### User experience
 
-> Please describe what the user experience looks like before and after this change
+Please describe the user experience before and after this change. Screenshots are welcome for additional context.
 
 ## Checklist
 
-If your change doesn't seem to apply, please leave them unchecked.
+If an item doesn't apply to your changes, leave it unchecked.
 
-* [ ] I have performed a self-review of this change
-* [ ] Changes have been tested
+* [ ] I have performed a self-review of this change according to the [development guidelines](https://splunk.github.io/addonfactory-ucc-generator/contributing/#development-guidelines)
+* [ ] Tests have been added/modified to cover the changes [(testing doc)](https://splunk.github.io/addonfactory-ucc-generator/contributing/#build-and-test)
 * [ ] Changes are documented
-* [ ] PR title follows [conventional commit semantics](https://www.conventionalcommits.org/en/v1.0.0/)
+* [ ] PR title and description follows the [contributing principles](https://splunk.github.io/addonfactory-ucc-generator/contributing/#pull-requests)

.github/workflows/automatic_doc_generation.yml

@@ -44,4 +44,4 @@ jobs:
         if: ${{ env.isPR }}
         run: |
           git add \*.md
-          git diff --staged --exit-code || (git commit -S -m "doc: updated docs regarding generated conf, xml and html files" && git push)
+          git diff --staged --exit-code || (git commit -S -m "docs: updated docs regarding generated conf, xml and html files" && git push)

.github/workflows/build-test-release.yml

@@ -135,6 +135,7 @@ jobs:
           - "3.9"
           - "3.10"
           - "3.11"
+          - "3.12"
     steps:
       - uses: actions/checkout@v4
       - uses: actions/setup-python@v5
@@ -168,6 +169,7 @@ jobs:
           - "3.9"
           - "3.10"
           - "3.11"
+          - "3.12"
     steps:
       - uses: actions/checkout@v4
       - uses: actions/download-artifact@v4

.gitignore

@@ -14,7 +14,7 @@ pip-wheel-metadata/UNKNOWN.dist-info/top_level.txt
 pip-wheel-metadata/UNKNOWN.dist-info/METADATA
 .vscode/settings.json
 .DS_Store
-.venv
+.venv*
 output
 # The following files should never be checked into git but can not be in the
 # ignore file due to poetry issues

.markdownlint.yaml

@@ -33,6 +33,10 @@ MD025: false
 # MD028/no-blanks-blockquote
 MD028: false
 
+# MD029/ordered-list-item-prefix
+MD029:
+  style: one
+
 # MD033/no-inline-html
 MD033: false
 
@@ -43,7 +47,7 @@ MD034: false
 MD040: false
 
 # MD041/first-line-heading/first-line-h1
-MD041: false
+MD041: true
 
 # MD046/code-block-style
 MD046: false

.markdownlintignore

@@ -0,0 +1 @@
+.github/PULL_REQUEST_TEMPLATE.md

docs/additional_packaging.md

@@ -2,9 +2,22 @@
 
 To extend the build process, you can create a `additional_packaging.py` file in the same file level where you have your globalConfig file.
 
-This file should have the `additional_packaging` function, which accepts add-on name as its only argument.
+This file should at least have:
+
+- the `cleanup_output_files` function, which accepts `output_path` (str), `add-on name` (str) as its arguments.
+- the `additional_packaging` function, which accepts `add-on name` (str) as its only argument.
+
+First the `cleanup_output_files` function would be called from `ucc-gen` build process and then `additional_packaging` function.
 
 See the following example for proper usage:
 
-* Build custom UI after `ucc-gen` finishes all its necessary steps.
-* Use a workaround for a `ucc-gen` feature that has not been implemented.
+- Build custom UI after `ucc-gen` finishes all its necessary steps.
+- Use a workaround for a `ucc-gen` feature that has not been implemented.
+
+## Example
+
+Below is an example of additional_packaging.py containing both the implementations of functions.
+
+```python
+--8<-- "tests/testdata/test_addons/package_global_config_everything_uccignore/additional_packaging.py"
+```

docs/advanced/custom_mapping.md

@@ -1,3 +1,5 @@
+# Custom Mapping
+
 We can use this feature to map each field with meaningful value to display in the table. For example, the category field contains 1, 2, and 4 values, but when those values are displayed, the user might get confused as those values do not signify the meaning of their mapping. To avoid this confusion, the user can map each field with meaningful value as shown in the following example:
 
 If you have fields that are not mandatory but you would like to display them inside table, you can use default value option by providing ```"[[default]]"``` as one of parameters (check example bellow). It is a way to provide some meaningful information for form fields that have not been filled (fill empty cells in table).

docs/advanced/custom_warning.md

@@ -1,3 +1,5 @@
+# Custom Warning
+
 This feature allows us to pass broarder description on Input and Configuration page displayed under main description.
 
 ### Warning Properties

docs/advanced/dependent_dropdown.md

@@ -1,3 +1,5 @@
+# Dependent Dropdown
+
 This feature allows dynamic loading options for the `singleSelect` and the `multipleSelect` fields when the options for that field depend on other fields' values. It loads options via an API call to the endpoint mentioned in `endpointUrl` under options when any dependencies field is updated and all required dependencies fields are non-null.
 
 All non-required dependencies fields can be of any type, but all required dependencies fields should only be of single-select type.

docs/advanced/groups_feature.md

@@ -1,3 +1,5 @@
+# Groups Feature
+
 Using this functionality, the Inputs page form can be divided into distinct sections, each comprising relevant fields. If the `isExpandable` property is set to true in the global configuration file, the group will be in the [collapsible panel](https://splunkui.splunk.com/Packages/react-ui/CollapsiblePanel) type.
 
 The groups will be displayed at the bottom of the form.

docs/advanced/oauth_support.md

@@ -1,3 +1,5 @@
+# OAuth Support
+
 UCC allows you to add Auth support in the configuration page. In UCC, OAuth2.0 of the Authorization Code Flow `grant` type is used. It only supports the standard parameters specified in [RFCP749](https://www.rfc-editor.org/rfc/rfc6749) for obtaining an authorization code.
 
 Auth can be used inside the entity tag. Use `type: "oauth"` in the entity list and specify the `options` next to the `type: "oauth"`.
@@ -42,8 +44,9 @@ Auth can be used inside the entity tag. Use `type: "oauth"` in the entity list a
             + `disableonEdit`: When the form is in edit mode, the field becomes unable to be edited. The default value is false.
             + `enable`: The enable property sets whether a field is enabled or not. The default value is true.
 
-> [!WARNING]
-> The [Placeholder](https://splunkui.splunkeng.com/Packages/react-ui/Text?section=develop) attribute is deprecated and renounced. Instead, we recommend to use the "help" attribute.
+!!! warning "Placeholder deprecation"
+
+    The [Placeholder](https://splunkui.splunkeng.com/Packages/react-ui/Text?section=develop) attribute is deprecated and renounced. Instead, we recommend to use the "help" attribute.
 
 ### Usage
 

docs/advanced/os-dependent_libraries.md

@@ -1,3 +1,5 @@
+# OS-dependent libraries
+
 This feature allows you to download and unpack libraries with appropriate binaries for the indicated operating system during the build process.
 To do this, you need to expand the **meta** section in the global configuration with the **os-dependentLibraries** field. This field takes the following attributes:
 

docs/advanced/save_validator.md

@@ -1,3 +1,5 @@
+# Save Validator
+
 This feature allows you to pass a Javascript function as a string to apply customized validation to form data.
 
 By using this approach, you can write custom JavaScript code where you can write your business logic, and validating can return error messages which will be displayed at the top of the form.

docs/advanced/sub_description.md

@@ -1,3 +1,5 @@
+# Sub Description
+
 This feature allows us to pass a broader description on the Input and Configuration pages displayed under main description.
 
 ### Sub Descritpion Properties

docs/alert_actions/adaptive_response.md

@@ -1,3 +1,5 @@
+# Adaptive Response
+
 The Adaptive Response framework provides a mechanism for running preconfigured actions
 within the Splunk platform or by integrating with external applications.
 These actions can be automatically triggered by correlation search results or manually

docs/alert_actions/alert_scripts.md

@@ -1,3 +1,5 @@
+# Alert Action Scripts
+
 The following files would be created/ updated in the output folder once you executed the `ucc-gen` command:
 
 | File Location | Content Description | Action |

docs/alert_actions/index.md

@@ -2,6 +2,8 @@
 title: Alert Actions
 ---
 
+# Alert Actions
+
 The alert action can help a user to take action on the alerts that have been triggered. The knowledge from Splunk can be sent to an outside service or to pull additional or detailed information related to the trigger details.
 An add-on can have multiple alert actions based on the use cases the add-on provides. You can know more about alert actions from [this documentation](https://docs.splunk.com/Documentation/Splunk/latest/Alert/Aboutalerts).
 

docs/commands.md

@@ -0,0 +1,117 @@
+# Commands
+
+These are the commands that are available in UCC framework.
+
+## `ucc-gen build`
+
+The `ucc-gen build` command builds the add-on. As of now, running `ucc-gen` does the same thing as running `ucc-gen build`,
+but eventually calling `ucc-gen` without specifying a subcommand will be
+deprecated.
+
+It takes the following parameters:
+
+* `--source` - [optional] folder containing the `app.manifest` and app
+    source. The default is `package`.
+* `--config` - [optional] path to the configuration file. It defaults to
+    the globalConfig file in the parent directory of the source provided.
+    Only *.json* and *.yaml* files are accepted.
+* `--ta-version` - [optional] override current version of TA. The default
+    version is version specified in `globalConfig.json` or `globalConfig.yaml`.
+    A Splunkbase compatible version of SEMVER is used by default.
+* `-o` / `--output` - [optional] output folder to store the build add-on.
+   By default, it is saved in the `current directory/output` folder.
+    Absolute paths are accepted as well.
+* `--python-binary-name` - [optional] Python binary name to use when
+    installing Python libraries. The default is `python3`.
+* `-v` / `--verbose` - [optional] shows detailed information about
+    created/copied/modified/conflict files after build is complete.
+    This option is in the experimental mode. The default is `False`.
+* `--pip-version` - [optional] pip version that is used to install python libraries. The default is `latest`.
+* `--pip-legacy-resolver` - [optional] Use old pip dependency resolver by adding flag '--use-deprecated=legacy-resolver'
+    to pip install command. The default is`False`.
+    >**NOTE:** This flag is deprecated and will be removed from pip in the future.
+    Instead of using this flag, the correct solution would be to fix the packages your project depends on to work properly with the new resolver. Additionally, this flag is not compatible with pip version `23.2`. Use `23.2.1` instead.
+* `--pip-custom-flag` - [optional] Additional flag(s) that will be added to the `pip install` command.
+    By default, all the following flags are added to the `pip install` command: `--no-compile`, `--prefer-binary` and `--ignore-installed`.
+    If `--pip-custom-flag` is specified these three arguments will be missing so if you still want them in your command add them to the `--pip-custom-flag` argument.
+
+    Example:  `--pip-custom-flag="--no-compile --prefer-binary --ignore-installed --report path/to/report.json --progress-bar on"`
+
+* `--ui-source-map` - [optional] if present generates front-end source maps (.js.map files), that helps with code debugging.
+
+### Verbose mode
+
+The verbose mode is available for `v5.35.0` and later.
+
+Running `ucc-gen build -v` or `ucc-gen build --verbose` prints additional information about
+what was exactly created / copied / modified / conflicted after the build is complete. It does
+not scan the `lib` folder due to the nature of the folder.
+
+See the following description of what each state means:
+
+* `created`: the file is not in the original package and was created during the build process.
+* `copied`: the file is in the original package and was copied during the build process.
+* `modified`: the file is in the original package and was modified during the build process.
+* `conflict`: the file is in the original package and was copied during the build process, but may be generated by UCC itself, so incorrect usage can stop the add-on from working.
+
+## `ucc-gen init`
+
+`ucc-gen init` initializes the add-on. This is available on `v5.19.0` and later.
+The `ucc-gen init` command initializes the add-on and bootstraps some code in the
+modular input which you, as a developer, can extend for your needs.
+
+Apart from standard files needed for the add-on, it also adds search head
+clustering files in the `default/server.conf` file and reload triggers in the
+`default/app.conf` file. Those files will be soon generated automatically by the
+`ucc-gen build` command itself.
+during the add-on development.
+
+It takes the following parameters:
+
+* `--addon-name` - [required] add-on name. See the
+    [official naming convention guide](https://dev.splunk.com/enterprise/docs/releaseapps/splunkbase/namingguidelines/).
+* `--addon-rest-root` - [optional] add-on REST root, defaults to `--addon-name` if not provided.
+* `--addon-display-name` - [required] add-on "official" name.
+* `--addon-input-name` - [required] name of the generated input.
+* `--addon-version` - [optional] version of the generated add-on, with `0.0.1` by default.
+* `--overwrite` - [optional] overwrites the already existing folder.
+    By default, you can't generate a new add-on to an already existing folder.
+
+## `ucc-gen import-from-aob`
+
+Import from AoB (Add-on Builder), from `v5.24.0` and later. It is in the
+**experimental** state as of now, meaning that running this command may not
+produce a 100% UCC compatible add-on, but we are going to work on future
+improvements for the script itself.
+
+> **Note:** the `import-from-aob` command does not currently support Windows.
+
+The import functionality is based on the
+[ucc_migration_test](https://github.com/tmartin14/ucc_migration_test) bash
+script.
+One of the ways you can use it is to download an AoB-based add-on from
+Splunkbase, unarchive the folder, and then use
+`ucc-gen import-from-aob --addon-name <unarchived-folder-name>`. Or you can
+run the same command against your locally developed add-on, but it should be
+exported from AoB.
+
+It accepts the following parameters:
+
+* `--addon-name` - [required] add-on name.
+
+## `ucc-gen package`
+
+`ucc-gen package` can be used for `v5.30.0` and later. It packages the add-on so it can be installed.
+It mimics the basics of the `slim package` command. This command can be used for most of the simple cases.
+
+It does not support:
+
+* the `.slimignore` file.
+* the [dependencies section](https://dev.splunk.com/enterprise/docs/releaseapps/packageapps/packagingtoolkit/#Dependencies-section).
+
+It accepts the following parameters:
+
+* `--path` - [required] path to the built add-on (should include the `app.manifest` file).
+* `-o` / `--output` - [optional] output folder to store the packaged add-on.
+    By default, it will be saved in the `current directory` folder.
+    It accepts absolute paths as well.

docs/configurations/index.md

@@ -1,3 +1,5 @@
+# Configuration
+
 The `Configuration` tab can have multiple subtabs, for example, a tab for
 account configuration (to configure the account by adding account credentials),
 proxy configuration, and logging level configuration.
@@ -18,6 +20,7 @@ proxy configuration, and logging level configuration.
 | name<span class="required-asterisk">\*</span>                         | string | To define the particular tab name.                                                                                                                                                                                                                                                  |
 | title<span class="required-asterisk">\*</span>                        | string | To show the title of the tab.                                                                                                                                                                                                                                                       |
 | [entity](../entity/index.md)<span class="required-asterisk">\*</span> | array  | A list of fields and their properties.                                                                                                                                                                                                                                              |
+| [groups](../advanced/groups_feature.md)                               | array  | It is used to divide forms into distinct sections, each comprising relevant fields.                                                                                                                                                                                                 |
 | [table](../table.md)                                                  | object | To display accounts stanza in table                                                                                                                                                                                                                                                 |
 | style                                                                 | string | By specifying this property in the global config file, the forms can either be opened as a new page or in a dialog. <br>Supported values are "page" or "dialog". <br> Default value is **dialog**.                                                                                  |
 | options                                                               | object | This property allows you to enable the [saveValidator](../advanced/save_validator.md) feature.                                                                                                                                                                                      |

docs/configurations/logging.md

@@ -1,3 +1,5 @@
+# Logging
+
 The logging tab is a predefined component that allows to create the page for changing
 the log level. It is added in the `pages.configuration.tabs` array
 

docs/configurations/proxy.md

@@ -1,3 +1,5 @@
+# Proxy
+
 There are fields that need to be specified in order to enable proxy.
 
 ### Fields