docs: add section about how to debugging the _internal index (#1437)

**Issue number: [ADDON-75684](https://splunk.atlassian.net/browse/ADDON-75684)** ## Summary Documentation has been expanded with a section on debugging _internal index. ### Changes Added "Something went wrong" section to the "troubleshooting.md" documentation. ### User experience The user will be able to more easily find the rootcause of the error in the add-on ## Checklist If your change doesn't seem to apply, please leave them unchecked. * [x] I have performed a self-review of this change * [ ] Changes have been tested * [x] Changes are documented * [x] PR title follows [conventional commit semantics](https://www.conventionalcommits.org/en/v1.0.0/)
splunk · Nov 12, 2024 · 26a1b21 · 26a1b21
1 parent 4ec696a
commit 26a1b21
Show file tree

Hide file tree

Showing 3 changed files with 38 additions and 0 deletions.
diff --git a/docs/images/troubleshooting_error_example.png b/docs/images/troubleshooting_error_example.png
diff --git a/docs/images/troubleshooting_input_error.png b/docs/images/troubleshooting_input_error.png
diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md
@@ -52,3 +52,41 @@ To run it, copy the script to the add-on folder, and then read the
 instructions in the script itself. If it does not work for your particular case,
 feel free to adjust it yourself, or file a feature request for us to improve
 something.
+
+## Something went wrong
+
+![img.png](images/troubleshooting_input_error.png)
+
+During the creation of the add-on, an error, such as incorrect import, unhandled exception etc., can be made in the code of modinput scripts.
+When the add-on is built, this error is not caught and a package can be installed in Splunk without any problems. However, when you try to enter
+the add-on page, you will see the message presented above. As this message is not
+clear, it is not helpful in finding the root cause of the problem.
+
+Fortunately, most errors are logged in Splunk on the internal indexes. One of these indexes is `_internal`.
+
+First, search the `_internal` index for the word `ERROR` or `stderr`.
+Usually the source of these logs is the `splunkd` process, so it can be used to pre-filter data.
+
+`index = _internal source=*splunkd* ERROR` or `index = _internal source=*splunkd* stderr`
+
+In the case of small infrastructures, this simple query can return information you need.
+
+When the instances are large and they have many applications, each of them can have errors.
+In addition, some errors are recorded only once when the add-on is installed, so it can be difficult to determine the correct time range.
+
+In this case, you can narrow your search to a specific add-on. This may not be easy because no unique parameter is logged
+that would easily filter out data for a specific add-on. However, you can use the `scheme` parameter, which is directly correlated with the names of inputs in the add-on.
+
+The example of a logged error:
+![img.png](images/troubleshooting_error_example.png)
+
+Schema logs the input name with a colon at the end, so use an asterisk after entering the name:
+
+`index = _internal source=*splunkd* (component=ModularInputs stderr) OR component=ExecProcessor (scheme IN (example_input_one*, example_input_two*, example_input_three_abc*))`
+
+Or you can use a shortened version, using the wildcard mechanic:
+
+`index = _internal source=*splunkd* (component=ModularInputs stderr) OR component=ExecProcessor (scheme IN (*example_input*))`
+
+Since in this example we are talking about errors coming from the modular input scripts,
+another filtering factor, the `ModularInputs` component, is added. The `ExecProcessor` component is also included. It is responsible for running and managing scripts.