Skip to content

Commit

Permalink
feat: Support app-parser for vps
Browse files Browse the repository at this point in the history
Current vendor product by source config uses two files that must be kept in sync this solution allows using one or more files (customer choice) to define mapping.
  • Loading branch information
rfaircloth-splunk committed Feb 6, 2022
1 parent 2b20414 commit e792c77
Show file tree
Hide file tree
Showing 49 changed files with 484 additions and 209 deletions.
125 changes: 63 additions & 62 deletions docs/sources/Cisco/index.md

Large diffs are not rendered by default.

11 changes: 9 additions & 2 deletions docs/sources/index.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,15 @@
# Introduction
When using Splunk Connect for Syslog to onboard a data source, the SC4S filter (or "log path") performs the operations that are traditionally performed at index-time by the corresponding Technical Add-on installed there. These index-time operations include linebreaking, sourcetype setting and timestamping. For this reason, if a data source is exclusively onboarded using SC4S then you will not need to install its corresponding Add-On on the indexers. You must, however, install the Add-on on the search head(s) for the user communities interested in this data source.
When using Splunk Connect for Syslog to onboard a data source, the syslog-ng "app-parser" performs the operations that are traditionally performed at index-time by the corresponding Technical Add-on installed there. These index-time operations include linebreaking, source/sourcetype setting and timestamping. For this reason, if a data source is exclusively onboarded using SC4S then you will not need to install its corresponding Add-On on the indexers. You must, however, install the Add-on on the search head(s) for the user communities interested in this data source.

SC4S "unique" filters are based either on the port upon which events arrive or the hostname/CIDR block from which they are sent. The "soup" filters run for events that arrive on port 514 (default for syslog), and contain regex and other syslog-specific parsers to identify events from a specific source, apply the correct sourcetype, and set other metadata. Data sources which generate events that are not unique enough to accurately identify with soup filters _must_ employ the "unique" filters (port/hostname/CIDR block) instead -- the soup filters are unavailable for these sources.
SC4S is designed to process "syslog" refering to IETF RFC standards 5424, legacy BSD syslog, RFC3164 (Not a standard document), and may "almost" syslog formats.

When possible data sources are identified and processed based on characteristics of the event that make them unique as compared to other events for example. Cisco devices using IOS will include " : %" followed by a string. While Arista EOS devices will use a valid RFC3164 header with a value in the "PROGRAM" position with "%" as the first char in the "MESSAGE" portion. This allows two similar event structures to be processed correct.

When identification by message content alone is not possible for example the "sshd" program field is commonly used across vendors additional "hint" or guidance configuration allows SC4S to better classify events. The hints can be applied by
definition of a specific port which will be used as a property of the event or by configuration of a host name/ip pattern. For example "VMWARE VSPHERE" products have a number of "PROGRAM" fields which can be used to identify vmware specific events in the syslog stream and these can be properly sourcetyped automatically however because "sshd" is not uniuqe it will be treated as generic "os:nix" events until further configuration is applied. The administrator can take one of two actions to refine the processing for vmware

* Define a specific port for vmware and reconfigure sources to use the defined port "SC4S_LISTEN_VMWARE_VSPHERE_TCP=9000". Any events arriving on port 9000 will now have a metadata field attached ".netsource.sc4s_vendor_product=VMWARE_VSPHERE"
* Define a "app-parser" to apply the metadata field by using a syslog-ng filter to apply the metadata field.

## Supporting previously unknown sources.

Expand Down
2 changes: 1 addition & 1 deletion mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ nav:
- Development: "developing/index.md"
- Destinations: "destinations.md"
- Sources:
- About: sources/index.md
- Read First: sources/index.md
- Alcatel: sources/Alcatel/index.md
- Alsid: sources/Alsid/index.md
- Arista: sources/Arista/index.md
Expand Down
1 change: 1 addition & 0 deletions package/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@ RUN poetry export --format requirements.txt | pip3 install --user -r /dev/stdin
COPY package/etc/syslog-ng.conf /etc/syslog-ng/syslog-ng.conf
COPY package/etc/conf.d /etc/syslog-ng/conf.d
COPY package/etc/context_templates /etc/syslog-ng/context_templates
COPY package/etc/test_parsers /etc/syslog-ng/test_parsers
COPY package/etc/local_config /etc/syslog-ng/local_config
COPY package/etc/local_config /etc/syslog-ng/local_config
COPY package/sbin/entrypoint.sh /
Expand Down

This file was deleted.

10 changes: 10 additions & 0 deletions package/etc/conf.d/conflib/_splunk/splunkfields.conf
Original file line number Diff line number Diff line change
Expand Up @@ -78,3 +78,13 @@ block rewrite r_set_splunk_dest_update(
};
};


block rewrite r_set_splunk_vps(
#While the following is not used it remains to prevent breaking changes in content
vendor("${.netsource.sc4s_vendor}")
product("${.netsource.sc4s_product}")
) {
set("`vendor`", value(".netsource.sc4s_vendor") condition('`vendor`' ne ""));
set("`product`", value(".netsource.sc4s_product") condition('`product`' ne ""));
set("`vendor`_`product`", value(".netsource.sc4s_vendor_product"));
};
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ block parser app-netsource-aruba_clearpass() {
application app-netsource-aruba_clearpass[sc4s-network-source] {
filter {
program('CPPM_' type(string) flags(prefix))
or ( "${.netsource.sc4s_vendor}" eq "aruba" and "${.netsource.sc4s_product}" eq "clearpass")
or "${.netsource.sc4s_vendor_product}" eq "aruba_clearpass"
or "${SOURCE}" eq "s_ARUBA_CLEARPASS"
;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,8 @@ block parser app-netsource-brocade_syslog() {
};
application app-netsource-brocade_syslog[sc4s-network-source] {
filter {
"${.netsource.sc4s_vendor_product}" eq "brocade_syslog"
( "${.netsource.sc4s_vendor}" eq "brocade" and "${.netsource.sc4s_product}" eq "syslog")
or "${.netsource.sc4s_vendor_product}" eq "brocade_syslog"
or "${SOURCE}" eq "s_BROCADE"
};
parser { app-netsource-brocade_syslog(); };
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,8 @@ block parser app-netsource-buffalo_terastation() {
};
application app-netsource-buffalo_terastation[sc4s-network-source] {
filter {
"${.netsource.sc4s_vendor_product}" eq "buffalo_terastation"
( "${.netsource.sc4s_vendor}" eq "buffalo" and "${.netsource.sc4s_product}" eq "terastation")
or "${.netsource.sc4s_vendor_product}" eq "buffalo_terastation"
or "${SOURCE}" eq "s_BUFFALO_TERASTATION"
;
};
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,8 @@ block parser app-netsource-checkpoint_fw() {
};
application app-netsource-checkpoint_fw[sc4s-network-source] {
filter {
"${.netsource.sc4s_vendor_product}" eq "checkpoint_fw"
( "${.netsource.sc4s_vendor}" eq "checkpoint" and "${.netsource.sc4s_product}" eq "fw")
or "${.netsource.sc4s_vendor_product}" eq "checkpoint_fw"
or "${SOURCE}" eq "s_CHECKPOINT_FW"
;
};
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -149,7 +149,8 @@ application app-netsource-cisco_esa[sc4s-network-source] {
filter {
not "${fields.sc4s_vendor_product}" eq "cisco_esa"
and (
"${.netsource.sc4s_vendor_product}" eq "cisco_esa"
( "${.netsource.sc4s_vendor}" eq "cisco" and "${.netsource.sc4s_product}" eq "esa")
or "${.netsource.sc4s_vendor_product}" eq "cisco_esa"
or "${SOURCE}" eq "s_CISCO_ESA"
)
};
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,8 @@ block parser app-netsource-cisco_meraki() {
};
application app-netsource-cisco_meraki[sc4s-network-source] {
filter {
"${.netsource.sc4s_vendor_product}" eq "cisco_meraki"
( "${.netsource.sc4s_vendor}" eq "cisco" and "${.netsource.sc4s_product}" eq "meraki")
or "${.netsource.sc4s_vendor_product}" eq "cisco_meraki"
or "${SOURCE}" eq "s_CISCO_MERAKI"
};
parser { app-netsource-cisco_meraki(); };
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,8 @@ block parser app-netsource-cisco_wsa() {
};
application app-netsource-cisco_wsa[sc4s-network-source] {
filter {
"${.netsource.sc4s_vendor_product}" eq "cisco_wsa"
( "${.netsource.sc4s_vendor}" eq "cisco" and "${.netsource.sc4s_product}" eq "wsa")
or "${.netsource.sc4s_vendor_product}" eq "cisco_wsa"
or (
"${SOURCE}" eq "s_CISCO_WSA"
and "${.netsource.sc4s_vendor_product}" eq ""
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -50,10 +50,11 @@ block parser app-netsource-cisco_wsa_11_7() {
};
application app-netsource-cisco_wsa_11_7[sc4s-network-source] {
filter {
"${.netsource.sc4s_vendor_product}" eq "cisco_wsa11_7"
"${.netsource.sc4s_vendor_product}" eq "cisco_wsa11-7"
or (
"${SOURCE}" eq "s_CISCO_WSA"
and "${.netsource.sc4s_vendor_product}" eq "cisco_wsa11_7"
( "${.netsource.sc4s_vendor}" eq "cisco" and "${.netsource.sc4s_product}" eq "wsa11-7")
or "${SOURCE}" eq "s_CISCO_WSA"
and "${.netsource.sc4s_vendor_product}" eq "cisco_wsa11-7"
)
};
parser { app-netsource-cisco_wsa_11_7(); };
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,7 @@ block parser app-netsource-cisco_wsa_splunk() {
application app-netsource-cisco_wsa_splunk[sc4s-network-source] {
filter {
"${.netsource.sc4s_vendor_product}" eq "cisco_wsa_recommended"
or ( "${.netsource.sc4s_vendor}" eq "cisco" and "${.netsource.sc4s_product}" eq "wsa_recommended")
or (
"${SOURCE}" eq "s_CISCO_WSA"
and "${.netsource.sc4s_vendor_product}" eq "cisco_wsa_recommended"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,8 @@ block parser app-netsource-dell_poweredge_cmc() {
};
application app-netsource-dell_poweredge_cmc[sc4s-network-source] {
filter {
("${.netsource.sc4s_vendor_product}" eq "dell_poweredge_cmc"
( "${.netsource.sc4s_vendor}" eq "dell" and "${.netsource.sc4s_product}" eq "poweredge_cmc")
or ("${.netsource.sc4s_vendor_product}" eq "dell_poweredge_cmc"
or "${SOURCE}" eq "s_DELL_POWEREDGE_CMC")
and "${fields.sc4s_vendor_product}" eq ""
};
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,8 @@ block parser app-netsource-dell_rsa_secureid() {
};
application app-netsource-dell_rsa_secureid[sc4s-network-source] {
filter {
"${.netsource.sc4s_vendor_product}" eq "dell_rsa_secureid"
( "${.netsource.sc4s_vendor}" eq "dell" and "${.netsource.sc4s_product}" eq "rsa_secureid")
or "${.netsource.sc4s_vendor_product}" eq "dell_rsa_secureid"
or "${SOURCE}" eq "s_DELL_RSA_SECUREID"

};
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,8 @@ block parser app-netsource-ibm_datapower() {
application app-netsource-ibm_datapower[sc4s-network-source] {
filter {
(
"${.netsource.sc4s_vendor_product}" eq "ibm_datapower"
( "${.netsource.sc4s_vendor}" eq "ibm" and "${.netsource.sc4s_product}" eq "datapower")
or "${.netsource.sc4s_vendor_product}" eq "ibm_datapower"
or "${SOURCE}" eq "s_IBM_DATAPOWER"
)
};
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,8 @@ block parser app-netsource-infoblox() {
application app-netsource-infoblox[sc4s-network-source] {
filter {
(
"${.netsource.sc4s_vendor_product}" eq "infoblox"
( "${.netsource.sc4s_vendor}" eq "infoblox" and "${.netsource.sc4s_product}" eq "nios")
or "${.netsource.sc4s_vendor_product}" eq "infoblox"
or "${SOURCE}" eq "s_INFOBLOX"
)
and not message('CEF:0', type('string') flags(prefix))
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,8 @@ block parser app-netsource-proofpoint_pps() {
application app-netsource-proofpoint_pps[sc4s-network-source] {
filter {
(
"${.netsource.sc4s_vendor_product}" eq "proofpoint_pps"
( "${.netsource.sc4s_vendor}" eq "proofpoint" and "${.netsource.sc4s_product}" eq "pps")
or "${.netsource.sc4s_vendor_product}" eq "proofpoint_pps"
or "${.netsource.sc4s_vendor_product}" eq "proofpoint_pps_filter"
or "${.netsource.sc4s_vendor_product}" eq "proofpoint_pps_sendmail"
)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ block parser app-netsource-ubiquiti_unifi() {
application app-netsource-ubiquiti_unifi[sc4s-network-source] {
filter {
"${.netsource.sc4s_vendor_product}" eq "ubiquiti_unifi_fw"
or ( "${.netsource.sc4s_vendor}" eq "ubiquiti" and "${.netsource.sc4s_product}" eq "unifi")
or (
"${SOURCE}" eq "s_UBIQUITI_UNIFI"
and "${fields.sc4s_vendor_product}" eq ""
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ block parser app-netsource-dell_switch_n() {

};
};
application app-netsource-dell_switch_n[sc4s-network-source] {
application app-netsource-dell_switch_n[sc4s-syslog] {
parser { app-netsource-dell_switch_n(); };
};

27 changes: 27 additions & 0 deletions package/etc/conf.d/enrich/vendor_product_by_source.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
parser vendor_product_by_source {
channel {
if {
parser {
app-parser(topic(sc4s-vps));
};
} elif {
parser {
add-contextual-data(
selector(filters("`syslog-ng-sysconfdir`/conf.d/local/context/vendor_product_by_source.conf")),
database("`syslog-ng-sysconfdir`/conf.d/local/context/vendor_product_by_source.csv")
ignore-case(yes)
prefix(".netsource.")
);
};
if {
filter {
"${.netsource.sc4s_vendor_product}" eq ""
};
rewrite{
set('${.netsource.sc4s_vendor}_${.netsource.sc4s_product}' value('.netsource.sc4s_vendor_product}'));
};
};
};
};
};

Original file line number Diff line number Diff line change
@@ -1,99 +1,4 @@
filter f_test_test {
host("testvp-*" type(glob))
#or netmask(xxx.xxx.xxx.xxx/xx)
};
filter f_aruba_clearpass {
host("aruba-cp-*" type(glob))
#or netmask(xxx.xxx.xxx.xxx/xx)
};

filter f_brocade_syslog {
host("test_brocade-*" type(glob))
#or netmask(xxx.xxx.xxx.xxx/xx)
};

filter f_citrix_netscaler {
host("test_ctitrixns-*" type(glob))
#or netmask(xxx.xxx.xxx.xxx/xx)
};
filter f_dell_rsa_secureid {
host("test_rsasecureid*" type(glob))
#or netmask(xxx.xxx.xxx.xxx/xx)
};
filter f_juniper_netscreen {
host("jnpns-*" type(glob))
#or netmask(xxx.xxx.xxx.xxx/xx)
};
filter f_cisco_meraki {
host("testcm-*" type(glob))
#or netmask(xxx.xxx.xxx.xxx/xx)
};
filter f_cisco_wsa{
host("cisco-wsa-*" type(glob))
};
filter f_cisco_wsa_w3crecommended{
host("cisco-wsaw3c-*" type(glob))
};
filter f_cisco_wsa11_7{
host("cisco-wsa11-7" type(string) flags(prefix))
or host("cisco_wsa11_7" type(string) flags(prefix))
};
filter f_f5_bigip {
host("test-f5-*" type(glob))
#or netmask(xxx.xxx.xxx.xxx/xx)
};
filter f_infoblox {
host("infoblox-*" type(glob))
#or netmask(xxx.xxx.xxx.xxx/xx)
};
filter f_pfsense {
host("pfsense-*" type(glob))
#or netmask(xxx.xxx.xxx.xxx/xx)
};
filter f_proofpoint_pps {
host("pps-*" type(glob))
#or netmask(xxx.xxx.xxx.xxx/xx)
};
filter f_schneider_apc {
host("test_apc-*" type(glob))
#or netmask(xxx.xxx.xxx.xxx/xx)
};
filter f_ubiquiti_unifi_fw {
host("usg-*" type(glob))
#or netmask(xxx.xxx.xxx.xxx/xx)
};
filter f_tzfixny {
host("tzfny-*" type(glob))
#or netmask(xxx.xxx.xxx.xxx/xx)
};
filter f_cisco_esa {
host("cisco_esa" type(glob))
#or netmask(xxx.xxx.xxx.xxx/xx)
};
filter f_spectracom_ntp {
#Source uses invalid format can not use host must use IP or port
netmask(169.254.100.1/24)
};
filter f_sophos_webappliance {
host("test-sophos-webapp-" type(string) flags(prefix))
};

filter f_vmware_esx {
host("testvmwe-" type(string) flags(prefix))
};

filter f_dell_cmc {
host("test-dell-cmc-" type(string) flags(prefix))
};

filter f_ibm_datapower {
host("test-ibmdp-" type(string) flags(prefix))
};

filter f_mikrotik_routeros {
host("test-mrtros-" type(string) flags(prefix))
};

filter f_symantec_dlp {
host("test-dlp-" type(string) flags(prefix))
};
23 changes: 0 additions & 23 deletions package/etc/context_templates/vendor_product_by_source.csv.example
Original file line number Diff line number Diff line change
@@ -1,24 +1 @@
f_aruba_clearpass,sc4s_vendor_product,"aruba_clearpass"
f_test_test,sc4s_vendor_product,"test_test"
f_brocade_syslog,sc4s_vendor_product,"brocade_syslog"
f_cisco_meraki,sc4s_vendor_product,"cisco_meraki"
f_cisco_wsa,sc4s_vendor_product,"cisco_wsa"
f_cisco_wsa11_7,sc4s_vendor_product,"cisco_wsa11_7"
f_cisco_wsa_w3crecommended,sc4s_vendor_product,"cisco_wsa_recommended"
f_dell_rsa_secureid,sc4s_vendor_product,"dell_rsa_secureid"
f_f5_bigip,sc4s_vendor_product,"f5_bigip"
f_infoblox,sc4s_vendor_product,"infoblox"
f_pfsense,sc4s_vendor_product,"pfsense"
f_proofpoint_pps,sc4s_vendor_product,"proofpoint_pps"
f_schneider_apc,sc4s_vendor_product,"schneider_apc"
f_spectracom_ntp,sc4s_vendor_productm",spectracom_ntp"
f_ubiquiti_unifi_fw,sc4s_vendor_product,"ubiquiti_unifi_fw"
f_tzfixny,sc4s_time_zone,"America/New_York"
f_cisco_esa,sc4s_vendor_product,"cisco_esa"
f_sophos_webappliance,sc4s_vendor_product,"sophos_webappliance"
f_vmware_esx,sc4s_vendor,"vmware"
f_vmware_esx,sc4s_product,"vcenter"
f_dell_cmc,sc4s_vendor_product,"dell_poweredge_cmc"
f_ibm_datapower,sc4s_vendor_product,"ibm_datapower"
f_mikrotik_routeros,sc4s_vendor_product,"mikrotik_routeros"
f_symantec_dlp,sc4s_vendor_product,"symantec_dlp"
7 changes: 5 additions & 2 deletions package/etc/syslog-ng.conf
Original file line number Diff line number Diff line change
Expand Up @@ -28,12 +28,15 @@ global_options();
@include "conf.d/templates/*.conf"
@include "conf.d/conflib/*.conf"
@include "conf.d/conflib/*/*.conf"
@include "conf.d/local/config/app_parsers/*.conf"
@include "conf.d/local/config/app_parsers/*/*.conf"

@include "conf.d/filters/*/*.conf"
@include "conf.d/local/config/filters/*.conf"

@include "conf.d/local/config/app_parsers/*.conf"
@include "conf.d/local/config/app_parsers/*/*.conf"
@include "conf.d/enrich/*.conf"
@include "conf.d/enrich/*/*.conf"


@include "conf.d/sources/*.conf"
@include "conf.d/sources/*/*.conf"
Expand Down
Loading

0 comments on commit e792c77

Please sign in to comment.