Doc improvements (crowdsecurity#644)

* fix crowdsecurity#640 : add faq about metrics and 'is my setup working'

* up documentation go routines, fix crowdsecurity#637

* to merge along with crowdsecurity/hub#160 to improve forensic's doc

Author: buixor

docs/faq.md

@@ -53,13 +53,20 @@ SQLite by default as it's suitable for standalone/single-machine setups.
 
 ## How to control granularity of actions ? (whitelists, simulation etc.)
 
-{{v1X.crowdsec.name}} support both [whitelists]((/Crowdsec/v1/write_configurations/whitelist/) and [simulation](/Crowdsec/v1/references/simulation/) :
+{{v1X.crowdsec.name}} support both [whitelists](/Crowdsec/v1/write_configurations/whitelist/) and [simulation](/Crowdsec/v1/references/simulation/) :
 
  - Whitelists allows you to "discard" events or overflows
  - Simulation allows you to simply cancel the decision that is going to be taken, but keep track of it
 
  {{v1X.profiles.htmlname}} allows you to control which decision will be applied to which alert.
 
+## How to know if my setup is working correctly ? Some of my logs are unparsed, is it normal ?
+
+Yes, crowdsec parsers only parse the logs that are relevant for scenarios :)
+
+Take a look at `cscli metrics` [and understand what do they mean](/Crowdsec/v1/getting_started/crowdsec-tour/#reading-metrics) to know if your setup is correct.
+
+
 ## How to add whitelists ?
 
 You can follow this [guide](/Crowdsec/v1/write_configurations/whitelist/)

docs/v1.X/docs/getting_started/crowdsec-tour.md

@@ -190,6 +190,26 @@ INFO[0000] Local Api Bouncers Metrics:
 
 </details>
 
+### Reading metrics
+
+Those metrics are a great way to know if your configuration is correct :
+
+The `Acquisition Metrics` is a great way to know if your parsers are setup correctly :
+
+ - If you have 0 **LINES PARSED** for a source : You are probably *missing* a parser, or you have a custom log format that prevents the parser from understanding your logs.
+ - However, it's perfectly OK to have a lot of **LINES UNPARSED** : Crowdsec is not a SIEM, and only parses the logs that are relevant to its scenarios. For example, [ssh parser](https://hub.crowdsec.net/author/crowdsecurity/configurations/sshd-logs),  only cares about failed authentication events (at the time of writting).
+ - **LINES POURED TO BUCKET** tell you that your scenarios are matching your log sources : it means that some events from this log source made all their way to an actual scenario
+
+
+The `Parser Metrics` will let you troubleshoot eventual parser misconfigurations :
+ - **HITS** is how many events where fed to this specific parser
+ - **PARSED** and **UNPARSED** indicate how many events successfully come out of the parser
+
+For example, if you have a custom log format in nginx that is not supported by the default parser, you will end up seeing a lot of **UNPARSED** for this specific parser, and 0 for **PARSED**.
+
+For more advanced metrics understanding, [take a look at the dedicated prometheus documentation](/Crowdsec/v1/observability/prometheus/).
+
+
 ## Deploy dashboard
 
 ```bash
@@ -217,3 +237,11 @@ sudo cscli collections install crowdsecurity/nginx
 Collections are bundles of parsers/scenarios that form a coherent ensemble to analyze/detect attacks for a specific service. It is the most common way to deploy configurations.
 
 They can be found and browsed on the {{v1X.hub.htmlname}}
+
+## Scalability
+
+Crowdsec uses go-routines for parsing and enriching logs, pouring events to buckets and manage outputs.
+
+By default, one routine of each exists (should be enough to handle ~1K EP/s), and can be changed in `crowdsec_service` of the main configuration file via the [parser_routines](/Crowdsec/v1/references/crowdsec-config/#parser_routines), [buckets_routines](/Crowdsec/v1/references/crowdsec-config/#buckets_routines) and [output_routines](/Crowdsec/v1/references/crowdsec-config/#output_routines) directives.
+
+Please keep in mind that thanks to the [http API]({{v1X.lapi.swagger}}), the workload of log parsing can be splitted amongst several agents pushing to a single {{v1X.lapi.Htmlname}}.

docs/v1.X/docs/user_guide/forensic_mode.md

@@ -35,6 +35,10 @@ INFO[13-11-2020 13:05:24] (14baeedafc1e44c08b806fc0c1cd92c4/crowdsec) crowdsecur
 And as these alerts are as well pushed to database, it mean you can view them in metabase, or using cscli !
 
 
+!!! warning
+    To work in forensic mode, crowdsec-agent relies on [crowdsecurity/dateparse-enrich](https://hub.crowdsec.net/author/crowdsecurity/configurations/dateparse-enrich) to parse date formats. See dedicated hub page for supported formats.
+    
+
 ## Injecting alerts into existing database
 
 If you already have a running crowdsec/Local API running and want to inject events into existing database, you can run crowdsec directly :