diff --git a/chaos-days/blog/2024-10-24-Camunda-Exporter-MVP/current-miro.png b/chaos-days/blog/2024-10-24-Camunda-Exporter-MVP/current-miro.png new file mode 100644 index 000000000..d9b352aaa Binary files /dev/null and b/chaos-days/blog/2024-10-24-Camunda-Exporter-MVP/current-miro.png differ diff --git a/chaos-days/blog/2024-10-24-Camunda-Exporter-MVP/exporter-opened.png b/chaos-days/blog/2024-10-24-Camunda-Exporter-MVP/exporter-opened.png new file mode 100644 index 000000000..b96936fbe Binary files /dev/null and b/chaos-days/blog/2024-10-24-Camunda-Exporter-MVP/exporter-opened.png differ diff --git a/chaos-days/blog/2024-10-24-Camunda-Exporter-MVP/first-time-seeing-camunda-exporter.png b/chaos-days/blog/2024-10-24-Camunda-Exporter-MVP/first-time-seeing-camunda-exporter.png new file mode 100644 index 000000000..7b0fe9eb3 Binary files /dev/null and b/chaos-days/blog/2024-10-24-Camunda-Exporter-MVP/first-time-seeing-camunda-exporter.png differ diff --git a/chaos-days/blog/2024-10-24-Camunda-Exporter-MVP/how-brown-field.png b/chaos-days/blog/2024-10-24-Camunda-Exporter-MVP/how-brown-field.png new file mode 100644 index 000000000..6a2394490 Binary files /dev/null and b/chaos-days/blog/2024-10-24-Camunda-Exporter-MVP/how-brown-field.png differ diff --git a/chaos-days/blog/2024-10-24-Camunda-Exporter-MVP/index.md b/chaos-days/blog/2024-10-24-Camunda-Exporter-MVP/index.md new file mode 100644 index 000000000..f98670b34 --- /dev/null +++ b/chaos-days/blog/2024-10-24-Camunda-Exporter-MVP/index.md @@ -0,0 +1,242 @@ +--- +layout: posts +title: "Camunda Exporter MVP" +date: 2024-10-24 +categories: + - camunda + - exporter +tags: + - performance +authors: zell +--- + +# Chaos Day Summary + +After a long pause I come back with some interesting to share and experiment with. Right now we are re-architecture +Camunda 8. One important part (where I'm contributing on) is to get rid of Webapps Importer/Archivers and move +data aggregation closer to the engine (inside a Zeebe Exporter). + +Today, I want to experiment with the first increment/iteration of our so-called MVP. The MVP targets green field installations where you simply deploy Camunda (with a new Camunda Exporter enabled) without Importers. + +**TL;DR;** All our experiments were successful. The MVP is a success, and we are looking forward to further improvements, additions. Next stop Iteration 2: Adding Archiving historic data and prepare for data migration (and polish MVP). + +## Camunda Exporter + +The [Camunda Exporter project](https://github.com/camunda/product-hub/issues/2128) deserves a complete own blog post, here is just a short summary. + +Our current Camunda architecture looks something like this (simplified). + +![current](current-miro.png) + +It has certain challenges, like: + +* Space: duplication of data in ES +* Maintenance: duplication of importer and archiver logic +* Performance: Round trip (delay) of data visible to the user +* Complexity: installation and operational complexity (we need separate pods to deploy) +* Scalability: Importer are not scalable in the same way as Zeebe or brokers (and workload) are. + +These challenges we obviously wanted to overcome and the plan (as teasered earlier) is to get rid of the need of separate importers and archivers (and in general to have separate application; but this is a different topic). + +The plan for this project looks something like this: + +![plan](how-brown-field.png) + +We plan to: + +1. Harmonize the existing indices stored in Elasticsearch/Opensearch + 2. Space: Reduce the unnecessary data duplication +2. Move importer and archiver logic into a new Camunda exporter + 3. Performance: This should allow us to reduce one additional hop (as we don't need to use ES/OS as queue) + 4. Maintenance: Indices and business logic is maintained in one place + 5. Scalability: With this approach we can scale with partitions, as Camunda Exporters are executed for each partition separately (soon partition scaling will be introduced) + 6. Complexity: The Camunda Exporter will be built-in and shipped with Zeebe/Camunda 8. No additional pod/application needed. + +Note: Optimize is right now out of scope (due to time), but will later be part of this as well. + + +### MVP + +After we know what we want to achieve what is the Minimum viable product (MVP)? + +We have divided the Camunda Exporter in 3-4 iterations. You can see and read more about this [here](https://github.com/camunda/issues/issues/803). + +The first iteration contains the MVP (the first breakthrough). Providing the Camunda Exporter with the basic functionality ported from the Operate and Tasklist importers, writing into harmonized indices. + +The MVP is targeting green field installations (clean installations) of Camunda 8 with Camunda Exporter without running the old Importer (no data migration yet), + +![mvp](it1-mvp.png) + + + +## Chaos Experiment + +What I want to verify today, when I deploy the Camunda 8 stack with Camunda Exporter (and Importer disabled): + + * Are webapps schemas created in ES, by the new Camunda Exporter + * Is data exported into the indices + * Can Operate show data ? (right now just checking for basic functionality) + +Additionally, I would like to understand how the performance looks like, the system behaves with two ES exporters (the old ES exporter and the new Camunda Exporter), and more. + +For our experiment I use a [new defined realistic benchmark](https://github.com/camunda/camunda/issues/21472) (with a more complex process model). More about this in a separate blog post. + +### Expected + +I can deploy the newest helm charts (alpha stage), with disabling Importer manually and will be able to use Zeebe and Operate together. See verifications above. + +### Actual + +As always we use our [benchmark-helm charts](https://github.com/zeebe-io/benchmark-helm) (that building on top of our [Camunda Platform Helm](https://github.com/camunda/camunda-platform-helm) charts). + + +### Installation + +I had to adjust our benchmarks to [use the alpha snapshots ](https://github.com/zeebe-io/benchmark-helm/commit/db682a89788d6c511083ec743c6cf7d358155e3c) + +```yaml +dependencies: + - name: camunda-platform + repository: "oci://ghcr.io/camunda/helm" + version: "0.0.0-snapshot-alpha" + condition: "camunda.enabled" +``` + + +and [disable the Importer via ENV](https://github.com/zeebe-io/benchmark-helm/commit/aafac6e9ec78e9cfd2e59a5b6f30bf887a4fcbd0) + +```yaml +env: +- name: CAMUNDA_OPERATE_IMPORTERENABLED +value: "false" +``` + +With that we can install our chart: + +```shell +$ helm install zeebe-benchmark-test charts/zeebe-benchmark/ --render-subchart-notes -f charts/zeebe-benchmark/values-realistic-benchmark.yaml --set global.elasticsearch.prefix=null +``` + +### Basic First Verification + +After our benchmark chart is deployed we can already see the first time our Camunda Exporter running :tada: + +![firsttime](first-time-seeing-camunda-exporter.png) + + +Worth to mention that the Camunda Export already comes with some metrics, visibile on our Zeebe Dashboard + +![metrics](mvp-c8-exporter-metrics.png) +![metrics2](mvp-c8-exporter-metrics2.png) + + +The general overview also looks good. No obvious problem. + +![general](mvp-general-overview.png) + +Looking into logs we can see that at the start it fails temporarily because ES is not yet ready to accept the schema creation. + +``` +ERROR - Failed to open exporter 'CamundaExporter'. Retrying... +``` + +![log](exporter-opened.png) + +At some point the exporter can be opened and the loop stops. + +I think generally it shouldn't be an ERROR but more an WARN (but these are details we can fix). Follow-up. + + +### Verify Operate Data + +To make sure that Operate is not importing, I checked the Operate dashboard. We can see that there is no Importer metrics. Furthermore, in the configuration and logs we see no indication of importing. + +![op-metrics](no-importer-metrics.png) + + +We can now start to port-forward to operate: + +```shell +k port-forward svc/zeebe-benchmark-test-operate 8081:80 +``` + +When opening Operate we see unfortunately no data. + +![operate-no-data](mvp-no-data-operate.png) + +#### Investigating missing data + +We need to understand why there is no data available for Operate. + +What we saw is that the Camunda Exporter is open (logs), that it is also makes progress and data is written to elastic (metrics). What we haven't checked Elasticsearch in detail. + +Looking into ES dashboard we can see that indices are created, but the Operate indices seem to be empty. + +![es-indices](mvp-operate-indices-empty.png) + +When checking the Zeebe indices: + +![zeebe-indices](mvp-zeebe-indices-filled.png) + +we can see that they are filled. An attentive reader will also chekc that there actuall some prefix problem in the indices. + +Thanks to Deepthi which spotted this as well (and told me), we were exporting to the wrong index names. There was a [bug](https://github.com/camunda/camunda-platform-helm/blob/46f6ee9d828439b0b1cf37bae4d135ba5281a832/charts/camunda-platform-alpha/templates/zeebe/configmap.yaml#L66) existing in the current alpha Helm chart version. + +![wrong-prefix](mvp-wrong-prefix.png) + +This has been fixed with [PR-2506](https://github.com/camunda/camunda-platform-helm/pull/2506). Until this gets merged I changed this manually via: + +```shell +# Get the templates +helm template zeebe-benchmark-test charts/zeebe-benchmark/ --render-subchart-notes -f charts/zeebe-benchmark/values-realistic-benchmark.yaml --output-dir templates + +# Adjust the config map - remove the prefix +vim templates/zeebe-benchmark/charts/camunda-platform/templates/zeebe/configmap.yaml + +# Apply all manifests +k apply -f . --recursive +``` + +> **Note:** +> +> I also tried +> ```shell +> helm template charts/zeebe-benchmark/ --version 0.0.0-snapshot-alpha --show-only charts/camunda-platform/templates/zeebe/configmap.yaml --set global.elasticsearch.prefix=null +> ``` +> But this breaks the ES exporter. + +With this change we were can see that indices are correctly created and filled! + +![indices-filled](mvp-fixed-prefix-indices.png) + +Finally, we are able to see data in Operate! :rocket: **WITHOUT ANY IMPORTER.** + +![mvp-operate-data.png](mvp-operate-data.png) +![mvp-operate-instance.png](mvp-operate-instance.png) +![mvp-operate-pi.png](mvp-operate-pi.png) +![operate-overview](mvp-decisions.png) + + +## Conclusion + +The MVP is a success. We were able to provide a Camunda Exporter that creates the necessary harmonized schema and migrate the basic business logic from Operate and Tasklist into the exporter. This allows us to use only the Camunda Exporter without running any Importer pod/application. + +Great work Team :rocket: :tada: + +**Next stop:** + +_Iteration 2:_ + + * Implementing migration logic for old data + * Moving Archiver logic (for historical data) into the Exporter + * Polish MVP state (add some missing features like TreePath, etc.) + +### Additional notes + +This time I was not able to deep dive into performance or stability for this change. I plan to do this next. + +### Found Bugs + +* ERROR log level for logs that are transitive +* Auth/User indices are still prefixed with identity + diff --git a/chaos-days/blog/2024-10-24-Camunda-Exporter-MVP/it1-mvp.png b/chaos-days/blog/2024-10-24-Camunda-Exporter-MVP/it1-mvp.png new file mode 100644 index 000000000..9d2faee5b Binary files /dev/null and b/chaos-days/blog/2024-10-24-Camunda-Exporter-MVP/it1-mvp.png differ diff --git a/chaos-days/blog/2024-10-24-Camunda-Exporter-MVP/mvp-c8-exporter-metrics.png b/chaos-days/blog/2024-10-24-Camunda-Exporter-MVP/mvp-c8-exporter-metrics.png new file mode 100644 index 000000000..86392042e Binary files /dev/null and b/chaos-days/blog/2024-10-24-Camunda-Exporter-MVP/mvp-c8-exporter-metrics.png differ diff --git a/chaos-days/blog/2024-10-24-Camunda-Exporter-MVP/mvp-c8-exporter-metrics2.png b/chaos-days/blog/2024-10-24-Camunda-Exporter-MVP/mvp-c8-exporter-metrics2.png new file mode 100644 index 000000000..d332ef0ef Binary files /dev/null and b/chaos-days/blog/2024-10-24-Camunda-Exporter-MVP/mvp-c8-exporter-metrics2.png differ diff --git a/chaos-days/blog/2024-10-24-Camunda-Exporter-MVP/mvp-decisions.png b/chaos-days/blog/2024-10-24-Camunda-Exporter-MVP/mvp-decisions.png new file mode 100644 index 000000000..7928fa42b Binary files /dev/null and b/chaos-days/blog/2024-10-24-Camunda-Exporter-MVP/mvp-decisions.png differ diff --git a/chaos-days/blog/2024-10-24-Camunda-Exporter-MVP/mvp-fixed-prefix-indices.png b/chaos-days/blog/2024-10-24-Camunda-Exporter-MVP/mvp-fixed-prefix-indices.png new file mode 100644 index 000000000..24eee3991 Binary files /dev/null and b/chaos-days/blog/2024-10-24-Camunda-Exporter-MVP/mvp-fixed-prefix-indices.png differ diff --git a/chaos-days/blog/2024-10-24-Camunda-Exporter-MVP/mvp-general-overview.png b/chaos-days/blog/2024-10-24-Camunda-Exporter-MVP/mvp-general-overview.png new file mode 100644 index 000000000..c170abc2d Binary files /dev/null and b/chaos-days/blog/2024-10-24-Camunda-Exporter-MVP/mvp-general-overview.png differ diff --git a/chaos-days/blog/2024-10-24-Camunda-Exporter-MVP/mvp-no-data-operate.png b/chaos-days/blog/2024-10-24-Camunda-Exporter-MVP/mvp-no-data-operate.png new file mode 100644 index 000000000..854a5b9e2 Binary files /dev/null and b/chaos-days/blog/2024-10-24-Camunda-Exporter-MVP/mvp-no-data-operate.png differ diff --git a/chaos-days/blog/2024-10-24-Camunda-Exporter-MVP/mvp-operate-data.png b/chaos-days/blog/2024-10-24-Camunda-Exporter-MVP/mvp-operate-data.png new file mode 100644 index 000000000..b3a3134bc Binary files /dev/null and b/chaos-days/blog/2024-10-24-Camunda-Exporter-MVP/mvp-operate-data.png differ diff --git a/chaos-days/blog/2024-10-24-Camunda-Exporter-MVP/mvp-operate-indices-empty.png b/chaos-days/blog/2024-10-24-Camunda-Exporter-MVP/mvp-operate-indices-empty.png new file mode 100644 index 000000000..e34ee0dae Binary files /dev/null and b/chaos-days/blog/2024-10-24-Camunda-Exporter-MVP/mvp-operate-indices-empty.png differ diff --git a/chaos-days/blog/2024-10-24-Camunda-Exporter-MVP/mvp-operate-instance.png b/chaos-days/blog/2024-10-24-Camunda-Exporter-MVP/mvp-operate-instance.png new file mode 100644 index 000000000..d90f3df62 Binary files /dev/null and b/chaos-days/blog/2024-10-24-Camunda-Exporter-MVP/mvp-operate-instance.png differ diff --git a/chaos-days/blog/2024-10-24-Camunda-Exporter-MVP/mvp-operate-pi.png b/chaos-days/blog/2024-10-24-Camunda-Exporter-MVP/mvp-operate-pi.png new file mode 100644 index 000000000..2c6e130f4 Binary files /dev/null and b/chaos-days/blog/2024-10-24-Camunda-Exporter-MVP/mvp-operate-pi.png differ diff --git a/chaos-days/blog/2024-10-24-Camunda-Exporter-MVP/mvp-wrong-prefix.png b/chaos-days/blog/2024-10-24-Camunda-Exporter-MVP/mvp-wrong-prefix.png new file mode 100644 index 000000000..5e8d7c633 Binary files /dev/null and b/chaos-days/blog/2024-10-24-Camunda-Exporter-MVP/mvp-wrong-prefix.png differ diff --git a/chaos-days/blog/2024-10-24-Camunda-Exporter-MVP/mvp-zeebe-indices-filled.png b/chaos-days/blog/2024-10-24-Camunda-Exporter-MVP/mvp-zeebe-indices-filled.png new file mode 100644 index 000000000..2cd814138 Binary files /dev/null and b/chaos-days/blog/2024-10-24-Camunda-Exporter-MVP/mvp-zeebe-indices-filled.png differ diff --git a/chaos-days/blog/2024-10-24-Camunda-Exporter-MVP/no-importer-metrics.png b/chaos-days/blog/2024-10-24-Camunda-Exporter-MVP/no-importer-metrics.png new file mode 100644 index 000000000..a278c554f Binary files /dev/null and b/chaos-days/blog/2024-10-24-Camunda-Exporter-MVP/no-importer-metrics.png differ