[Release] logzio APM Collector 1.0.0

.github/workflows/logzio-apm-collector-test.yaml

@@ -0,0 +1,94 @@
+name: Test `logzio-apm-collector` chart
+
+on:
+  pull_request:
+    branches:
+      - master
+    paths:
+      - 'charts/logzio-apm-collector/Chart.yaml'
+      - 'charts/logzio-apm-collector/templates/**'
+      - 'charts/logzio-apm-collector/values.yaml'
+jobs:
+  test-helm-chart:
+    name: Test Helm Chart on Kind
+    runs-on: ubuntu-latest
+    steps:
+      - name: Generate random id
+        id: random_id
+        run: echo "::set-output name=rand::$(echo $RANDOM)"
+
+      - name: Set ENV_ID
+        run: echo "ENV_ID=apm-test-run-${{ steps.random_id.outputs.rand }}" >> $GITHUB_ENV
+
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Set up Go
+        uses: actions/setup-go@v5
+        with:
+          go-version: '1.20'
+
+      - name: Set up Helm
+        uses: azure/[email protected]
+
+      - name: Set up kubectl
+        uses: azure/setup-kubectl@v4
+
+      - name: Install Kind
+        run: |
+          curl -Lo ./kind "https://kind.sigs.k8s.io/dl/v0.11.1/kind-Linux-amd64"
+          chmod +x ./kind
+          mv ./kind /usr/local/bin/kind
+
+      - name: Create Kind cluster
+        run: |
+          kind create cluster --name kind-${{ github.run_id }}-${{ matrix.mode }}
+          kubectl cluster-info
+
+      - name: Deploy Helm Chart
+        run: |
+          cd charts/logzio-apm-collector
+          helm upgrade --install \
+          --set enabled=true \
+          --set spm.enabled=true \
+          --set serviceGraph.enabled=true \
+          --set global.logzioTracesToken=${{ secrets.LOGZIO_TRACES_TOKEN }} \
+          --set global.logzioSpmToken=${{ secrets.LOGZIO_METRICS_TOKEN }} \
+          --set global.logzioRegion="us" \
+          --set global.env_id=${{ env.ENV_ID }} \
+          logzio-apm-collector .
+          kubectl rollout status deployment/logzio-apm-collector --timeout=300s
+          kubectl rollout status deployment/logzio-apm-collector-spm --timeout=300s
+
+      - name: Run trace generator
+        run: |
+          kubectl apply -f tests/resources/tracegen-apm.yaml
+          kubectl rollout status deployment/trace-gen --timeout=300s
+
+      - name: Run otel demo for service graph
+        run: |
+          helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
+          helm repo update
+          helm install otel-demo -f tests/resources/otel-demo-apm.yaml open-telemetry/opentelemetry-demo --version 0.32.5
+          kubectl rollout status deployment/otel-demo-loadgenerator --timeout=300s
+
+      - name: sleep for 3 minutes
+        run: sleep 180
+
+      - name: Run Go Tests
+        env:
+          LOGZIO_METRICS_API_KEY: ${{ secrets.LOGZIO_METRICS_API_KEY }}
+          LOGZIO_TRACES_API_KEY: ${{ secrets.LOGZIO_TRACES_API_KEY }}
+        run: |
+          go get go.uber.org/zap
+          go test -v ./tests/traces_e2e_test.go ./tests/common.go
+          go test -v ./tests/apm_metrics_e2e_test.go ./tests/common.go
+
+      - name: Cleanup Environment
+        run: |
+          helm uninstall logzio-apm-collector
+
+      - name: Delete Kind cluster
+        if: always()
+        run: kind delete cluster --name kind-${{ github.run_id }}-${{ matrix.mode }}
+

charts/logzio-apm-collector/.helmignore

@@ -0,0 +1,23 @@
+# Patterns to ignore when building packages.
+# This supports shell glob matching, relative path matching, and
+# negation (prefixed with !). Only one pattern per line.
+.DS_Store
+# Common VCS dirs
+.git/
+.gitignore
+.bzr/
+.bzrignore
+.hg/
+.hgignore
+.svn/
+# Common backup files
+*.swp
+*.bak
+*.tmp
+*.orig
+*~
+# Various IDEs
+.project
+.idea/
+*.tmproj
+.vscode/

charts/logzio-apm-collector/CHANGELOG.md

@@ -0,0 +1,7 @@
+# Changes by Version
+
+<!-- next version -->
+
+## 1.0.0
+- Initial release 
+- Kubernetes APM Agent for Logz.io, based on OpenTelemetry Collector

charts/logzio-apm-collector/Chart.yaml

@@ -0,0 +1,11 @@
+apiVersion: v2
+name: logzio-apm-collector
+version: 1.0.0
+description: Kubernetes APM agent for Logz.io based on OpenTelemetry Collector
+type: application
+home: https://logz.io/
+icon: https://logzbucket.s3.eu-west-1.amazonaws.com/logz-io-img/logo400x400.png
+maintainers:
+  - name: Naama Bendalak
+    email: [email protected]
+appVersion: 0.116.1

charts/logzio-apm-collector/README.md

@@ -0,0 +1,180 @@
+# Logz.io APM Collector Helm Chart
+> [!IMPORTANT]
+> Kubernetes APM Collection Agent is still In development
+
+This Helm chart deploys an agent, which leverages the OpenTelemetry Collector, that collects traces and span metrics from Kubernetes clusters and sends them to Logz.io.
+
+
+## Prerequisites
+- Kubernetes 1.24+
+- Helm 3.9+
+
+## Installation
+### Add Logz.io Helm Repository
+Before installing the chart, add the Logz.io Helm repository:
+```shell
+helm repo add logzio-helm https://logzio.github.io/logzio-helm
+helm repo update
+```
+
+### Install the Chart
+
+The chart provides options for enabling the following:
+1. Traces
+2. SPM (Service Performance Monitoring)
+3. Service Graph 
+
+
+```shell
+helm install -n monitoring --create-namespace \
+--set enabled=true \
+--set spm.enabled=true \
+--set serviceGraph.enabled=true \
+--set global.logzioTracesToken="<<LOGZIO_TRACES_TOKEN>>" \
+--set global.logzioSpmToken="<<LOGZIO_SPM_TOKEN>>" \
+--set global.logzioRegion="<<LOGZIO_REGION_CODE>>" \
+--set global.env_id="<<ENV_ID>>" \
+logzio-apm-collector logzio-helm/logzio-apm-collector
+```
+
+> [!NOTE]
+> To disable either one of SPM or Service Graph remove the relevant `--set XXX.enabled` line from the above command.
+
+> [!IMPORTANT]
+> Values of `<<LOGZIO_TRACES_TOKEN>>`, `<<LOGZIO_SPM_TOKEN>>` and `<<LOGZIO_REGION_CODE>>` can be found in your Logz.io account.  
+> For `<<ENV_ID>>` define any environment identifier attribute (for example, the cluster name).
+
+
+## Configuration
+
+- [All configuration options](./VALUES.md)
+- [Instrumentation](#instrumentation)
+- [Custom Trace Sampling rules](#custom-trace-sampling-rules)
+- [Enable File Storage extension](#enable-file-storage-extension)
+
+
+## Instrumentation
+If you're using manual instrumentation or an instrumentation agent, configure it to export data to the Logz.io APM collector by setting the export/output address as follows:
+
+```
+logzio-apm-collector.monitoring.svc.cluster.local:<<PORT>>
+```
+
+> [!IMPORTANT]
+> Replace `<<PORT>>` based on the protocol your agent uses:
+> - 4317 for GRCP
+> - 4318 for HTTP
+>
+> For a complete list, see `values.yaml` >> `traceConfig` >> `receivers`.
+
+
+## Custom trace sampling rules
+To customize the Traces Sampling rules in the OpenTelemetry Collector, you can follow the below steps:
+
+- **Step 1**: Create [customized Tail sampling rules configuration](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/tailsamplingprocessor).
+
+- **Step 2**: Update the `values.yaml` file:
+
+Get the current Chart's `values.yaml` file:
+```shell
+helm get values logzio-apm-collector -n monitoring -a > new-values.yaml
+```
+
+Edit the section under `traceConfig` >> `processors` >> `tail_sampling` in `new-values.yaml` to contain the custom config which you created in step 1.
+
+- **Step 3**: Apply the config:
+```shell
+helm upgrade logzio-apm-collector logzio-helm/logzio-apm-collector -n monitoring -f new-values.yaml
+```
+
+
+## Enable File Storage extension
+The [OpenTelemetry File Storage extension](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/storage/filestorage) allows storing temporary data on disk rather than in memory. This helps reduces memory stress, particularly in high-load scenarios, and ensures that state is persisted on disk.
+
+To enable the File Storage extension, follow the below steps:
+
+### **Step 1**: Add the File Storage Extension
+
+Update the `traceConfig` and/or `spmConfig` in `values.yaml` to include the File Storage extension.
+
+1. Add your custom configuration under `extensions` section
+2. Include the extension in the `service` configuration and in the exporter's `sending_queue` settings.
+
+Example:
+
+```yaml
+traceConfig:
+  exporters:
+    logzio:
+      ...
+      sending_queue:
+        storage: file_storage
+  ...
+  extensions:
+    ...
+    file_storage:
+      directory: /var/lib/otelcol
+  ...
+  service:
+    extensions: [health_check, file_storage]
+```
+
+### **Step 2**: Configure Disk Storage Path
+
+Edit the `extraVolumes` and `extraVolumeMounts` in `values.yaml`, to contain the path where the data should be saved on disk.
+Ensure this path matches the one set in `extensions.file_storage.directory` in step 1:
+
+```yaml
+extraVolumes:
+  - name: varlibotelcol
+    hostPath:
+      path: /var/lib/otelcol  # use the same path as `extensions.file_storage.directory`
+      type: DirectoryOrCreate
+extraVolumeMounts:
+  - name: varlibotelcol
+    mountPath: /var/lib/otelcol  # use the same path as `extensions.file_storage.directory`
+```
+
+> [!NOTE]
+> If you're using `file_storage` extension with the `compaction.directory` setting, make sure to mount that path as well
+
+
+### **Step 3**: Add user with permissions
+
+Edit the `initContainers` and `containerSecurityContext` in `values.yaml` as explained below.
+
+1. Add `initContainers` to set a user with permission to the directory.
+Ensure this path matches the one set in `extensions.file_storage.directory` in step 1:
+
+```yaml
+initContainers:
+ - name: file-storage-permissions
+   image: busybox:latest
+   command:
+     - sh
+     - '-c'
+     - 'chown -R 10001: /var/lib/otelcol'  # use the path given as per `extensions.file_storage.directory` & `extraVolumeMounts[x].mountPath`
+   volumeMounts:
+     - name: varlibotelcol  # use the name of the volume used for persistence
+       mountPath: /var/lib/otelcol  # use the path given as per `extensions.file_storage.directory` & `extraVolumeMounts[x].mountPath`
+```
+
+2. Configure `containerSecurityContext` to set the user and group for the container to match the permissions assigned in the `initContainers`:
+
+```yaml
+containerSecurityContext:
+  runAsUser: 10001  # use the user defined in initContainers per your setup
+  runAsGroup: 10001  # use the group defined in initContainers per your setup
+```
+
+### Step 4: Apply the config:
+```shell
+helm upgrade logzio-apm-collector logzio-helm/logzio-apm-collector -n monitoring -f values.yaml
+```
+
+
+## Uninstalling
+To uninstall the `logzio-apm-collector` chart, you can use:
+```shell
+helm uninstall -n monitoring logzio-apm-collector
+```