[transformer / replacer] text/template interfering with expected result

[sdurrheimer]

Hi there,

Is there a way to make updatecli not trying evaluate the go template functions in a transformer / replacer?
Like replace with the raw string, don't try to text/template it.

I'm trying to sync a file from one repo to another during release, and as the destination is a helm chart, I wanted to use a transformer / replacer to change one line into a some go template code meant to be used by the helm chart later.

With the following config, updatecli will try to evaluate the include function, when I just want it to replace with this exact string:

Code:

targets:
  push-crds-to-janitor-crds-helm-chart-repo:
    name: Push the CRDs file to the helm chart repo
    sourceid: crd-yaml
    scmid: helm-chart-repo
    kind: file
    spec:
      file: templates/crds.yaml
      forcecreate: true
    transformers:
      - replacer:
          from: "creationTimestamp: null"
          to: >
            labels:
                {{- include "rancher-janitor-operator-crds.labels" . | nindent 4 }}

Result:

Loading Pipeline "updatecli.yaml"
ERROR: failed loading pipeline(s)
	* "updatecli.yaml" - template: cfg:98: function "include" not defined

But If I try to escape the {{ and }}, the include function is no more called, but the escape characters appear in the end result:

• {{"{{"}}- include "rancher-janitor-operator-crds.labels" . | nindent 4 {{"}}"}}
• {{`"{{- include "rancher-janitor-operator-crds.labels" . | nindent 4 }}"`}}

All end up with the following, similar to the previous one, error:

SOURCES
=======
Pipeline "Push release and updated CRDs to the helm charts repos" failed
Skipping due to:
	sources stage:	"template: cfg:84: function \"include\" not defined"

If I escape more, the error moves to the next step:

• {{`{{"{{\"{{\"}}"}}`}}- include "rancher-janitor-operator-crds.labels" . | nindent 4 {{`{{"{{\"}}\"}}"}}`}}

TARGETS
========
Pipeline "Push release and updated CRDs to the helm charts repos" failed
Skipping due to:
	targets stage:	"template: cfg:84: function \"include\" not defined"

And now adding more escape options is becoming difficult

One more escape level, almost there but not good enough:

• {{`{{"{{\"{{\\\"{{\\\"}}\"}}"}}`}}- include "rancher-janitor-operator-crds.labels" . | nindent 4 {{`{{"{{\"{{\\\"}}\\\"}}\"}}"}}`}}

error:

TARGETS
========

push-crds-to-janitor-crds-helm-chart-repo
-----------------------------------------

**Dry Run enabled**

⚠ updated the [dry run] content of the file "/var/folders/zm/3ys3kgl958339xjq5dh82qgc0000gn/T/updatecli/gitlab/uks/charts/rancher-janitor-operator-crds/templates/crds.yaml"

--- /var/folders/zm/3ys3kgl958339xjq5dh82qgc0000gn/T/updatecli/gitlab/uks/charts/rancher-janitor-operator-crds/templates/crds.yaml
+++ /var/folders/zm/3ys3kgl958339xjq5dh82qgc0000gn/T/updatecli/gitlab/uks/charts/rancher-janitor-operator-crds/templates/crds.yaml
@@ -4,10 +4,10 @@
 metadata:
   annotations:
     controller-gen.kubebuilder.io/version: v0.11.3
-  creationTimestamp: null
-  name: nodecleaners.rancher.domain.tld
   labels:
     {{- include "rancher-janitor-operator-crds.labels" . | nindent 4 }}
+
+  name: nodecleaners.rancher.domain.tld
 spec:
   group: rancher.domain.tld
   names:

Pipeline "Push release and updated CRDs to the helm charts repos" failed
Skipping due to:
	targets stage:	"template: cfg:84: function \"include\" not defined"

[olblak]

Hi @sdurrheimer,

Thanks for the detailed issue, because what you highlight here seems to be a design issue introduced by #195 and #242.

Updatecli leverage Go template at two different moments in time:

• During the initial configuration parsing (Updatecli don't know yet the content of manifest)
• After each source so we can render dynamic variable {{ source "sourceID" }} such as

sources:
  stage1:
    kind: githubrelease
    spec:
      ...
  stage2:
   kind: yaml
   depends_on:
     - stage1
   spec:
     file: example.key
     key: '{{ source "stage1" }}

Reverting #195 would be a huge breaking change which I am not ready to do.

I am open to suggestion

I can add an command flag such as --no-intial-templating which would either disable the first templating and/or the runtime templating.

We could add a new pipeline settings named nodynamicvariable such

name: example
nodynamicvariable: true
sources:
  # ...
targets:
  # ...

I probably need a bit of time to explore other possibilities

[sdurrheimer]

Hi,

The dynamic variable interpolation is a interesting feature, it would be a shame to have to disable it globally because of one transformer hitting an issue.

I would have seen a boolean option like you are mentioning, but at a closer level where you would like to not do the go template interpolation.
For e.g, for this transformer, use the raw string, don't interpolate it. This way I would still be able to use {{source "stage1"}} in other places in the same config.

[olblak]

A potential workaround would be to chain transformer rule such as

targets:
  default:
    kind: shell
    spec:
      command: echo 
    transformers:
      - addprefix: '{'
      - addprefix: '{'
        replacer:
          from: hello
          to: world
      - addsuffix: '}'
      - addsuffix: '}'

I know It's a bit ugly, my initial idea was to do something like

targets:
  default:
    kind: shell
    spec:
      command: echo 
    transformers:
      - addprefix: '{{'
      - replacer:
          from: hello
          to: world
      - addsuffix: '}}'

But {{ will not pass the initial go template rendering and return an error

[sdurrheimer]

Thanks, I will have a try with the workaround.

Something worth mentioning, transformer should only work if specified at the target level

It's true that I didn't understand why my transformer set on the targets level was evaluated so many time through the execution, as we can see with the escaping chain I tried to make it work.

[sdurrheimer]

Ok, in my situation le addprefix and addsuffix will not work because I'm making a replacement within a file, so they will be put before and after the whole file content.

They would need to be within the replacer to only affect the subelement of the file I want to modify.

[sdurrheimer]

Another idea, but kinda a breaking change, is to use your custom delimiter (template.Delims()) to not conflict with other template delimiter we would like to put in the result.

Could also be an option of the tool, by default {{ and }}, but can be changed to avoid conflicts with what your are currently manipulating in strings.

[sdurrheimer]

Which made me think of the following workaround that seems to work:

push-crds-to-janitor-crds-helm-chart-repo:
    name: Push the CRDs file to the rancher-janitor-operator-crds helm chart repo
    sourceid: crd-yaml
    scmid: janitor-crds-helm-chart-repo
    kind: file
    spec:
      file: templates/crd.yaml
      forcecreate: true
    transformers:
      - replacer:
          from: "creationTimestamp: null"
          to: >
            labels:
                #>#>- include "rancher-janitor-operator-crds.labels" . | nindent 4 <#<#
      - replacer:
          from: '#>'
          to: '{'
      - replacer:
          from: '<#'
          to: '}'

[olblak]

I really like your idea of being able to change the delimiter when needed. It's probably the cleaner way to solve that problem

I still have an eye open for using HCL and/or Cuelang for writing manifest, and that problem wouldn't appear there but my previous attempt with HCL wasn't successful, even thought it was a long time ago. I failed on the dynamic variable problem

I try to keep the Updatecli core as agnostic as possible to keep that possibility open in the future.

In the current state of Updatecli we have two types of templating. One Updatecli's dagnostic which happen after loading files with the following extension [".tpl",".yaml",".yml"]. At that moment, if the templating engine find something like {{ source "sourcedID" }} then it ignores it until the next "templating" stage.

The second "templating" type happens after each source, and replace {{ source "sourceID" }} with the correct value if one is available. This is why a source output containing mustache like syntax can mess up an Updatecli stage. I thought a lot of about that problem and I couldn't find a solution.

Having a transformer rule at the target level means that it will only be interpreted once, when loading the manifest.
The reason why we need your workaround is because - addprefix: '{{' will fails the initial as we have open but not closed mustache

[olblak]

I have been thinking to this problem and while providing a different replacer than '{{' would be easy, my real question is more where should we provide that customization...

manifest

If specified via the manifest then it would only be apply after the initial templating as we need to generate and parse the manifest first...

command line

If specified via the command paramenter such as updatecli diff --template-delimeter '[[' --config updatecli.d then it would be applied to all manifests within updatecli.d which is not necessarily what we would like