Tag repo before running benchmark and include it in json

[holly-cummins]

Ideally we'd only tag if the code is different to the last tag.
We can then include this tag in the json.

This supports quarkusio/benchmarks#66.

[edeandrea]

How should we tag it? By timestamp? Label?

I could also add an input field on the job template in the perf lab where the user could put something in, but then we'd have to validate/ensure it doesn't exist/etc.

[holly-cummins]

How should we tag it? By timestamp? Label?

Good question! It will be displayed on the repo link, so I think shorter is better, to make it easier to type and so it doesn't take up too much space. There's a balance between shortness and informativeness. We could do just an increment, like r32. Or we could do a version scheme, like v0.3.2. GitHub recommend using semantic versioning schemes, but that would be hard to calculate in an automated way.

Timestamps would be informative, but they're rather long, and it could get confusing if we do the check for a current tag before retagging. It would mean that if the repo code stabilises, we could be showing a tag like "2026-03-05" for a benchmark that was run in 2027.

I could also add an input field on the job template in the perf lab where the user could put something in, but then we'd have to validate/ensure it doesn't exist/etc.

It feels like something more systematic might make the results easier to interpret (and make it easier to kick off jobs in an automated way for new releases, say). And the person kicking off the job might not have enough information to assign a semantic version tag, say.

Tricky!

[edeandrea]

And the person kicking off the job might not have enough information to assign a semantic version tag, say.

Very true. I was more thinking they could type something about why they ran the job (new release of version X of framework Y, wanted to adjust cpu/memory/etc).

Although those would all be very long, but still might be interesting to capture.

I'm still torn as to the right answer.

If we do semantic versioning, how do we apply the versions? If we have a version of the code in the repo, then there may be many executions against that version.

So that brings us back to date/time stamps.

Maybe <branch_name>-yyyymmdd-hhmmss?

[holly-cummins]

Very true. I was more thinking they could type something about why they ran the job (new release of version X of framework Y, wanted to adjust cpu/memory/etc).

Although those would all be very long, but still might be interesting to capture.

I agree there's no harm in having that in the json. But probably not as the basis for the tag.

I'm still torn as to the right answer.

If we do semantic versioning, how do we apply the versions? If we have a version of the code in the repo, then there may be many executions against that version.

Exactly. I think it's important that the version be the same even if the execution is different, so we need to be a bit smart in assigning the version/tag – it has to be the same for the same level of the codebase, even if the execution happened at a different time.

So that brings us back to date/time stamps.

Maybe <branch_name>-yyyymmdd-hhmmss?

But the aim is to allow someone to get to/understand the version of the benchmark + automation that was run. We're trying to tag the repo, not the execution. If we assign different tags to the same code, then that's confusing. (If we want to show the timestamp on the graph, we can just do that, and we don't need to tag the repo.)

[edeandrea]

But the aim is to allow someone to get to/understand the version of the benchmark + automation that was run. We're trying to tag the repo, not the execution. If we assign different tags to the same code, then that's confusing. (If we want to show the timestamp on the graph, we can just do that, and we don't need to tag the repo.)

I think I understand what you are saying, so let me throw out a theoretical example.

We tag this repo (spring-quarkus-perf-comparison) with a version representing some state of the tests (call it v1.0.0 for this example).

Then maybe the tests themselves don't change for a year, but new versions of spring/quarkus come along and we re-run the tests against those versions.

Does that mean there will still only be a single tag in the results (v1.0.0)? I would think we'd want to differentiate tags based the composition of the run? Or are you saying thats irrelevant, that the version should only capture the state of the benchmarks themselves?

[holly-cummins]

I think I understand what you are saying, so let me throw out a theoretical example.

We tag this repo (spring-quarkus-perf-comparison) with a version representing some state of the tests (call it v1.0.0 for this example).

Then maybe the tests themselves don't change for a year, but new versions of spring/quarkus come along and we re-run the tests against those versions.

Does that mean there will still only be a single tag in the results (v1.0.0)?

That's right. Actually, I guess it depends, because the dependencies are baked into the repo. So we could do it two ways:

• New commit has means new tag
• Exclude the dependencies (pom.xml) from analysis when deciding if the benchmark has 'changed', so we only look at application code and automation scripts. This is sort of logical, because versions could be overridden on the command line anyway, but a lot more work.

I would think we'd want to differentiate tags based the composition of the run? Or are you saying thats irrelevant, that the version should only capture the state of the benchmarks themselves?

I think we would want some unique signature for an individual execution, but I don't think it should be the repo tag. Otherwise we'd have different repo tags representing the exact same source code and automation, and it's hard to know what's an apples-to-apples comparison. So instead "we'd want to differentiate tags based on the composition of the run," it would be "we'd want a unique identifier based on the composition of the run." Having stable tags for stable code means we can do things like "Hmm, Quarkus performance has doubled, but the repo tag is the same. What's changed? Ah, someone did a run with a different version and this other version has different performance." or the opposite, "Hmm, Quarkus performance has doubled? How did that happen? Ah, the repo tag is the different, someone changed the scripts to add isolation."

[edeandrea]

So the tag is different than the identifier? Essentially in the output we just want to know which version of the automation produced that result?

If thats the case, then we should have a release process for the automation + apps (although the app versions are most likely overridden during the run - so just because, say, in a tag of the repo it says quarkus 3.30.1, that doesn't necessarily mean that quarkus 3.30.1 was used during the run).

We're also already somewhat capturing that during the run. It defaults to the main branch of https://github.com/quarkusio/spring-quarkus-perf-comparison but it can be overridden on a per-run basis.

Maybe we should do versioned releases, and then have a latest alias or something? That way the perf lab automation doesn't need to care about figuring out the latest version. Either that or we create a script that knows how to look through the tags to determine what "latest" means, and then that tag is what's used/recorded.

In any event, once that is figured out, that info is already there in the result json (env.repo element).