Mutation Report Publisher failed to load. No HTML report found.. | Azure Devops

[DokterICT]

Describe the bug
We get sometimes errors in Azure Devops using Stryker with the error message:
Mutation Report Publisher failed to load. No HTML report found..

When we run the pipeline 5 times, this happens 4 times. 1 of the 5 times, we get an Report shown up in DevOps,
There are no changes on the code
We implement an YAML pipeline with multiple reports.

Can you assist on those error(s)?
Correct Result:

Error:

Pipeline:

    - task: DotNetCoreCLI@2
      displayName: 'Install dotnet-stryker'
      inputs:
        command: custom
        custom: tool
        arguments: 'install dotnet-stryker --tool-path $(Agent.BuildDirectory)/tools'

    - task: PowerShell@2
      displayName: 'dotnet tool update dotnet-stryker'
      inputs:
        workingDirectory: '$(System.DefaultWorkingDirectory)\tests\'
        targetType: inline
        script: dotnet tool update dotnet-stryker

    - task: PowerShell@2
      displayName: 'Run dotnet-stryker for xxxxApi.Tests'
      inputs:
        workingDirectory: '$(System.DefaultWorkingDirectory)\tests\xxxxApi.Tests'
        targetType: inline
        script: dotnet stryker --config-file "../stryker-config.json"

    - task: PowerShell@2
      displayName: 'Run dotnet-stryker for xxx.Business.Tests'
      inputs:
        workingDirectory: '$(System.DefaultWorkingDirectory)\tests\xxx.Business.Tests'
        targetType: inline
        script: dotnet stryker --config-file "../stryker-config.json"

    - task: PowerShell@2
      displayName: 'Run dotnet-stryker for xxx.DataAccess.Tests'
      inputs:
        workingDirectory: '$(System.DefaultWorkingDirectory)\tests\xxxxxxxxx.Tests'
        targetType: inline
        script: dotnet stryker --config-file "../stryker-config.json" --mutate "**/*Repository.cs"

    - task: PublishMutationReport@1
      displayName: 'Publish Mutation Test Report'
      inputs:
        reportPattern: '**/mutation-report.html'

Config json file:

{
  "stryker-config": {
    "mutation-level": "Standard",
    "reporters": [ "html", "progress" ],
    "thresholds": {
      "high": 80,
      "low": 65,
      "break": 50
    }
  }
}

[rouke-broersma]

Could you share the logs of the Stryker run in the pipeline that show the report was generated?

[DokterICT]

[rouke-broersma]

Does the Publish Mutation Testing Report step show that files were found and that they were uploaded?

You can find logging for the mutation report tab in the javascript console. Could you post the relevant logs in the javascript console with the mutation testing report tab open on a pipeline where the report fails to load?

[DokterICT]

How do I login to the javascript console ? :)

[rouke-broersma]

F12 to open your browser's developer console, then go to the console tab.

[DokterICT]

Good morning Rouke, sorry for the late response, but I was on vacation.
Hereby the logging of the Console

[rouke-broersma]

The logging shows that no attachments were found for this specific pipeline, which is why not report is shown in the UI. Does the Publish Mutation Testing Report step show that files were found and that they were uploaded?

It should look like this:

[DokterICT]

That is correct:

[rouke-broersma]

And this screenshot is of a pipeline where you do not have the reports available? Or is his a screenshot of the step in a pipeline where you did have the reports?

[DokterICT]

This is a screenshot, where we don't have access to the Reports.

[rouke-broersma]

In that case I'm afraid I am out of ideas. The extension is not receiving the reports from the azure devops api so there's nothing to display. Sorry.

[cizra]

Run your mutation test locally and check the size of your report. The ADO integration works fine if the mutation report is somewhere <15MB. However, we have a large project where the mutation report is 250MB and it invariably fails to upload every time.

@rouke-broersma would it be possible to use ADO pipeline artifacts, instead of build attachments? Artifacts don't appear to have a size limit.

[rouke-broersma]

@cizra I don't know for sure but seeing as pipeline artifacts upload compressed to blob storage and download an uncompress during the pipeline run I kinda assume access to pipeline artifacts contents is not possible from a UI component. If you have some information that it might be possible I could probably look into it.

[cizra]

Okay, I can imagine a couple of other options:

• try to optimize the size of the report somehow
• compress the report - I used 7zip to test on my report of 249MB of mutants. Here are the results:
  • Zip of Windows 11: 137MB
  • gz: 132.5
  • Zip of 7zip: 132MB
  • bz2: 18MB (just barely outside the limit)
  • xz: 0.6MB
  • 7z: 0.75MB

Out of these, xz sounds the most promising, using the LZMA2 encoding. I wonder if https://github.com/LZMA-JS/LZMA-JS or https://github.com/SortaCore/lzma2-js or something would be usable for this?

What do you think, @rouke-broersma ?