Release 3.25

• Support for fuse-overlayfs as alternative to kernel-based overlayfs
  BenchExec uses overlayfs for providing a virtualized file-system in its containers (unless configured otherwise by the user). Unfortunately, the overlayfs implementation in the Linux kernel does not support all use cases such as overlayfs for the root directory or triple-nested containers (cf. #776 and #1067). Now BenchExec makes use of the alternative implementation fuse-overlayfs if installed in version 1.10 or newer as an automatic fallback when necessary. This makes BenchExec work again in its default configuration without requiring parameters like --read-only-dir /.
  We are glad about this long-awaited feature being contributed by GSoC participant @younghojan! Thanks!
• Improve LXCFS integration
  If installed, BenchExec uses LXCFS to provide a better virtualization of the environment visible inside the container, for example by virtualizing the uptime. Now we use LXCFS also to virtualize CPU information in /proc/cpuinfo and /sys/devices/system/cpu. This allows the benchmarked process to see more easily how many CPU cores they are allowed to use.
• New tool-info module for super_prove.

Release 3.24

• Duplicate tasks in benchmark definitions ignored.
  Inside a single <tasks> tag in a benchmark definition it can happen that a specific task is included more than once.
  These copies of the task would be exactly identical and thus redundant, but they also create problems for benchmarking because they violate the assumption that tasks have unique identifiers. So now benchexec simply ignores such duplicates of tasks.
  This makes it easier to write benchmark definitions with broad wildcards, for example if you want to include both foo*.yml and *foo.yml but some tasks match both wildcards.
• Two new tool-info modules for KeY CLI and Pono.

The HTML tables produced by table-generator also contain a major technical improvement thanks to our GSoC participant @EshaanAgg, but no user-visible changes.

Release 3.23

As announced previously, this release works only on Python 3.8 and newer!

• Fix a potential crash for CPAchecker when using /usr/bin/cpachecker.

Release 3.22

This will be the last release of BenchExec to support Python 3.7.
Future versions will require Python 3.8, and in 2025 we are planning to drop support for Python 3.8 and 3.9.
Please comment here if these plans would create problems for you.

BenchExec is now available as an official package in the NixOS distribution. Thank you @lorenzleutgeb!

• BenchExec now handles new restrictions imposed by Ubuntu 24.04.
  Our Ubuntu package is recommended for installation because it automatically does everything for making BenchExec work out of the box. Users of other installation methods need to tweak their system config, and both our documentation and the error message of BenchExec now inform about what is necessary. Thank you @younghojan!
• BenchExec is now easier to use on systems without systemd but with cgroups v2.
  This is a common situation in containers, and BenchExec will now automatically use the /benchexec cgroup if it exists and has no running processes. This makes it unnecessary to manually start BenchExec in a fresh cgroup, but the /benchexec cgroup still needs to be created upfront. We give examples how to do this in our documentation.
• Several robustness improvements to BenchExec's container mode for non-standard environments.
  This covers for example containers with invalid cgroup mounts, systems with procfs mounts in several places, missing DBus, and Docker Desktop.
• Several fixes and improvements for the HTML tables produced by table-generator.
  Thank you @EshaanAgg and @JawHawk!
  • Filters for text columns are now case insensitive.
  • Plots now have a reset button for clearing configuration changes.
  • Text filters now work even if special characters like _ or parentheses are used.
  • Changes to the plot configuration no longer break the application if it was opened from paths with spaces and other special characters.
  • The drop-down area for status filters now immediately shows the correct value when opening a table via a link with preconfigured filters.
  • The filter for the left-most column is now correctly usable again after a task-id filter in the filter sidebar was set and cleared.
• Fix handling of tools that read from stdin when asked to print their version.
  In such a case, the tool (and thus BenchExec) would previously hang but now stdin of the tool is connected to /dev/null (just like during the actual execution) and the tool immediately gets EOF.
• Improvements to our documentation.
  We now have a quickstart tutorial for runexec and a guide specifically for executing BenchExec in containers. Thank you @incaseoftrouble!
• Improvements for several tool-info modules.
• Integration of BenchExec and the cluster management tool SLURM.
  It is not officially part of BenchExec and we do not provide any guarantees related to it, but our repository now contains an
  integration of BenchExec and SLURM that users of SLURM might find helpful. Thank you @leventeBajczi! Further contributions in this area are also welcome.

We celebrate that this release sets a new record for contributions from non-maintainers and thank all contributors!

Release 3.21

• table-generator computes scores according to SV-COMP'24 scoring scheme.
  This changes only the scoring for witness-validation results.
• Support for property files at HTTP(S) URLs in table-generator.
  Tables can already be produced not only from local result files, but also from files that are downloaded on the fly.
  Now this also works for results with property files.
• Fix for tool-info module witnesslint.

Release 3.20

• Two tool-info modules improved.

Release 3.19

• Tool-info modules can now provide URLs that will be used for links in HTML tables.
  There are two new methods, project_url() and url_for_version() that can be implemented, and table-generator will put links to these URLs in the summary table of the HTML tables (for the tool name and the version). Most existing tool-info modules were extended with project_url().
• Many new and improved tool-info modules.

Release 3.18

The big change in this release is the long-awaited support for cgroups v2!
Please refer to the installation instructions for how to use it (on Ubuntu/Debian, installing our package is enough). Note that this of course has not been tested yet on as many different systems as our support for cgroups v1, so there might still be some rough edges. Please provide feedback if you encounter any problems or have questions. On systems with cgroups v1, everything should work the same way as before.
Thanks a lot to @globin for providing the implementation of this!

There are also some other minor improvements:

• If the system administrator has configured a lower frequency limit than what the CPU supports, BenchExec now reports this limit as the CPU speed.
• Fixes in table-generator for result files that conform to the specification but were not produced by benchexec.
• A few minor improvements for better integration with Podman containers.

Release 3.17

• Even more robust handling of child cgroups created within a run
  Despite the improvements from BenchExec 3.13 there could be crashes because sometimes cgroups vanish while we iterate over them. Avoiding this did not fully work, maybe due to some delays in the kernel, so now we handle this.
• Actually fix handling of non-printable characters in environment variables.
  The fix that was part of BenchExec 3.12 only worked if a non-printable character was the first character of the value of the environment variable.

Release 3.16

This release works only on Python 3.7 and newer!

• Improve performance of the "Merging results" step of table-generator.
  In the common case of all run sets having the same set of tasks, this step is now much faster.
• Avoid deadlock in table-generator for cases with many child processes and heavy system load.
• On systems with many cores and if there are many input files or columns, table-generator will now use more than 32 child processes.
• Support the scoring schema used in SV-COMP'23 for witness validators, and show the witness category as part of the task identifier in tables.
  Note that just like for the category correct-unconfirmed external postprocessing of the results is necessary for this.
• If the pattern inside a <propertyfile> tag in the benchmark definition does not match a file, BenchExec now terminates with an error instead of just logging a warning.
• Improved logging in table-generator.
  The log messages about missing run results and missing properties, which could occur many times in certain use cases, are now omitted. Instead, some numbers about the size of the resulting tables and the number of missing results will be logged.