Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Try to compress better than pngcrush #444

Closed
txtsd opened this issue Jul 23, 2022 · 20 comments
Closed

Try to compress better than pngcrush #444

txtsd opened this issue Jul 23, 2022 · 20 comments

Comments

@txtsd
Copy link

txtsd commented Jul 23, 2022

Currently pngcrush at max settings always compresses a little more than oxipng at max settings.

Source used is “Cosmic Cliffs” in the Carina Nebula (NIRCam and MIRI Composite Image) from webbtelescope.org.

λ wget "https://stsci-opo.org/STScI-01G8GYD8B1H306SSPR4MY6NGN0.png"
--2022-07-23 12:53:58--  https://stsci-opo.org/STScI-01G8GYD8B1H306SSPR4MY6NGN0.png
Loaded CA certificate '/etc/ssl/certs/ca-certificates.crt'
Resolving stsci-opo.org (stsci-opo.org)... 65.9.95.4, 65.9.95.10, 65.9.95.43, ...
Connecting to stsci-opo.org (stsci-opo.org)|65.9.95.4|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 50166110 (48M) [image/png]
Saving to: ‘STScI-01G8GYD8B1H306SSPR4MY6NGN0.png’

STScI-01G8GYD8B1H306SSPR4MY6NGN0.png            100%[======================================================================================================>]  47.84M  5.78MB/s    in 8.0s    

2022-07-23 12:54:08 (5.96 MB/s) - ‘STScI-01G8GYD8B1H306SSPR4MY6NGN0.png’ saved [50166110/50166110]
λ pngcrush -brute -rem alla -rem allb -rem text -reduce -check STScI-01G8GYD8B1H306SSPR4MY6NGN0.png pngcrush.png
  Recompressing IDAT chunks in STScI-01G8GYD8B1H306SSPR4MY6NGN0.png to pngcrush.png
   Total length of data found in critical chunks            =  50145112
pngcrush: iCCP: known incorrect sRGB profile
   Best pngcrush method        =  16 (ws 15 fm 5 zl 2 zs 2) =  45336617
CPU time decode 101.124237, encode 2127.660889, other 0.256804, total 2229.506836 sec
λ oxipng --opt max --strip all --alpha --interlace 0 --threads 8 --out oxipng.png STScI-01G8GYD8B1H306SSPR4MY6NGN0.png
Processing: STScI-01G8GYD8B1H306SSPR4MY6NGN0.png
    11264x3904 pixels, PNG format
    3x8 bits/pixel, RGB
    IDAT size = 50144491 bytes
    File size = 50166110 bytes
Trying: 216 combinations
Found better combination:
    zc = 1  zs = 2  f = 5        45398613 bytes
    IDAT size = 45398613 bytes (4745878 bytes decrease)
    file size = 45398670 bytes (4767440 bytes = 9.50% decrease)
Output: oxipng.png
λ ls -l *.png
-rw-r--r-- 1 txtsd users 45398670 Jul 23 12:59 oxipng.png
-rw-r--r-- 1 txtsd users 45336617 Jul 23 13:35 pngcrush.png
-rw-r--r-- 1 txtsd users 50166110 Jul 21 23:25 STScI-01G8GYD8B1H306SSPR4MY6NGN0.png
@TPS
Copy link

TPS commented Jul 23, 2022

Using many tools can approach state-of-the-art. It'd be instructive to try FileOptimizer any way you can — it's the best I've found in the last decade or so, & almost certainly can do even better.

@txtsd
Copy link
Author

txtsd commented Jul 23, 2022

Thanks, but it's windows-only. I'm a Linux user.

I'm just trying to remove pngcrush from my screenshot flow. If oxipng can match or outcrush pngcrush, my screenshot flow will take a lot less time and CPU.

@TPS
Copy link

TPS commented Jul 23, 2022

I'm just trying to remove pngcrush from my screenshot flow. If oxipng can match or outcrush pngcrush, my screenshot flow will take a lot less time and CPU.

Optimum PNG compression is always time-consuming. If you're optimizing filesize for web use, maybe consider .WebP or .JxL? E.g., the large PNG from your example link above quite quickly becomes a 43 638 670 B lossless WebP.

@TPS
Copy link

TPS commented Jul 30, 2022

Thanks, but it's windows-only. I'm a Linux user.

Also, FO runs fine on ReactOS, which, in turn, runs well on Linux via VirtualBox 5.

@8573
Copy link

8573 commented Aug 2, 2022

FO runs fine on ReactOS, which, in turn, runs well on Linux via VirtualBox 5.

One also can look at File Optimizer's source code as linked from #440 (comment). It looks to me like a big shell script that happens to be written in C++: it works by running other programs on the file to be optimized. One could find the part of the linked cppMain.cpp that deals with PNG files and then run the commands that it would run (maybe as a shell script, so that it's easy to do this again). To me, this seems an easier and lighter-weight solution than setting up a virtual machine to run File Optimizer.

@TPS
Copy link

TPS commented Aug 2, 2022

To me, this seems an easier and lighter-weight solution than setting up a virtual machine to run File Optimizer.

This would be true iff many of the programs run weren't Windows-only binaries themselves. In simpler cases, that's actually quite doable, but the PNG toolchain is the most complex & has many such components.

@C0rn3j
Copy link

C0rn3j commented Dec 9, 2022

Running through already-max-optimized PNGs that were optimized with 6.0.1 with the new 7.0.0, I am seeing additional 2% to 28% decrease.

Unfortunately, retesting against your example, I get worse results.

% oxipng --opt max --strip all --alpha --interlace 0 --threads 8 --out oxipng.png STScI-01G8GYD8B1H306SSPR4MY6NGN0.png
Processing: STScI-01G8GYD8B1H306SSPR4MY6NGN0.png
    11264x3904 pixels, PNG format
    3x8 bits/pixel, RGB
    IDAT size = 50144491 bytes
    File size = 50166110 bytes
Trying: 10 filters
Found better combination:
    zc = 12  f = BigEnt    48290674 bytes
    IDAT size = 48290674 bytes (1853817 bytes decrease)
    file size = 48290731 bytes (1875379 bytes = 3.74% decrease)
Output: oxipng.png

Rerunning the 6.0.1 output against 7.0.0 provides zero new filesavings.

45398670 Dec  9 11:05 oxipng5.0.1.png
45398670 Dec  9 11:03 oxipng6.0.1.png
48290731 Dec  9 10:58 oxipng7.0.0.png

@TPS
Copy link

TPS commented Dec 27, 2022

Rerunning the 6.0.1 output against 7.0.0 provides zero new filesavings.

Maybe try again w/ original input (to avoid local minima) &/or against recent v8.𝑥, while tweaking options further? (This is mostly why I tend to shun PNG altogether nowadays in favor of something more modern, or fire up FO if it's something serious & unavoidable.)

@Oondanomala
Copy link

Its a libdeflate problem, see ebiggers/libdeflate#241

@ebiggers
Copy link

ebiggers commented Jan 2, 2023

The master branch of libdeflate now compresses the IDAT of STScI-01G8GYD8B1H306SSPR4MY6NGN0.png to 45,281,576 bytes at level 12.

@ebiggers
Copy link

The improvements were released in libdeflate v1.17. This project uses the Rust bindings https://github.com/adamkewley/libdeflater, so I guess you'll need to first get libdeflater to update libdeflate, then update libdeflater in this project.

@TPS
Copy link

TPS commented Jan 22, 2023

libdeflater has just been updated to v0.12.0, which updates to libdeflate v1.17.

@shssoichiro Is there an ETA for when this updated library will be available in an OxiPNG release?

@XhmikosR
Copy link
Contributor

XhmikosR commented Apr 7, 2023

@shssoichiro friendly ping to update to the latest libdeflater :)

@parasew
Copy link

parasew commented Apr 8, 2023

Hi all! I have provided a patch with updated Cargo.toml, where libdeflater and Zopfli are updated. Here is the gist for the test, and you can see that libdeflater 0.13.0 makes a huge difference.

  • oxipng -o6 yields 48290731 (0.11.0) / 45277379 (0.13.0)
  • oxipng --zc 12 -o max 48290731 (0.11.0) / 45277379 (0.13.0)

All tests have been made the same file the OP was mentioning. gist for the test

@txtsd
Copy link
Author

txtsd commented Apr 9, 2023

This is great! Thanks for y'all's help! I'll close this when it lands in a release.

@andrews05
Copy link
Collaborator

This has landed in #525.

@TPS
Copy link

TPS commented Jun 27, 2023

This has landed in #525.

Still hoping for a full release & associated binaries.

@XhmikosR
Copy link
Contributor

Alternatively, one could modify the GitHub Actions to create and upload the binaries for each workflow run.

@TPS
Copy link

TPS commented Jul 10, 2023

Alternatively, one could modify the GitHub Actions to create and upload the binaries for each workflow run.

A PR for this, if possible? 🙇🏾‍♂️

AlexTMjugador added a commit that referenced this issue Jul 10, 2023
As commented in issues #444 and #518, there is some user interest for
distributing binaries for each unstable commit, and target ARM64
platforms. Personally, I think both suggestions are useful for the
project, as uploading binary artifacts for each commit might help
interested users to catch regressions and give feedback earlier, and
powerful ARM64 platforms are becoming increasingly popular due to some
cloud services (e.g., Amazon EC2, Azure VMs, Oracle Cloud) offering
cheaper plans for this hardware, in addition to the well-known push for
ARM by Apple with their custom M1 chips.

These changes make the CI target ARM64 as a first-class citizen. Because
the public GitHub actions runners can only be hosted on x64 for now, I
resorted to cross-compilation, [Debian's
multiarch](https://elinux.org/images/d/d8/Multiarch_and_Why_You_Should_Care-_Running%2C_Installing_and_Crossbuilding_With_Multiple_Architectures.pdf),
and QEMU to build, get ARM64 C library dependencies, and run tests,
respectively.

When the CI workflow finishes, a release CLI binary artifact is now
uploaded, which can be downloaded from the workflow run page on the
GitHub web interface.

In addition, these changes also introduce some cleanup and miscellaneous
improvements and changes to the CI workflow:

- Tests are run using [`nextest`](https://nexte.st/) instead of `cargo
  test`, which substantially speeds up their execution. (On my
  development workstation, `cargo test --release` takes around 10.67 s,
  while `cargo nextest run --release` takes around 6.02 s.)
- The dependencies on unmaintained `actions-rs` actions were dropped in
  favor of running Cargo commands directly, or using
  `giraffate/clippy-action` for pretty inline annotations for Clippy.
  This gets rid of the deprecation warnings for each workflow run.
- Most CI steps are run with a nightly Rust toolchain now, which allows
  to take advantage of the latest Clippy lints and codegen improvements.
  In my experience, when not relying on specific nightly features or
  compiler internals, Rust does a pretty good job at making it possible
  to rely on a rolling-release compiler for CI, as breakage is extremely
  rare and thus offset by the improved features.
- The MSRV check was moved to a separate job with less steps, so that it
  takes less of a toll on total workflow run minutes.
AlexTMjugador added a commit that referenced this issue Jul 10, 2023
… and more

As commented in issues #444 and #518, there is some user interest for
distributing binaries for each unstable commit, and target ARM64
platforms. Personally, I think both suggestions are useful for the
project, as uploading binary artifacts for each commit might help
interested users to catch regressions and give feedback earlier, and
powerful ARM64 platforms are becoming increasingly popular due to some
cloud services (e.g., Amazon EC2, Azure VMs, Oracle Cloud) offering
cheaper plans for this hardware, in addition to the well-known push for
ARM by Apple with their custom M1 chips.

These changes make the CI target ARM64 as a first-class citizen. Because
the public GitHub actions runners can only be hosted on x64 for now, I
resorted to cross-compilation, [Debian's
multiarch](https://elinux.org/images/d/d8/Multiarch_and_Why_You_Should_Care-_Running%2C_Installing_and_Crossbuilding_With_Multiple_Architectures.pdf),
and QEMU to build, get ARM64 C library dependencies, and run tests,
respectively.

When the CI workflow finishes, a release CLI binary artifact is now
uploaded, which can be downloaded from the workflow run page on the
GitHub web interface.

In addition, these changes also introduce some cleanup and miscellaneous
improvements and changes to the CI workflow:

- Tests are run using [`nextest`](https://nexte.st/) instead of `cargo
  test`, which substantially speeds up their execution. (On my
  development workstation, `cargo test --release` takes around 10.67 s,
  while `cargo nextest run --release` takes around 6.02 s.)
- The dependencies on unmaintained `actions-rs` actions were dropped in
  favor of running Cargo commands directly, or using
  `giraffate/clippy-action` for pretty inline annotations for Clippy.
  This gets rid of the deprecation warnings for each workflow run.
- Most CI steps are run with a nightly Rust toolchain now, which allows
  to take advantage of the latest Clippy lints and codegen improvements.
  In my experience, when not relying on specific nightly features or
  compiler internals, Rust does a pretty good job at making it possible
  to rely on a rolling-release compiler for CI, as breakage is extremely
  rare and thus offset by the improved features.
- The MSRV check was moved to a separate job with less steps, so that it
  takes less of a toll on total workflow run minutes.
AlexTMjugador added a commit that referenced this issue Jul 25, 2023
… and more

As commented in issues #444 and #518, there is some user interest for
distributing binaries for each unstable commit, and target ARM64
platforms. Personally, I think both suggestions are useful for the
project, as uploading binary artifacts for each commit might help
interested users to catch regressions and give feedback earlier, and
powerful ARM64 platforms are becoming increasingly popular due to some
cloud services (e.g., Amazon EC2, Azure VMs, Oracle Cloud) offering
cheaper plans for this hardware, in addition to the well-known push for
ARM by Apple with their custom M1 chips.

These changes make the CI target ARM64 as a first-class citizen. Because
the public GitHub actions runners can only be hosted on x64 for now, I
resorted to cross-compilation, [Debian's
multiarch](https://elinux.org/images/d/d8/Multiarch_and_Why_You_Should_Care-_Running%2C_Installing_and_Crossbuilding_With_Multiple_Architectures.pdf),
and QEMU to build, get ARM64 C library dependencies, and run tests,
respectively.

When the CI workflow finishes, a release CLI binary artifact is now
uploaded, which can be downloaded from the workflow run page on the
GitHub web interface.

In addition, these changes also introduce some cleanup and miscellaneous
improvements and changes to the CI workflow:

- Tests are run using [`nextest`](https://nexte.st/) instead of `cargo
  test`, which substantially speeds up their execution. (On my
  development workstation, `cargo test --release` takes around 10.67 s,
  while `cargo nextest run --release` takes around 6.02 s.)
- The dependencies on unmaintained `actions-rs` actions were dropped in
  favor of running Cargo commands directly, or using
  `giraffate/clippy-action` for pretty inline annotations for Clippy.
  This gets rid of the deprecation warnings for each workflow run.
- Most CI steps are run with a nightly Rust toolchain now, which allows
  to take advantage of the latest Clippy lints and codegen improvements.
  In my experience, when not relying on specific nightly features or
  compiler internals, Rust does a pretty good job at making it possible
  to rely on a rolling-release compiler for CI, as breakage is extremely
  rare and thus offset by the improved features.
- The MSRV check was moved to a separate job with less steps, so that it
  takes less of a toll on total workflow run minutes.
shssoichiro pushed a commit that referenced this issue Jul 31, 2023
… and more

As commented in issues #444 and #518, there is some user interest for
distributing binaries for each unstable commit, and target ARM64
platforms. Personally, I think both suggestions are useful for the
project, as uploading binary artifacts for each commit might help
interested users to catch regressions and give feedback earlier, and
powerful ARM64 platforms are becoming increasingly popular due to some
cloud services (e.g., Amazon EC2, Azure VMs, Oracle Cloud) offering
cheaper plans for this hardware, in addition to the well-known push for
ARM by Apple with their custom M1 chips.

These changes make the CI target ARM64 as a first-class citizen. Because
the public GitHub actions runners can only be hosted on x64 for now, I
resorted to cross-compilation, [Debian's
multiarch](https://elinux.org/images/d/d8/Multiarch_and_Why_You_Should_Care-_Running%2C_Installing_and_Crossbuilding_With_Multiple_Architectures.pdf),
and QEMU to build, get ARM64 C library dependencies, and run tests,
respectively.

When the CI workflow finishes, a release CLI binary artifact is now
uploaded, which can be downloaded from the workflow run page on the
GitHub web interface.

In addition, these changes also introduce some cleanup and miscellaneous
improvements and changes to the CI workflow:

- Tests are run using [`nextest`](https://nexte.st/) instead of `cargo
  test`, which substantially speeds up their execution. (On my
  development workstation, `cargo test --release` takes around 10.67 s,
  while `cargo nextest run --release` takes around 6.02 s.)
- The dependencies on unmaintained `actions-rs` actions were dropped in
  favor of running Cargo commands directly, or using
  `giraffate/clippy-action` for pretty inline annotations for Clippy.
  This gets rid of the deprecation warnings for each workflow run.
- Most CI steps are run with a nightly Rust toolchain now, which allows
  to take advantage of the latest Clippy lints and codegen improvements.
  In my experience, when not relying on specific nightly features or
  compiler internals, Rust does a pretty good job at making it possible
  to rely on a rolling-release compiler for CI, as breakage is extremely
  rare and thus offset by the improved features.
- The MSRV check was moved to a separate job with less steps, so that it
  takes less of a toll on total workflow run minutes.
shssoichiro added a commit that referenced this issue Sep 2, 2023
… and more (#534)

As commented in issues #444
and #518, there is some user
interest for distributing binaries for each unstable commit, and target
ARM64 platforms. Personally, I think both suggestions are useful for the
project, as uploading binary artifacts for each commit might help
interested users to catch regressions and give feedback earlier, and
powerful ARM64 platforms are becoming increasingly popular due to some
cloud services (e.g., Amazon EC2, Azure VMs, Oracle Cloud) offering
cheaper plans for this hardware, in addition to the well-known push for
ARM by Apple with their custom M1 chips.

These changes make the CI target ARM64 as a first-class citizen. Because
the public GitHub actions runners can only be hosted on x64 for now, I
resorted to cross-compilation, [Debian's
multiarch](https://elinux.org/images/d/d8/Multiarch_and_Why_You_Should_Care-_Running%2C_Installing_and_Crossbuilding_With_Multiple_Architectures.pdf),
and QEMU to build, get ARM64 C library dependencies, and run tests,
respectively.

When the CI workflow finishes, a release CLI binary artifact is now
uploaded, which can be downloaded from the workflow run page on the
GitHub web interface.

In addition, these changes also introduce some cleanup and miscellaneous
improvements and changes to the CI workflow:

- Tests are run using [`nextest`](https://nexte.st/) instead of `cargo
test`, which substantially speeds up their execution. (On my development
workstation, `cargo test --release` takes around 10.67 s, while `cargo
nextest run --release` takes around 6.02 s.)
- The dependencies on unmaintained `actions-rs` actions were dropped in
favor of running Cargo commands directly, or using
`giraffate/clippy-action` for pretty inline annotations for Clippy. This
gets rid of the deprecation warnings for each workflow run.
- Most CI steps are run with a nightly Rust toolchain now, which allows
to take advantage of the latest Clippy lints and codegen improvements.
In my experience, when not relying on specific nightly features or
compiler internals, Rust does a pretty good job at making it possible to
rely on a rolling-release compiler for CI, as breakage is extremely rare
and thus offset by the improved features.
- The MSRV check was moved to a separate job with less steps, so that it
takes less of a toll on total workflow run minutes.

## Pending tasks

- [x] Generate universal macOS binaries with `lipo` (i.e., containing
both `aarch64` and `x64` code)
- [x] Tirelessly fix the stupid errors that tend to happen when
deploying a new CI workflow for the first time
- [x] Think what to do with the `deploy.yml` workflow. Should it fetch
artifacts from the CI job instead of building them again?
- [x] Maybe bring back 32-bit Windows binaries. Are they actually useful
for somebody, or just a way to remember the good old days?

---------

Co-authored-by: Josh Holmer <[email protected]>
@andrews05
Copy link
Collaborator

Now released in v9.

Pr0methean pushed a commit to Pr0methean/oxipng that referenced this issue Dec 1, 2023
… and more (shssoichiro#534)

As commented in issues shssoichiro#444
and shssoichiro#518, there is some user
interest for distributing binaries for each unstable commit, and target
ARM64 platforms. Personally, I think both suggestions are useful for the
project, as uploading binary artifacts for each commit might help
interested users to catch regressions and give feedback earlier, and
powerful ARM64 platforms are becoming increasingly popular due to some
cloud services (e.g., Amazon EC2, Azure VMs, Oracle Cloud) offering
cheaper plans for this hardware, in addition to the well-known push for
ARM by Apple with their custom M1 chips.

These changes make the CI target ARM64 as a first-class citizen. Because
the public GitHub actions runners can only be hosted on x64 for now, I
resorted to cross-compilation, [Debian's
multiarch](https://elinux.org/images/d/d8/Multiarch_and_Why_You_Should_Care-_Running%2C_Installing_and_Crossbuilding_With_Multiple_Architectures.pdf),
and QEMU to build, get ARM64 C library dependencies, and run tests,
respectively.

When the CI workflow finishes, a release CLI binary artifact is now
uploaded, which can be downloaded from the workflow run page on the
GitHub web interface.

In addition, these changes also introduce some cleanup and miscellaneous
improvements and changes to the CI workflow:

- Tests are run using [`nextest`](https://nexte.st/) instead of `cargo
test`, which substantially speeds up their execution. (On my development
workstation, `cargo test --release` takes around 10.67 s, while `cargo
nextest run --release` takes around 6.02 s.)
- The dependencies on unmaintained `actions-rs` actions were dropped in
favor of running Cargo commands directly, or using
`giraffate/clippy-action` for pretty inline annotations for Clippy. This
gets rid of the deprecation warnings for each workflow run.
- Most CI steps are run with a nightly Rust toolchain now, which allows
to take advantage of the latest Clippy lints and codegen improvements.
In my experience, when not relying on specific nightly features or
compiler internals, Rust does a pretty good job at making it possible to
rely on a rolling-release compiler for CI, as breakage is extremely rare
and thus offset by the improved features.
- The MSRV check was moved to a separate job with less steps, so that it
takes less of a toll on total workflow run minutes.

- [x] Generate universal macOS binaries with `lipo` (i.e., containing
both `aarch64` and `x64` code)
- [x] Tirelessly fix the stupid errors that tend to happen when
deploying a new CI workflow for the first time
- [x] Think what to do with the `deploy.yml` workflow. Should it fetch
artifacts from the CI job instead of building them again?
- [x] Maybe bring back 32-bit Windows binaries. Are they actually useful
for somebody, or just a way to remember the good old days?

---------

Co-authored-by: Josh Holmer <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

9 participants