Image compresses better with duplicate entries kept in palette

[Rainboom-Dash]

I'm getting increased file sizes with the latest commits that I built on Sept 4th versus the latest stable release
I've tested the default libdeflate compression and zopfli and both are increasing file size considerably

This is not the only image that is getting increased file sizes but it is one of the more significant ones that has a few percent increase (maybe because the file size is smaller to begin with)
On average it seems the latest commits decrease file size slightly... but there are some regressions
I thought maybe it was just the libdeflate since the version was changed but it seems zopfli is affected as well...


The image I tested with has had optipng run on it (the site I downloaded from automatically does that)

[andrews05]

Hi @Rainboom-Dash, thanks for posting this.
This is quite interesting. The image has many duplicate entries in the palette which somehow are helping it to compress better. The current code assumes that duplicate entries are bad and will always remove them to reduce the palette size. Some investigation will be needed to understand this better and see how we can improve it, but in the meantime you can avoid palette reduction by using the --np flag.

@ace-dent Do you know anything about deliberate duplicates in a palette like this? I'm pretty sure optipng wouldn't do that.

[ace-dent]

@Rainboom-Dash - very interesting file. Where did it come from and do you know what software produced it?
@andrews05 - never seen this before... will investigate! Metadata suggests software called GLDPNG ...? There is always the potential that the input palette is more optimal; so perhaps a --np pass should always be run for the most extreme settings?... not sure if some smarter detection is possible...?

The pngthermal image shows the surprisingly very low entropy achieved across the whole image...

[Rainboom-Dash]

I downloaded from https://derpibooru.org/images/3198446 which was taken from https://www.deviantart.com/boxwari/art/Flurry-Musicante-981750954 which I just now realized I probably should of downloaded from Deviantart just to get closer to the source

It does seem like --np gets a few percent better compression here

[TPS]

FWICT, this is 2 separate issues now:

• current stable compresses this better than binary from latest commit

• GLDPNG compresses this better than 1 or both of those w/o --np Irrelevant per Image compresses better with duplicate entries kept in palette #553 (comment)

• Other tools compress this better than either of those

Did I misunderstand?

Also, does the observation of the duplicate palette entries corresponding to allowing better compression make sense, or is that a potential correlation/causation error?

[ace-dent]

@andrews05 - I wonder if the edge preference heuristic I suggested, is failing here...?

[ace-dent]

Attached version produced with other tools. Reduced palette and smaller IDAT size.

[andrews05]

@ace-dent The target IDAT size is 47918, as achieved with --np (or v8).
Disabling the edge heuristic in oxipng made little difference.

I took a quick look at the GLDPNG source. It's definitely not an optimiser or doing anything interesting with the palette, so either something else is responsible for this or it's just accidental (entirely possible).

@TPS The structure/order of palette can have a big impact on line filtering. It's not unthinkable that some weird optimisation might be achieved by spreading out the colours like this, it's just not something I've seen or considered before. Nor can I imagine how one might implement such a strategy.

@Rainboom-Dash Would you be able to post a couple more samples that have regressed like this?

[TPS]

@andrews05 Since @ace-dent has found tools that solve this sensibly (optimized reduced pallette & IDAT), IMHO, not worth pursuing this weird "ordering," but rather, whatever those tools are doing.

[Rainboom-Dash]

@andrews05 I believe the other losses are due to libdeflate and it's usually around 0.1% file size increase or less... and usually it has decreased file size not increased... it's just sometimes it increases... but yeah, it's pretty small and I think that will just happen when the algorithm changes..

[andrews05]

@Rainboom-Dash Good to know. This one remains a unique mystery then.

I've just posted a PR to resolve this by keeping the original palette, same as it did in v8. But I think we should keep the issue open for further investigation.

[TPS]

@andrews05 Per @ace-dent's note above, this title change doesn't represent the current best for this image, just the weird quirky original? So maybe analyze the intermediate & try to meet/beat that? 🙇🏾‍♂️

@ace-dent It'd probably really help if you'd post what toolchain you used to get your result. 🙇🏾‍♂️

[andrews05]

@TPS If you check the image @ace-dent posted, it's larger than the result of both oxipng v8 and oxipng master, whose sizes are shown in the first post. So I'm not quite sure what it's demonstrating, unless he posted the wrong image?

[TPS]

Attached version produced with other tools. Reduced palette and smaller IDAT size.

@ace-dent Did you post the correct version of the image?