Vips warnings and image artifacts at moderate concurrency

[willtrking]

Hey there, been running into issues while working on a image server that uses bimg to perform jpeg -> webp conversion and scaling

At even moderate concurrency (20+ concurrent requests) in artificial tests, I've been getting tons of vips warning: VipsJpeg in my output, causing significant amounts of requests (50%+) to fail. If I don't reject images where bimg.Resize throws an error, I see significant artifacts in browser while tests are running.

The conversion funnel is simple, downloading an image from s3 (confirmed that warnings are NOT caused by incomplete downloads) and then converting the byte slices as follows

finalBuf, finalBufErr := bimg.Resize(respBuf, bimg.Options{
	Type:        bimg.WEBP,
	Width: 750,
	Quality:     40,
	Compression: 6,
})

I've also tried a simple jpeg scaling without webp conversion, and run into the same issue, code for that is simply

finalBuf, finalBufErr := bimg.Resize(respBuf, bimg.Options{
	Width: 750,
})

That converter is wrapped up in a simple fasthttp handler, no crazy logic yet as project is in early stages and I'm just testing out different image converters/resizers.

Server is running on an EC2 c4.8xlarge, so 36 "cores" and 60 GB of memory.

Should also be noted that this happens regardless of what I set runtime.GOMAXPROCS to, and I'm not fiddling with any of the vips concurrency settings.

Should also be noted that https://github.com/DAddYE/vips does not run into any issues, although is definitely slower (although hard to judge exactly given the error rate)

Love the library overall, been using it extensively in other areas, but the issues I see here make it difficult for me to use for this project.

EDIT:

Vips version info:
vips-8.4.5-Fri Mar 10 07:17:47 UTC 2017

uname -a output:
Linux ip-10-0-1-249 4.4.41-36.55.amzn1.x86_64 #1 SMP Wed Jan 18 01:03:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

EDIT EDIT:

Did more testing, discovered issue actually arises reliably at 20 concurrent requests (originally tested 50-100)

[phpb-com]

Have you tried setting VIPS_CONCURRENCY environment variable to 50 or so? Not sure if it will help but worth trying.

[willtrking]

Tried it out, setting VIPS_CONCURRENCY to anything other then 1 produced the following message in addition to the original warning:

VipsRegion: valid clipped to nothing

[h2non]

Unfortunately, this is something I've experienced in the past too. I'm still not sure the root of the issue (but I didn't dig into details yet). Any help here is always welcome!

[touhonoob]

Hi @willtrking
Could you provide a sample application which can reproduce the issue?
warning: VipsJpeg might not be the root cause of failed requests according to some discussions about libvips.

• Integrate with forthcoming libvips v8.5.0 logging improvements lovell/sharp#607
• https://github.com/jcupitt/libvips/issues/544

[jcupitt]

Hello, this should be fixed by https://github.com/jcupitt/libvips/issues/639 , which is released as libvips 8.5.4. I'm sorry this has been causing problems. There's a new thing in the test suite which hammers on exactly this error, so it shouldn't be able to regress again.

There was some stuff about this in the release notes for 8.5:

https://jcupitt.github.io/libvips/2017/03/16/What's-new-in-8.5.html#new-sequential-mode

There have been a couple of small problems with the new system, but it does seem to be working now, and it's a good improvement over the old mechanism.

[willtrking]

Hmm, 8.5.4 doesn't seem to fix it for me. I'm using the tarball release from https://github.com/jcupitt/libvips/releases. Still getting the following logs, as well as image artifacts.

(bimg:39): VIPS-WARNING **: 
(bimg:39): VIPS-WARNING **: read gave 68 warnings

When I serialize bimg (vips) with a simple mutex, there are no issues.

EDIT: Sorry about the close/reopen. Hit the wrong button

[kirillDanshin]

hmm. I think I know how to fix it. pending analysis.

[jcupitt]

@willtrking oh dear, then I guess it's a problem in the bimg layer. Could it be auto-rotate? You need to render via a memory buffer for this.

There's a page on how to resize with libvips:

https://github.com/jcupitt/libvips/wiki/HOWTO----Image-shrinking

for reference.

[willtrking]

@jcupitt Sorry about delay here, had to work on other projects for work so this took a back seat. Not doing any autorotate stuff (to my knowledge), to be specific I'm always running a crop/resize of a jpeg (the end width/height does vary, although the originating width/height is always within a few pixels of the same value) and converting the output to a lossy webp.

Turned out the mutex was probably hurting more then it was helping, so I've removed it.

Running now with libvips 8.5.5 , I've managed to tame down the amount of warnings I'm getting by disabling orc and setting VIPS_CONCURRENCY to my CPU count (in my case 8, I'm running a couple of AWS m4.2xlarge instances). Within the context of bimg I'm doing the following once when my server starts:

os.Setenv("VIPS_CONCURRENCY", strconv.Itoa(runtime.NumCPU()))
bimg.Initialize()

Adding os.Setenv("VIPS_TRACE", "true") seems to properly start tracing, so I'm assuming VIPS_CONCURRENCY is also properly being set.

I'm still getting the following warnings, although I haven't seen any artifacts:

(bimg:39): VIPS-WARNING **: read gave 10 warnings

(bimg:39): VIPS-WARNING **: 

(bimg:39): VIPS-WARNING **: read gave 2 warnings

(bimg:39): VIPS-WARNING **: 

When the server produces warnings as above, there's no guarantee that artifacts will be produced.

One thing worth mentioning is that I was incorrect in my initial issue, https://github.com/DAddYE/vips does seem to produce the same artifacts / warnings, although reproduction may be under different levels of concurrency from bimg.

Very willing to help contribute to a fix if indeed bimg is the culprit, although I'm at a bit of a loss in terms of starting points. Also if it helps I can send a VIPS_TRACE output. I think one of the most beneficial things would be to have an option that allows bimg to return an error if Resize causes a vips warning, which if there's interest I'd be willing to contribute toward. This way any warnings could either fail a request, or retry the Resize

[jcupitt]

Hi again, libvips is being used at very high levels of concurrency without issue in several other projects, so while it could be a libvips issue I suspect it is probably something about the way bimg is using it. A retry on warning option seems to be ducking the problem: it would be better to fix it.

I'm afraid I've never used Go so I'm not clear about setting up a dev environment. If you could write a very brief idiot's guide for ubuntu and point me at a test-case for this issue, I'll have a go at reading the bimg source and looking for problems.

[vansante]

Did anyone manage to resolve this? I'm getting image artifacts randomly on images I am resizing whenever I generate more than 1 at the same time. It seems to happen more often for high resolution large image.