Content-aware image resizer
▶️ ️Demo- ✏️ How it works
- 📄 Seam carving paper by Shai Avidan and Ariel Shamir
Background image by Ian Dooley
Content-aware image resizing might be applied when it comes to changing the image proportions (i.e. reducing the width while keeping the height) and when losing some parts of the image is not desirable. Doing the straightforward image scaling in this case would distort the objects in it. To preserve the proportions of the object while changing the image proportions we may use the Seam Carving algorithm.
In the example below, you may see how the original image width was reduced by 50% using content-aware resizing (left image) and straightforward scaling (right image). In this particular case, the left image looks more natural since the proportions of the balloons were preserved.
The Seam Carving algorithm’s idea is to find the seam (continuous sequence of pixels) with the lowest contribution to the image content and then carve (remove) it. This process repeats over and over again until we get the required image width or height. In the example below, you may see that the hot air balloon pixels contribute more to the content of the image than the sky pixels. Thus, the sky pixels are being removed first.
Btw, finding the seam with the lowest energy is a pretty computationally expensive task (especially for large images). To make the seam search faster the dynamic programming approach might be applied.
The importance of each pixel (so-called pixel's energy) is being calculated based on its color (R, G, B) difference between two neighbor pixels. More detailed explanation with examples you may find in Content-aware image resizing in JavaScript blog post. Now, if we set the pixel energy to some really low level artificially (i.e. by drawing a mask on top of them), the Seam Carving algorithm would perform an object removal for us for free.
You may launch the JS IMAGE CARVER and play around with resizing of your custom images. You may also check its source-code.
Here are some more examples of how the algorithm copes with more complex backgrounds.
Mountains on the background are being shrunk smoothly without visible seams.
The same goes for the ocean waves. The algorithm preserved the wave structure without distorting the surfers.
But also we need to keep in mind that the Seam Carving algorithm is not a silver bullet, and it may fail to resize the images where most of the pixels are edges (looks important to the algorithm). In this case, it starts distorting even the important parts of the image. In the example below the content-aware image resizing looks pretty similar to a straightforward scaling since for the algorithm all the pixels look important, and it is hard for it to distinguish Van Gogh's face from the background.
- Downscaling of the images without distorting their content-intensive parts
- Objects removal during resizing
- Image upscaling
- Upscale the image to its original size after objects removal
- Real-time resizing
