{"id":4991,"date":"2017-01-23T10:00:37","date_gmt":"2017-01-23T15:00:37","guid":{"rendered":"https:\/\/pyimagesearch.com\/?p=4991"},"modified":"2021-04-17T14:47:18","modified_gmt":"2021-04-17T18:47:18","slug":"seam-carving-with-opencv-python-and-scikit-image","status":"publish","type":"post","link":"https:\/\/pyimagesearch.com\/2017\/01\/23\/seam-carving-with-opencv-python-and-scikit-image\/","title":{"rendered":"Seam carving with OpenCV, Python, and scikit-image"},"content":{"rendered":"

\"seam_carving_animation\"<\/p>\n

Easily one of my all-time favorite papers in computer vision literature is Seam Carving for Content-Aware Image Resizing<\/a><\/em> by Avidan and Shamir from Mitsubishi Electric Research Labs (MERL).<\/p>\n

Originally published in the SIGGRAPH 2007 proceedings, I read this paper for the first time during my computational photography class as an undergraduate student.<\/p>\n

This paper, along with the demo video from the authors<\/a>, made the algorithm feel like magic<\/em><\/strong>, especially to a student who was just getting his feet wet in the world of computer vision and image processing.<\/p>\n

The seam carving algorithm works by finding connected pixels called seams<\/em> with low energy <\/em>(i.e., least important) that traverse the entire image from left-to-right or top-to-bottom.<\/p>\n

These seams are then removed from the original image, allowing us to resize the image while preserving the most salient regions (the original algorithm also supports adding seams<\/em>, allowing us to increase<\/em> the image size as well).<\/p>\n

In the remainder of today’s blog post I’ll discuss the seam carving algorithm, how it works, and how to apply seam carving using Python, OpenCV, and sickit-image.<\/p>\n

To learn more about this classic computer vision algorithm, just keep reading!<\/strong><\/em><\/p>\n

\n
\n \"\"\n

Looking for the source code to this post?<\/h4>\n Jump Right To The Downloads Section <\/path><\/svg><\/a>\n <\/div>\n<\/div>\n<\/div>\n

Seam carving with OpenCV, Python, and scikit-image<\/h2>\n

The first part of this blog post will discuss what<\/em> the seam carving algorithm is and why<\/em> we may prefer to use it over traditional resizing methods.<\/p>\n

From there I’ll demonstrate how to use seam carving using OpenCV, Python, and scikit-image.<\/p>\n

Finally, I’ll wrap up this tutorial by providing a demonstration of the seam carving algorithm in action.<\/p>\n

The seam carving algorithm<\/h3>\n

Introduced by Avidan and Shimar in 2007, the seam carving algorithm is used to resize (both downsample and upsample) an image by removing\/adding seams<\/em> that have low energy<\/em>.<\/p>\n

Seams are defined as connected pixels that flow from left-to-right or top-to-bottom provided that they traverse the entire<\/em> width\/height of the image.<\/p>\n

Thus, in order to perform seam carving we need two important inputs:<\/p>\n

    \n
  1. The original image.<\/strong> This is the input image that we want to resize.<\/li>\n
  2. The energy map.<\/strong> We derive the energy map from the original image. The energy map should represent the most salient regions of the image. Typically, this is either the gradient magnitude representation (i.e., output of Sobel, Scharr, etc. operators), entropy maps, or saliency maps.<\/li>\n<\/ol>\n

    For example, let’s take a look at the following image:<\/p>\n

    \"Figure
    Figure 1:<\/strong> Our input image to the seam carving algorithm [source: Wikipedia<\/a>].<\/figcaption><\/figure>Using this image as an input, we can compute the gradient magnitude to serve as our energy map:<\/p>\n

    \"Figure
    Figure 2:<\/strong> Computing the gradient magnitude representation of of the input image. This representation will serve as our energy map [source: Wikipedia<\/a>].<\/figcaption><\/figure>Given our energy map we can then generate a set of seams that either span the image from left-to-right or top-to-bottom:<\/p>\n

    \"Figure
    Figure 3:<\/strong> Generating seams from the energy map. Low energy seams can be removed\/duplicated to perform the actual resizing [source: Wikipedia<\/a>].<\/figcaption><\/figure>These seams are efficiently computed via dynamic programming<\/a> and are sorted by their energy. Seams with low energy <\/em>are placed at the front of the list while high energy seams<\/em> are placed at the back of the list.<\/p>\n

    To resize an image we either remove seams<\/em> with low energy to downsample an image or we duplicate seams<\/em> with low energy to upsample the image.<\/p>\n

    Below is an example of taking the original image, finding the seams with the lowest energy, and then removing them to reduce the final size of the output image:<\/p>\n

    \"Figure
    Figure 4:<\/strong> Removing low energy seams from an image using the seam carving algorithm [source: Wikipedia<\/a>].<\/figcaption><\/figure>For more information on the seam carving algorithm, please see the original publication<\/a>.<\/p>\n

    Why use traditional seam carving over traditional resizing?<\/h3>\n

    Keep in mind that the purpose of seam carving is to preserve the most salient (i.e., “interesting”) regions of an image while still resizing the image itself.<\/p>\n

    Using traditional methods for resizing changes the dimensions of the entire<\/em> image — no care is taken to determine what<\/em> part of the image is most or least important.<\/p>\n

    Seam carving instead applies heuristics\/path finding derived from the energy map to determine which<\/em> regions of the image can be removed\/duplicated to ensure (1) all “interesting” regions of the image are preserved and (2) this is done in an aesthetically pleasing way.<\/p>\n

    Note:<\/strong> Preserving the most interesting regions of an image in an aesthetically pleasing manner is a lot harder than it sounds. While seam carving may seem like magic, it’s actually not — and it has its limitations. See the “Summary” section for more information on these limitations.<\/em><\/p>\n

    To compare traditional resizing versus seam carving, consider the following input image:<\/p>\n

    \"Figure
    Figure 5:<\/strong> An example image to resize.<\/figcaption><\/figure>\n

    This image has a width of 600 pixels and I would like to resize it to approximately 500 pixels.<\/p>\n

    Using traditional interpolation methods my resized image would look like this:<\/p>\n

    \"Figure
    Figure 6:<\/strong> Resizing an image using traditional interpolation techniques. Notice how the height changes along with the width to retain the aspect aspect ratio.<\/figcaption><\/figure>\n

    However, by applying seam carving I can “shrink” the image along the horizontal dimension and still preserve the most interesting regions of the image without <\/em>changing the image height:<\/p>\n

    \"Figure
    Figure 7:<\/strong> Resizing the image using seam carving.<\/figcaption><\/figure>\n

    Utilizing seam carving in computer vision and image processing<\/h3>\n

    In this section I’ll demonstrate how to use seam carving with OpenCV, Python, and scikit-image.<\/p>\n

    I’ll assume you already have OpenCV installed on your system — if not, please refer to this page<\/a> where I provided resources to install OpenCV on many different operating systems.<\/p>\n

    From there, you should ensure you have scikit-image<\/a> installed as well. This page<\/a> provides more information on installing scikit-image, but for most systems you can use pip<\/code> :<\/p>\n\n\n

    $ pip install --upgrade scikit-image\n<\/pre>\n\n\n
    Let’s go ahead and see how we can apply seam carving to our own images.<\/p>\n
    Open up a new file, name it seam_carving.py<\/code> , and insert the following code:<\/p>\n\n\n
    # import the necessary packages\nfrom skimage import transform\nfrom skimage import filters\nimport argparse\nimport cv2\n\n# construct the argument parse and parse the arguments\nap = argparse.ArgumentParser()\nap.add_argument(\"-i\", \"--image\", required=True,\n\thelp=\"path to input image file\")\nap.add_argument(\"-d\", \"--direction\", type=str,\n\tdefault=\"vertical\", help=\"seam removal direction\")\nargs = vars(ap.parse_args())\n<\/pre>\n\n\n
    Lines 2-5<\/strong> import our required Python packages while Lines 8-13<\/strong> parse our command line arguments. This script will require one argument followed by a second optional one:<\/p>\n