Support for stitching multi channel images (e.g. as numpy arrays instead of filenames)

[Nelson-Gon]

Hi,

Thanks for this package. I was trying it out and noticed that currently Stitcher.stich() only supports a list of filenames.

I was wondering whether it could be possible to support a list of NumPy arrays instead, for instance. This would be useful for example in cases where you only want to stitch specific channels of different images.

Currently, it also seems to fail with multiple channel images with the error:

...
---> 53             raise StitchingError("No match exceeds the " "given confidence threshold.")
     54 
     55         return indices

StitchingError: No match exceeds the given confidence threshold.

Thank you and apologies if you've tackled this issue already elsewhere.
NelsonGon

[lukasalexanderweber]

Can you give me some more details on your use case? How many bands do the images to bei stitched have?

Having the "raw" numpy arrays in memory as Input can leed to memory issues on high resolution Images, thats why its not supported (yet)

(Sorry for closing, that was by mistake)

[lukasalexanderweber]

Can you give me some more details on your use case? How many bands do the images to bei stitched have?

any updates?

[Nelson-Gon]

Hi sorry,

For example if each image has the shape (channels, height, width), would it be possible to stitch together each of the first channels. So:

Stitcher.stitch([x[0] for x in multi_channel_images])

or

Stitch.stitch([imread(x)[0] for x in image_files])

or maybe some channel flag to specify which channels to stitch?

[lukasalexanderweber]

On which channels do you want to perform the image registration (Feature Detection, Matching)? Do you have an example dataset?

I think that we should be able to utilize the obtained cameras to stitch all channels together.

I don't think that we can perform the image registration part on more or less than three bands (but we need to check this).

[Nelson-Gon]

I unfortunately do not have a sample dataset right now but will look into creating one. For the registration, I think it should be possible since we can register two images with (height, width) independently, at least in packages like skimage. I think the same should be possible with opencv.

[shuckerino]

Hello, I just found this package and had also the same question.

Currently, only filenames are valid as for the input of the stitcher. In my use case, I directly read in frames from two webcams and want to pass these frames into the stitcher. Because of that, my frames have no filename or whatsoever. I am fairly new to programming and am now trying to implement this feature by myself, but it would be great if it was an official feature.

Maybe an important question for me would be, if this package is even suitable to perform image stitching for multiple images (later I am aiming for 4 cameras) in real time with decent fps. Would be happy to get some feedback if this would be even possible.

[lukasalexanderweber]

read in frames from two webcams and want to pass these frames into the stitcher

Yes this could be an interesting feature, so you wouldn't need to store the data into files and reload them during the stitching

Maybe an important question for me would be, if this package is even suitable to perform image stitching for multiple images (later I am aiming for 4 cameras) in real time with decent fps. Would be happy to get some feedback if this would be even possible.

Please open a new discussion and provide a sketch with your planed setup of the webcams.

[lchunleo]

i also have the same question of instead of passing files names, but just two frame of numpy array to be ready by the Stitcher. nothing complicated, just numpy of the image instead of file names. Wish to know how to do a temp rectification of this ?

[lukasalexanderweber]

This is supported since https://github.com/OpenStitching/stitching/releases/tag/v0.5.0

[pmb399]

Hello, I am also interested in stitching multi-channel images provided as 3d numpy arrays. I shared some code on StackOverflow how I process a single channel of 2d data that is provided as numpy array, but wonder how feature matching could be achieved between two or more images with multiple channels, where the matcher bundles all hits from one image to another while mapping each channel to its respective counterpart.

I uploaded some test data as 3d numpy arrays in the community image repo which can be loaded with np.load(). The first two dimensions are spatial, the third coordinate are the channels. I chose to include 4 different channels in the data file. Maybe this can help to resume discussion.

[lukasalexanderweber]

hey @pmb399

thanks for uploading the data. I have problems reading it. The images are really small: (70, 59, 4)

I extracted the first three bands for both and it looks like this:

Can you provide code how you load and display the data or update the PR in the images repo if the image data is wrong?

[pmb399]

Hi @lukasalexanderweber

The provided data is actually a mesh scan over a motorized (x,y) stage collected pixel-by-pixel which is why the resulting images are so small. For openCV feature identification and stitching, I tend to resize the data by some factor (fx=fy=3 generally works well, but in this case fx=fy=6 seems better) which appears to aid feature identification.

As for loading, the number of bands depends on user input during data collection. Since sometimes I may only deal with one band, I prefer to convert the data to RGB with a colour mapper. For image processing, the intensities don't really matter, as long as they are scaled back after stitching to their respective original value - so I try to maximize the full scale of the mapper. Code is below:

from matplotlib.colors import LinearSegmentedColormap
from matplotlib._cm import _jet_data
from matplotlib import cm
import numpy as np
import cv2 as cv

# Load data
img1 = np.load('Img1_4channels.npy')
img2 = np.load('Img2_4channels.npy')

rois_img1 = list()
rois_img2 = list()
scaling_imgs1 = dict()
scaling_imgs2 = dict()

# Extract regions of interest

for roi in range(img1.shape[2]):
    rois_img1.append(img1[:,:,roi])
    scaling_imgs1[roi] = np.max(img1[:,:,roi])
for roi in range(img2.shape[2]):
    rois_img2.append(img2[:,:,roi])
    scaling_imgs2[roi] = np.max(img2[:,:,roi])

# Get maximum counts, so that we can scale all ROI slices (ROIs are independent)
scale_max = dict()
for i in range(img1.shape[2]):
    scale_max[i] = max(scaling_imgs1[i],scaling_imgs2[i])
    
# Convert each ROI (slice) to false-colour map
# i.e. expand 2d array of size (m,n) to 3d array of size (m,n,3)

# First, Use colour mapper with high N value to maintain data integrity
mymap = cm.get_cmap(LinearSegmentedColormap('myjet',_jet_data,N=10000))

# Second, apply colour map to data - don't forget to flip data to get correctly displayed CV images
# Also, remove the alpha channel and sort RGB / BGA order
# Return as bytes

false_color_imgs1 = list()
false_color_imgs2 = list()

for idx,roi_img in enumerate(rois_img1):
    img = mymap(np.flipud(roi_img/scale_max[idx]),bytes=True)[:,:,[2,1,0]]
    
    # Data size generally relatively small, resize and increase images
    # Factor 3 typically works well, enlarge to 6x even better here
    false_color_imgs1.append(cv.resize(img, (0,0), fx=6, fy=6, interpolation=cv.INTER_NEAREST))
    
for idx,roi_img in enumerate(rois_img2):
    img = mymap(np.flipud(roi_img/scale_max[idx]),bytes=True)[:,:,[2,1,0]]
    
    # Data size generally relatively small, resize and increase images
    # Factor 3 typically works well, enlarge to 6x even better here
    false_color_imgs2.append(cv.resize(img, (0,0), fx=6, fy=6, interpolation=cv.INTER_NEAREST))
    
# Show the data
for idx, img in enumerate(false_color_imgs1):
    cv.imshow(f'Img 1 - ROI {idx+1}',img)
    cv.waitKey(0)
    
for idx, img in enumerate(false_color_imgs2):
    cv.imshow(f'Img 2 - ROI {idx+1}',img)
    cv.waitKey(0)

[lukasalexanderweber]

thanks for the code. I might need some time till I'm able to work on this.
The workflow I'd suggest would be that a user can define three bands (which could also be the same band if there is only one) with which the registration part is done (the cameras are obtained). You might also want to reuse the obtained seam masks (#217). Then each band is stitched by itself, using an rgb image of one single band (b1, b1. b1). After all bands are stitched, you put together the bands ob the various results. So you have stitched(b1. b1, b1) till stitched(bn, bn, bn) you create stitched(b1...bn) by extracting the first band of each individual result.