Name: Shuai Victor Zhou

SID: 3035912996

Part A: Image Warping and Mosaicing

Overview

In this part of the project, we identify homographies that map from a scene to another similar scene. By finding the correct map that correlates to our homography, we are able to warp images, allowing for the rectification of images as well as the creation of mosaics.

Shoot the Pictures

By fixing the center of projection (COP), I took the following photos to create mosaics with.

Recover Homographies

The homography between two images can be represented by a 3x3 homography matrix, H. This matrix is the mapping such that

for corresponding points p and p’ on the first and second images, respectively. H takes on the form

To find these values, we’re looking to solve a system of equations that maps to

for some weight w on each one of these points. We’re able to solve this system using just 4 distinct points, but this can make our results prone to a lot of noise. To help with this, we use more points to create an overdetermined system and solve it using least-squares.

Warp the Images

Once we have the homography matrix H, we can begin to warp an image. We do this by first taking the corners of our original image and mapping all of them all using H. Since this mapping is a homography, we know that for an image corners a, b, c, and d, all pixels of the mapped image will be contained within the polygon described by the corners Ha, Hb, Hc, and Hd. Therefore, upon finding the new corners, we can simply apply H^{-1} on each of the points within the new polygon to map back to the original image and then populate the new image with their corresponding values.

Image Rectification

With the ability to warp images, we’re also able to rectify images by mapping images to new hand-picked points.

Blend the Images into a Mosaic

To generate a mosaic, we first take two images of a scene and their correspondences. We then take the average of their correspondences and warp both images to the averages correspondences. Next, we sum up these two images. Below are the results of this.

The overlap is clearly more highlighted due to the inherent nature of us simply summing up our two images. To account for this, we generate a mask for the overlap. Then, we apply the mask over our mosaic to retrieve elements within the overlapped region. We normalize this region only and then place it back into the mosaic. Finally, we renormalize the entire image again. This provides us with a weighted normalization of our image, and we produce the following results.

Conclusion

In this part of project 4, we were able warp images and create mosaics using a homography matrix. The math and logic behind this process seems to be close in relation to what was done for the CS180 project 3, where we conducted face morphings.

Part B: Feature Matching for Autostitching

Overview

In this part of the project, we create an auto-detection of points to use for the stitching together of images. These points are found using optimal Harris corners that are correctly mapped to as inliers.

Adaptive Non-Maximal Suppression (ANMS)

In this part, we first identify the Harris corners of the image. Then, for each corner, we find the closest other corner that is a decent amount “larger” than this current corner; we store the distance between them as the radius. Finally, we select the points with the largest radius.

Feature Descriptor Extraction

For each of the corners we obtained through ANMS, we select the 40x40 region around it. We then resize this region into an 8x8 region. For each region’s color channels, we normalize the color channel by subtracting each channel by its mean and dividing by its standard deviation. Then we flatten the image with all its channels into a 1x192 array, and we stack all of these feature arrays together. Below are some examples of the 8x8 regions.

Feature Matching

Now that we have features from each of the images that we’re stitching together, we wish to match the corresponding features to each other. This is done by calculating the pairwise sum of the square of the differences, finding the Lowe’s ratio through the nearest neighbor distances, and then finding finding all values below our Lowe’s threshold. We then acquire the indexes of points from each image that correspond to each other.

Random Sample Consensus (RANSAC)

In RANSAC, we run through a predetermined number of iterations. For each iteration, we sample 4 random pairs from the corresponding points that we find during feature matching. Using these 4 points, we generate a homography matrix H and then apply this H onto all of the source points (points that are being mapped to the target points). For each of these newly mapped points, we find out how many of the points are “close” to where they should actually be according to the feature matching (here, we required the points to be at most 3 pixels away from where they should be). For each homography matrix, the one that generates the greatest number of points that satisfy our conditions is used for our final mosaic, which we generate in the same way that we did for the first part of this project.

Automatically Stitched Mosaics

Below are our final results, with the manually stitches mosaics on the left and the automatically stitched mosaics on the right.