Author: mshende

Glitter pieces which are measured off, predicted on, and the prediction follows what we would actually expect

There are quite a few pieces of glitter which are measured to have an intensity of 0 (or < 0.1 normalized) in the test image, but then predicted to have an intensity        > 0.6 normalized. If we look at the intersection of the scan lines which correspond to the intensity plots of this centroid, we see that they intersect right around where the light was displayed on the monitor for the test image. So we would expect to see this centroid in our test image.

 

 

 

 

 

 

 

Below, I have shown the centroid on the image (pink dot), and we can verify that we don't see a glitter piece lit there, nor are there any lit pieces nearby. If there was another lit piece close to this centroid, we may believe that we just matched the centroid found in the test image to the wrong centroid from our glitter characterization. This does not seem to be the case though, so it's a bit of a mystery.

UPDATE: we have ideas 🙂

1. threshold intensities the same way I did in my gaussian optimization (finding receptive fields) in the camera calibration error function
2. throw out gaussians that are "bad"..."bad" can mean low eigenvalue ratio or intensity plots not well-matched

Aperture vs. Exposure

I played around with the aperture of the camera, and took many pictures with varying aperture - as the f-number decreases (aperture becomes larger), we see that a lot more light is being let in. As this happens, it seems that the center of a glitter piece slowly shifts, probably because if the exposure remains the same for all aperture settings, some images will have saturated pixels. The image below shoes multiple apertures (f/1.8, f/2, f/2.2,  ..., f/9), where the exposure was kept the same in all of the images. We can see that in some of the smaller f-number images, there seem to be some saturated pixels.

 

 

 

 

 

 

 

 

After realizing that the exposure needed to be different as the aperture varies, I re-took the pictures with f-number f/1.8, f/4 and f/8, and varied the shutter speed and it looks like the shift happens less if image taken with some exposure and low f-number look similar in brightness to an image taken with some exposure and a higher f-number.

 

 

 

 

 

 

 

 

Next Steps

1. fix the intensity plots in the camera calibration and re-run
2. try throwing out weird-gaussian centroids and re-run calibration
3. take iphone pictures with the XR, figure out which exposure works best with its aperture (f/1.8) so that we get images that look similar to our camera images (f/8, 1 second exposure)

Goal: I am working on accurately calibrating a camera using a single image of glitter.

Paper Title: SparkleCalibration: Glitter Imaging Model for Single-Image Calibration ... maybe

Venue: CVPR 2020

For the last week, I have been specifically working on getting the calibration pipeline and error function up and running using the receptive fields of the glitter. Now, in the error function, I am predicting the intensity of each piece of glitter, and comparing this predicted intensity to the actual intensity (measured from the image we capture from the camera). I am also using a single receptive field for all of the glitter pieces instead of different receptive fields for each glitter piece, because we found that there were enough 'bad' receptive fields to throw off the normalization of the predicted intensities in the optimization.

 

 

 

 

 

 

 

 

 

This plot shows the predicted vs. measured intensities of all of the glitter pieces we "see" in our image (many have very low intensities since most glitter pieces we "see" are not actually lit. Here we see that there is a slightly noticeable trend along the diagonal as we expect to see. The red points are the glitter pieces which are incorrectly predicted to be off, the green points are the glitter pieces which are correctly predicted to be on, the black points are the glitter pieces which are incorrectly predicted to be on, and the rest are all of the other glitter pieces.

I also tried using the old on/off method for the error function (distance from the light location as the error function) and found that the results were quite a bit worse than the receptive field method (yay!)

Goal: My next tasks are the following:

• search for the right gaussian to use for all the glitter pieces as the receptive field
• run the checkerboard calibration for my currently physical setup
• re-take all images in base and shifted orientations, re-measure the physical setup, take 'white-square' test images in this setup, and maybe some iphone pictures

 

This past week, I have been working on solving for the gaussian which fits the intensity plots of my glitter pieces, and using this to solve for the center of the receptive field of the glitter. Below is an example of an intensity plot of a centroid:

 

 

 

 

 

 

 

 

 

 

Below are the equations I am using to write the optimization function to solve for the receptive field. Here I am computing the integral of the response of a piece of glitter at every location on the monitor, where I(x,y) is a binary of whether the monitor is lit or not at each location:

Right now, I am solving this using the circular gaussian with a single sigma, and I spent a few days working through writing the optimization function. Yesterday I was able to write it in a way that speeds it up (each iteration is looking at 200 frames and ~700 centroids, so it was very slow).

My next step is to work on analyzing the final error that I am getting and looking at the optimal solution for the center of the receptive field and the sigma value (I am getting a solution, but have not spent too much time trying to understand its validity).

This post is more of a re-cap of where we are in the pipeline, and a gut-check on what is left to be done. It takes about half a day in total to complete steps 1-5 end-to-end.

Pipeline so far:

1. Capture Pictures
   • four directions (horizontal, vertical, diagonal positive, diagonal negative) for a base location of the glitter and some moved location of the glitter (8 sets of images total)
2. Extract Centroids
   • using an intensity threshold (150) and some distance threshold (2) - 2 centroids are considered the same if they are less than the distance threshold away from each other. A cluster of pixels is considered a centroid if the pixels are all above the intensity threshold - do this for all 8 sets of images
3. Reduce Centroids
   • look at the intensity plots for each centroid, and any centroid with more than 1 intensity peak is thrown out (these could be pieces of glitter that are bent, or two pieces of glitter overlapping) - do this for all 8 sets of images
4. Match Centroids Across Directions
   • find the centroids that appear 'lit' in some frames in all 4 scan line directions - should be left with two sets of centroids, one for the base location of the glitter and one for the shifted location of the glitter
   • base: 30,805 centroids
   • shifted_up: 11,818 centroids
5. Screen Map
   • solve for the point on the screen which causes each centroid to be 'lit' using the intersection of the scan lines which produced the highest peak of intensity for each centroid
   • base: 29,327 centroids
   • shifted_up: 10,677 centroids
6. (TO DO) Measure Physical Setup
   • convert all screen locations and centroid locations to real-world 3D coordinates
7. (TO DO) Compute Surface Normals
   • most likely in the 'base' location of the glitter using real-world coordinates
8. (TO DO) Match Centroids Across Glitter Locations
   • should be similar to the process for matching across scan line directions
   • I should do this sooner rather than later because we need to see how many centroids survive this step and actually are seen in both sets of images
9. (TO DO) Surface Normal Error Analysis
   • using the surface normals from the base location of the glitter, estimate the screen map for the shifted centroids and compute the error to the actual screen map (or something like that)
10. ...more stuff...
11. (TO DO) CALIBRATION!

These images below show the screen mappings of a few centroids from the base location of the glitter:

This last image shows an example of a centroid that has a 'bad' intersection of lines. The threshold being used to determine if the point of intersection is 'bad' is whether it is within 2*gaussian standard deviation of the line is furthest from or not. There are also some cases where the point of intersection has a negative value (when either the horizontal or vertical lines are on the edge) - there are about 15 such centroids (not pictured)

 

I have been working to debug my centroid extraction and centroid matching code, and am not fully satisfied with what I have yet. That being said, I am getting there! Below are some images that we can talk through and I can try to explain why I think it is not 100% right yet:

Base max image in all four directions with the matching centroids:

https://www.dropbox.com/s/ywkkooqjg5lsyf2/base_maxImage_matching_centroids.png?dl=0

 

 

 

 

 

 

 

Shifted Up max image in all four direction with the matching centroids:

https://www.dropbox.com/s/eefqogfo7b6rore/shifted_up_maxImage_matching_centroids.png?dl=0

 

 

 

 

 

 

 

All 8 max images (all four directions from two glitter positions) with all centroids per directions as well as the matching centroids:

https://www.dropbox.com/s/sc8ahjwk9gdqe5s/allCentroids_vs_matchingCentroids.png?dl=0

For the past couple of weeks, I have been struggling through some image capture issues and, more recently, some monitor issues. After reducing the number of scan lines I am displaying in each direction and subsequently increasing the gap between each scan line, we have good-looking intensity plots! Below is a montage of ~100 centroids (single-pixel) found in the set of horizontal scan line images and plots of their raw intensities over time (all of the frames of horizontal scan lines).

 

 

 

 

 

 

 

 

 

 

Next steps:

1. I need to look at these intensities for actual centroids (not just single pixels), which means I also need to decide which threshold value to use for intensity (what constitutes as a 'lit' centroid). I'm currently thinking somewhere around ~150-200?
2. The peaks are still about 10 frames wide, which may wider than we want, so maybe more spread apart scan lines will address this (and lead to fewer pictures having to be taken each time)
3. Start writing code for screen mapping...

This week there were two things I was working on simultaneously - getting the homography pipeline working with our automated method of finding markers and starting to generate the images of scan lines that we will later use for glitter characterization.

Homographies

End-to-end I can now:

• take a picture of the glitter sheet with the markers attached (which have finally been permanently affixed to the four corners),
• find the four points associated with each of the four markers (16 points total),
• use these 16 points to compute the homography to map our image to a 3000x3000 image (I hand measured where the markers are on the glitter sheet and scaled these measurements up to 3000x3000)

Below is an example of an image taken in our setup of the glitter sheet, and the 3000x3000 image I get when I compute and apply the homography mapping the markers to the corners (also applied a slight zoom-out):

 

 

 

 

 

 

 

 

 

In order to test how accurate the homography is, I used the above image as a "base" image, found the four markers in the post-homography image (on the right), and then used these as control points to compute a new homography mapping a series of other images (tilted at various angles) to that base image. I then computed the distance between the markers in each of the post-homography images and the markers in the post-homography base image.

Below are the results for error calculations per marker over a series of images I took. Each image is at more and more of an angle w.r.t. the camera. For the first row, I compute the error if I compute and apply the homography from the base image to itself.

Image	Upper Left Marker	Upper Right Marker	Lower Right Marker	Lower Left Marker
Base	0.0548	0.0806	0.0036	0.0303
Tilted Back 1	0.3457	0.1485	0.1646	0.2493
Tilted Back 2	0.8675	0.5837	0.7798	0.4999
Tilted Back 3	0.8253	0.4140	0.3949	0.3547
Tilted Back 4	1.1967	0.5119	0.4784	0.2572
Tilted Back 5	N/A	N/A	N/A	N/A
Tilted Right 1	0.3293	0.2611	0.2982	0.4730
Tilted Right 2	N/A	N/A	N/A	N/A
Tilted Right 3	N/A	N/A	N/A	N/A
Tilted Right 4	N/A	N/A	N/A	N/A

Some images have N/A as the error for the markers - either the markers were not found in the original image and I was unable to apply the homography, or I was able to apply the homography, but the image post-homography was too blurry to find the markers and compute the error. Such images were at too much of an angle w.r.t. the camera, so we wouldn't use such images in our experiments anyway.

For example, the following images are results from Tilted Right 2. The four markers were correctly located in the original image, and so I was able to apply the homography to get the top-right image. However, when I tried to locate the markers in the post-homography image, only the top two markers were found (bottom image). The bottom two markers are not as in-focus and look a little more like blobs, so they were not detected.

 

Scan Lines

Generating the scan line images has been the other task I have spent some time on this week. In doing this, there are a couple of parameters we have to play with, I think the idea being that we go with the smallest size lines that are still 'bright enough'. I am generating images with lines of a given width, and then applying a gaussian filter to them. I am testing different line-widths as well as different gaussian standard deviations. Below are some examples images I have generated with 2 different line widths, and 4 different gaussians for each line width.

The next task will be to actually try taking some pictures of the glitter uses some of these scan lines, and look at those images to determine which line width-gaussian combination is best.

Camera

I have been having some trouble focusing this current Nikon camera, but I think the new camera will work better for this. Once the new camera comes in, I will actually test out some of these scan lines in order to figure out which set of parameters we want to use for our experiments. I also have started working through some of the centroid-detection code, and will test this once I am able to take proper, focused pictures of the glitter!

So I spent the first half of the week writing up my proposal to present a poster at Grace Hopper this year. Phew, that's done! On to more fun things...

I am working on trying to find the right pipeline to automatically detect markers on four corners of our glitter sheet in an effort to reduce the error in our homography pipeline. The most success I have had so far is with using CALTag markers.

This is an example of a tag - I can specify the size of the tag. In this case, this is a 4x4 marker, and each of the 16 unique QR code in this tag is an 8x8 bit code. I can generate 4 different such tags to be used as our markers, and place them in each of the four corners of our glitter sheet. The rest of this post will focus on some of my tests using these codes and some of the issues I have had with them.

Find each tag individually

If I look at an image of just the 1st tag, and try to find the tag in the image, you can see that all 25 points were found. Similarly, if I look at an image of just the 3rd tag, all 25 points on the tag were found.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

However, when I did this test on code 2 and code 4, it did not find any of the points in the marker.

Many iterations and failures later...

After many failed attempts with these markers, I finally succeeded (mostly). I changed the markers from 4x4 to 3x3 markers so that the individual quads would be larger. I then tried to take a picture with one marker alone in the middle of the glitter,and it failed. At this point I realized that the noise from the glitter was causing problems. So, I tried putting the markers on a piece of paper, more or less creating some sort of boundary, and it found the whole marker.

Finally, I arrived at this: I used a piece of cardboard that is exactly the same size as the glitter, and cut it down to look like a border.

(Sorry for the poor picture quality)

It is having a hard time finding marker 2 (the lower right one) no matter where I place it on the border, so I think there is something wrong with that particular marker - I can just print a new random one and try that. But, all of the other 3 markers were found correctly.

Next, I need to try this out in our actual real-world setup and see how it does with a little bit less light.

Question: Will a border like this remove too many pieces of glitter from our system, or will that be okay? This seems to be the only effective way that I can remove noise from the image.

Things to try:

• I will also try reducing the size of the individual markers and then can also make the border a little smaller, and see if the CALTag code can find smaller sized markers.
• I can also generate 2x2 codes, so those may actually be better and easier to see for us, so I will try those as well.